A Probabilistic Framework For Modular Continual Learning
Authors: Lazar Valkov
Akash Srivastava
Dipak Chaudhari
Swarat Chaudhuri
Charles Sutton
Venue: Submitted to ICLR 2023 (2023)
Keywords: continual learning
modular
Bayesian networks
Bayesian optimisation

Abstract: Continual learning (CL) algorithms seek to accumulate and transfer knowledge across a sequence of tasks and achieve better performance on each successive task. Modular approaches, which use a different composition of modules for each task and avoid forgetting by design, have been shown to be a promising direction to CL. However, searching through the large space of possible module compositions remains a challenge. In this work, we develop a scalable probabilistic search framework as a solution to this challenge. Our framework has two distinct components. The first is designed to transfer knowledge across similar input domains. To this end, it models each module’s training input distribution and uses a Bayesian model to find the most promising module compositions for a new task. The second component targets transfer across tasks with disparate input distributions or different input spaces and uses Bayesian optimisation to explore the space of module compositions. We show that these two methods can be easily combined and evaluate the resulting approach on two benchmark suites designed to capture different desiderata of CL techniques. The experiments show that our framework offers superior performance compared to state-of-the-art CL baselines.

Under review as a conference paper at ICLR 2023
A PROBABILISTIC FRAMEWORK FOR MODULAR CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Continual learning (CL. algorithms seek to accumulate and transfer knowledge across a sequence of tasks and achieve better performance on each successive task. Modular approaches, which use different composition of modules for each task and avoid forgetting by design, have been shown to be a promising direction to CL. However, searching through the large space of possible module compo- sitions remains a challenge. In this work, we develop a scalable probabilistic search framework as a solution to this challenge. Our framework has two distinct components. The first is designed to transfer knowledge across similar input do- mains. To this end, it models each module's ' training input distribution and uses a Bayesian model to find the most promising module compositions for a new task. The second component targets transfer across tasks with disparate input distribu- tions different input spaces and uses Bayesian optimisation explore the space of module compositions. We show that these two methods can be easily combined and evaluate the resulting approach on two benchmark suites designed to capture different desiderata of CL techniques. The experiments show that our framework offers superior performance compared sfute-of-the-arr CL baselines.
INTRODUCTION
The continual learning (CL) (Thrun& Mitchell 1995) setting calls for algorithms that can solve sequence of learning problems while performing better on every successive problem. A CL algo- rithm should avoid catastrophic forgetting - i.e., not allow later tasks to overwrite what has been learned from carlier tasks and achieve transfer across large sequence of problems. Ideally, the algorithm should be able to transfer knowledge across similar input distributions (perceptual transfer). dissimilar input distributions and different input spaces (non-perceptual transfer), and to problems with a few training examples (few-shot transfer). It is also important that the algorithm's computational and memory demands scale sub-linearly with the number of encountered tasks.
Recent work (Vlkov et al., 2018: Veniat et al. 2020 Ostapenko et al. 2021 has shown modular algorithms to be promising approach to CL. These methods represent neural network compo- sition of modules, in which each module is reusable parameterised function trained to perform an atomic transformation of its input. During learning, the algorithms accumulate library of diverse modules by solving the encountered problems in a sequence. Given a new problem, they seek to find the best composition of pre-trained and new modules, out of the set of all possible compositions, as measured by the performance on a held-out dataset. Unlike CL approaches which share the same parameters across all problems, modular algorithms can introduce new modules and, thus, do not have an upper bound on the number of solved problems.
However, scalability remains a key challenge modular approaches to CL., as the set of mod- ule compositions is discrete and explodes combinatorially. Prior work has often sidestepped this challenge by introducing various restrictions on the compositions, for example, by only handling perceptual transfer (Veniat et al. 2020) or by ignoring non-perceptual transfer and being limited by the number of modules that can be stored in memory (Ostapenko et al. 202T). The design of CL. algorithms that relax these restrictions and can also scale remains open problem.
In this paper. we present probabilistic framework as a solution to the scalability challenges in modular CL. We observe that searching over module compositions efficiently is difficult because

---

sequence in order to Under review as a conference paper at ICLR 2023
Past solutions
Set of paths, I
2
π
m
2
2
m
m4
m
2
m
m2
m
m2
m
m]
m3
m
m
m
m
m
11
m
m4
PT.1
nPT2
NN
Figure 1: The figure depicts the set of all paths, I1 that modular algorithm considers when solving the 4th problem V in a sequence. The modular architecture has 2 layers. The library comprises all previously trained modules: A4 = {ml. my. m2, m2). Paths in I7 (Eq. 2 select a pre-trained module for the first layer, enabling perceptual transfer. Paths in nT reuse modules in both layers. They can perform few-shot transfer since they only require few examples (to select the correct path). Paths in I. (Eq. 6 achieve non-perceptual transfer by reusing module in the second layer, allowing applications to input domains.
evaluating most of them involves training their new modules. This difficulty would be overcome if we could approximate module composition 's final performance without training its new modules.
Accordingly, our method divides the search space into subsets of module compositions which achieve different types of forward transfer and can be searched through separately. then explores each subset using a subset-specie signed to take advantage of the subset's properties. Querying each probabilistic model is efficient, as it does not involve training new parameters, which in turn enables scalable search method.
probabilistic model over the choice of pre-trained modules, de-
Operationally, we first develop a probabilistic model over a set of module compositions which can achieve perceptual and few-shot transfer. The model exploits the fact that the input distribution on which module is trained can indicate how successfully said module would process a set of inputs. Second, identify subset of module combinations capable of non-perceptual transfer and, using a new kernel, define a probabilistic model over this subset. We show that each of the two probabilistic models can be used conduct separate searches through module combinations, which can then be combined into scalable modular CL algorithm capable of perceptual, few-shot and non-perceptual transfer. Using two benchmark suites that evaluate different aspects of CL, we show that our approach achieves different types of knowledge transfer in large search spaces, is applicable different input domains and modular neural architectures, and outperforms competitive baselines.
BACKGROUND
A continual learning algorithm is tasked with solving sequence of problems S (V1, 4T) usually provided one at a time. We consider the supervised setting, in which each problem is char- acterised by tuple
(D,T), where D is the input domain, comprised of an input space and an input distribution, and T is task, defined by label space and a labelling function (Pan & Yang 2009). A CL algorithm aims transfer knowledge between the problems improve each problem's 's generalisation performance. The knowledge transfer to a single problem can be defined as the difference in performance when the rest of the problems are not available.
CL algorithms have several desiderata. First, an algorithm should be plastic, i.c., learn to solve new problems. Second, it should be stable and avoid catastrophic forgetting. Third, should be capable of forward transfer: the ability to transfer knowledge newly encountered problem. In particular, we distinguish between three types knowledge transfer: between problems with similar input distributions (perceptual), between problems with different input distributions or different input- spaces (non-perceptual) and to problems with a few training examples (few-shot). Fourth, a CL algorithm should also be capable of backward transfer. meaning its performance previously encountered problems should increase after solving new ones. Finally, the resource demands of CL. algorithm should scale sub-linearly with the number of solved problems.
2

---

Under review as a conference paper at ICLR 2023
Modular approaches represent deep neural network Çe as a composition of modules Ço m o m² o m', where the first module that is applied to the imput is m'. Each module m' represents nonlinear transformation, parameterised by 0(m'). It can consist of one or more hidden layers, each with a potentially different type (e.g. convolutional, fully connected, LSTMM, Transformer, tct) and with different activation function (e.g. ReLU or tanh). Given a problem, one needs to select optimal values for all parameters θ and return the resulting neural network Çe as the solution.
Modular approaches have been successfully applied to continual learning. After solving L - 1 prob- lems, they accumulate a library of previously trained modules that each have been used to help perform one or more of the previous tasks. Denote this library by A, - (m)) where m denotes a module in layer that was trained on the j-th problem. is then possible to construct different modular neural networks by either selecting a pre-trained module from the library or by training a new one from scratch for each of the layers. We denote the set of all modular neural networks as II, {((m) Jo<e U {m!)) x ... x (fmfttsct hie U (m!)), where m; denotes new module used in layer i with randomly initialised parameters. We refer to each element of II, as a path, as it can be seen as guiding the forward computation through different modules. Figure | illustraate the set of all paths IT, for the fourth problem of some sequence, using a modular architecture with 2 module layers.
Let a new problem Vr be specified by a training and a validation datasets, D (X", Y") and D (X val Yval). Evaluating each path π requires training the network on D and evaluating on Dval Let [pree] and ग nene denote the pre-trained and randomly initialised modules, respectively, from л. Training results in parameters 6%ew (0(m²) mý E nr!ool for each of the new modules. Parameters from the pre-trained modules are frozen. Thus, evaluating a path can be computationally expensive and one wants evaluate as few paths as possible. However, the search space grows quickly. If there are J pre-trained modules for each module types in the library, the search space consists of (J + 1 unique paths. This necessitates search strategy S(II,) which can prioritise the most promising paths for evaluation.
Modular algorithms can achieve many of the CL desiderata (Valkov et al. 2018). Catastrophic forgetting is prevented by freezing the parameters of all pre-trained modules in the library. A path with only randomly initialised parameters can be selected, ensuring the methods' plasticity. As more problems are solved, the pre-trained modules in the library can be reused in order to achieve all types of forward transfer, as discussed in the following sections. However, a key challenge has been to attain all transfer using search strategy which scales to the enormous search space.
PROBABILISTIC MODULAR CONTINUAL LEARNING
We propose a probabilistic approach which uses the information available about the new problem to compute a distribution over different choices for pre-trained modules. The distribution does not model the new modules, and this allows efficient queries. This leads to a principled search strategy.
Specifically, we split 1I, into subsets 11 C 11, of paths in which the reused modules are at the same layer positions. For each subset, we define a probability distribution over the choice of pre-trained modules, p(r X Y X", Y Ez). This distribution uses the available training and validation datasets, and the previously evaluated paths Ez. Here, denotes the index of the current path suggestion. This distribution can be queried efficiently, as it does not involve randomly initialised modules. Now we define search strategy for each path subset 11f. At step j. this strategy selects the unevaluated path with the highest posterior probability of its pre-trained modules:
SMAP =
arg max p( (pr) X", Y X Y F.))
Multiple subsets of II, can be explored by applying this search strategy to each sequentially.
Next, we first apply our approach to subsets of paths which are capable of perceptual and few-shot transfer. Second, we apply it to subset of paths capable of non-perceptual transfer. In both cases, we define a suitable Bayesian model, which takes advantage of each subset's 's properties. Finally, we show how the two search strategies can be combined.
3

---

Under review as a conference paper at ICLR 2023
4 EFFICIENT PERCEPTUAL TRANSFER AND FEW-SHOT TRANSFER
To achieve perceptual transfer, a model needs to transfer knowledge on how to transform the input. For path, this means reusing the first I {1....; L module layers. Additionally, a path that reuses all L layers can allow for few-shot transfer, since there are no new parameters that need to be learned.
Therefore, we start by considering paths where the first layers are selected from the library, while the rest are selected to be new randomly initialised modules, i.e.
u { iT {m'z.. S 1) U (m) (}).
(2)
Each subset grows polynomially with the size of the library and exponentially with the number of layers, which makes naive search inapplicable. Several previous papers consider this search space. For example. MNTDP (Veniat et al. 2020) only consider paths where all of the pre-trained modules come from the same previous solution. This leads to a rigid approach which cannot compose pre- trained modules in novel ways. In contrast, LMC (Ostapenko et al.) 202T) propose to model each module's input density, enabling flexible input-specific composition of modules. However, their approximations double the storage requirements and requires that all modules be loaded in memory, which could hinder scaling to a large number of pre-trained modules.
In this section, we present probabilistic model, which allows us efficiently explore novel input- specific module compositions. In contrast to LMC, we efficiently approximate each module's input density and also allow for prior knowledge over the choice of pre-traineds modules to be incorporated. Evaluating our approximations can be done without loading the corresponding pre-trained modules in memory and we show that this leads scalable search strategy.
Our motivating intuition is to select pre-trained modules so that the distribution of imputs each module receives in the new problem will match the input distribution said module trained This minimizes the amount of distribution shift faced by each module, hopefully increasing the perfor- mance of the full network. Using this insight, for given value of (, we define a probabilistic model over the choices of pre-trained modules and their N inputs (including X the problem inputs, and H' relevant hidden states). Figure 2 shows the corresponding graphical model for = 3. The hidden states and the choice of pre-trained modules are la- tent variables which are sequentially used to generate the ob- served inputs. This allows to infer the distribution over pre-trained modules, given the observed inputs:
M
M
M
X
H
H
N
Figure 2: graphical model of the joint distribution over PT path with three pre-trained modules π = (M*, M2, Ma) and hidden states H. and H?.
p(M ... M X1,..., XN, H
H
H
H
N
} Ji [((rr)) (3)
II
pl P(X,HHH
[p [p(H)- Hy M p(H
As a result, then model the posterior of a PT path only being dependent the training inputs, p( [oo))x") = =(m=,__. ..... m [x...., xN), and express it in terms of quantities which we can approximate, as derived in Appendix(A]
p(m², ..... m I xN) x p(m² m' xI,...,NN) = p(m m' nn. .i...... h2²)
N
qhi, Im²) Imm))
00h//-1/0 Il p(m')
(4)
Emitr qhh (m (p(n+t1))
where qhh5-1 (m²) denotes our approximation to the training input distribution of the module m'.
We define prior distribution over pre-trained modules which gives preference to modules which helped achieve a higher accuracy, on the problems which they have been trained on. Our motivation is that such modules have learned more generalisable transformation of their inputs. We use softmax to define a prior, p(M'). over the modules layer which is proportional to each module's previously achieved accuracy. The prior is defined in Appendix B. eq. 12
4

---

Under review as a conference paper at ICLR 2023
We set out to construct approximation of module's training imput distribution, p HH (H'+))M') which is easy to compute in terms of implementation, storage and computational resources. After a module is trained,
perform dimensionality reduction of each sample from the module's training distribution. We use random projection (Johnson] 1984) to reduce each sample's dimensionality from to < Finally, we compute the sample mean and covariance of the projected data samples in order to approximate their distribution using a multivariate Gaussian. This greatly reduces the number of parameters required to approximate the resulting distribution from + c² k + ka. where A A is layer-specific randomly generated matrix used for random projection (Pedregosa et al. 2011 Surprisingly, our ablation experiments, presented in Appendix E.2 suggest that using Gaussian approximation over randomly projected inputs also more reliable than using a Gaussian approximation over the original inputs.
When maximising the posterior using Eq. 4 compute q(h)-'mm) mm²) & (AAtt1h)
Search Strategy The number of possible PT paths of length inereases exponentially with I. As a result, evaluating p( (X") for all PT paths in the SMAP search strategy (Eq. I is infeasible. Instead, we augment the search with a greedy policy: after selecting the first pre-trained modules, we freeze this selection and reuse the same modules for PT paths with > which reuse more pre-trained modules. Also, for each value of I, the search only recommends the path which has the highest probability under our model, since this is the path which is best equipped to handle the inputs. We define the resulting search strategy over all IT as:
D (I,T") := {x PP
(5)
where x PT. = π PT
1] U U (m m arg max p(T 'PP, [:: 1 1mmmm X xN).
states that in order to select the most promising PT path with pre-trained modules, n PPT, we choose the previously selected 1 praited modules, =PT, 1 1], and append the optimal choice for layer 1, m which maximises the resulting posterior distribution.
The storage requirement of this search is associated with the number of modules in the library. Therefore, it scales linearly with the number of solved problems, Ott). The exact number depends on the type of similarity between the encountered problems. If perceptual transfer and few-shot transfer is often possible, the number of modules in the library will grow more slowly. Our greedy search strategy requires that each module from the library loaded at constant memory requirement. The computational complexity of this search remains constant with the number of solved problems since it selects a constant number of paths for each problem: L + (L suggested by our search strategy, plus randomly initialised model).
once, which leads to
5 EFFICIENT NON-PERCEPTUAL TRANSFER
To achieve non-perceptual transfer, an algorithm can use pre-trained modules for the last module layers. This represents knowledge on how to transfer latent representation of the input to task- specific prediction. Therefore, we identify subsets 11, e 11, which consists of paths capable of non-perceptual transfer. Each path has its first L - modules randomly initialised while the rest / modules are selected from library:
NT NT.
T
NT. = (mi)a U (m2c)lis--4i)
(6)
Probabilistic Model While the traitrdined modules in PT paths process the inputs, the pre-trained modules in NT paths predict the outputs. As a result, we build a probabilistic model which captures how good an NT path is in predicting the correct outputs after training. Concretely, for NT paths with the same number / of pre-trained modules, we model the posterior over the choice of pre-trained modules given the validation dataset as:
p( (mre val Y p( val X _ (((T))))))m))))
We define a prior over [ure) which assigns equal non-zero values only to pre- pre-trained modules which have been used together to solve previous problem. This reflects our prior assumption that using novel combination of modules for non-perceptual transfer is unnecessary for our sequences.
5

---

Under review as a conference paper at ICLR 2023
We model p( (YYval XX rrbeel using Gaussian process (GP), which before recommending path step J. is fit on previous evaluations Ey. For numerical stability, we model the logarithm, predicting:
p X
[prel n
GP x(m; pre)
(pre
To enable this, we define a kernel function &. between the pre-trained modules two NT paths, based on the standard Euclidean distance in function space. Let the inner product between Ω + Rº and g 12 + IR" be: (f 9) = Jn f() g(z) This allows us to compute the distance between two functions
( ((x)) (f(z) - g(z))d2
= :150)=111--0) VU -U-VVV----0
9
9)
We approximate this value using Monte Carlo integration with a set of inputs, Z, from the func- tions' common input space. We then define the kernel funetion between the chosen pre-trained modules of two NT paths with the same value, using the squared exponential kernel function and the distance between the functions computed by their last modules:
ppre K6 <<(TT.
pmee
Z) = o² exp { (d(()))
[mee 2 /
(9)
where a and y are the kernel hyperparameters which are fit maximise the marginal likelihood of a GP's training data (Rasmussen & Williams| 2006).
To create for a value of 1, we store a few samples from the input distribution of each pre-trained module at layer L - 1 + 1. Given a new problem, we create Z by combining all the stored hidden activations. This allows us to compare functions on regions of the input space with higher density.
Search Strategy We modify SMAP and apply it on a single subset TT NT. where min is the user- defined minimum number of modules which need to be transferre.. Due to our choice of prior, max- imising the posterior using Eq. 7 ammount to maximising the log-likelihood over paths with nonzero probability. However, GP-based approximation provides a distribution of the likelihood's 's value (reflecting the uncertainty of the GP's prediction). To account for this, we use an acquisition func- tion, namely Upper Confidence Bound (UCB) (Shahriari et al.) 2015) which provides an optimistic prediction of the likelihood. We compute it as UCB(H, or) = + Slo where a hyperparameter, and μ and respectively the mean and the standard deviation of the GP-predicted distribution. The resulting search strategy resembles Bayesian Optimisation:
SBO-MAP (ain arg max UCB (log (a , r
(10)
wEll;
At each step j. we GP (Eq. with kernel x (Eq. 9 on the of evaluations the previous paths Ej. and select the unevaluated path π whose pre-trained modules maximise the UCB.
The GP makes it possible to detect when further improvement unlikely, allowing us to perform early stopping. For this purpose we compute the Expected Improvement, [pee] El [Iog xvaal ) of the path selected for evaluation at each step j. If it is lower than a exit the search and do recommend any more paths for evaluation. El-based certain threshold,
early stopping has been previously suggested in |guuyen et al. (2017) and, similarly to MMkaroval al. (2022), our preliminary experiments showed that leads to fewer evaluations, compared to using UCB for early stopping.
In the worst case Sgo MAp would propose (t-11 paths for evaluation which scales linearly with the number of solved problems. While early stopping reduces the number of evaluated paths, our experiments suggest said number still scales linearly with the number of problems. However, our ablation experiments, presented in Appendix EE demonstrate that our search strategy exhibits good anytime performance. This means that our Bayesian optimisation algorithm is capable quickly finding path which achieves positive non-perceptual transfer. Therefore, it is possible constrain SRO.MAP to evaluating a constant number paths and still achieve non per-perctttual transfer across a long sequence of problems.

---

Under review as a conference paper at ICLR 2023
SA
MCL-RS
HOUDINI
MNTDP-D
PeCL, CCL
Slew
75.47
78.14
80.82
82.18
88.12
Star
74.25
76.16
74.40
77.95
78.15
Soar
72.27
73.39
72.27
75.48
75.72
A
S
71.51
73.85
71.75
73.71
75.73
SPI
93.61
93.63
93.61
93.72
93.79
S
73.88
76.67
79.59
81.67
81.92
S
73.61
75.08
73.61
74.54
74.49
Aggregated:
76.37
78.13
78.01
79.89
81.13
Stow
0.
5.87
4.54
11.42
46.07
S
0.
17.22
34.27
34.29
34.29
SThnt
0.
5.64
13.41
15.41
S
0.
0.43
12.53
12.53
Ty
S
0.
4.61
1,46
1.74
12.04
SPI
0.
0.
0.
00.20
00.20
SFF
0.
0.
0.
0.
Aggregated:
0.
4.82
5,75
10.8
17.22
Table 1: Results on compositional benchmarks that do not evaluate non- nen-perepttul transfer.
SA
MCL-RS
HOUDINI
MNTDP-D
PeCL
NoCL
CCL
S
89.01
90.85
89.32
90.62
90.26
92.20
92.82
Á
S
87.94
92.22
92.99
87.94
87.92
91.92
91.93
Aggregated:
88.48
91.54
91.16
89.28
89.09
92.06
92.38
0.
1.81
11.04
9.70
7.61
18.89
22 22.28
Tr
S
0.
25 35.68
30 27
0.
0.
23.65
23.65
Aggregated:
O.
13.75
20.66
4.85
3.805
21.27
22.97
Table 2: Results compositional benchmarks that evaluate non- non-perceptual transfer.
EXPERIMENTS
We evaluate the perceptual, non-perceptual and few-shot transfer capabilities, as well as the scala- bility, of our methods. Also, assess the applicability of our methods to disparate input domains and neural architectures. To this end, we perform experiments on a new benchmark suite as well as on the CTrL (Veniat et al. 2020) benchmark suite.
We derive three modular CL algorithms: PeCL, which can perform perceptual and few-shot transfer 5); NoCL, which can perform non-perceptual transfer using SBO-MAP (LI)T, 10; and CCL, which combines both search strategies using SCCL( U II.)
using Se (II 5Eq5):
(Eq.
SET (II SBO-MAP NT. (T. We compare our algorithms to a number of competitive modular CL baselines: Standalone (SA), which trains new model for every problem; MCL-RS, which ran- domly selects paths from II, (shown to be a competitive baseline high-dimensional search spaces and in neural architecture search (Li & Talwalkar 2020);; HOUDINI Valkov (Valov et al., 2018), with fixed neural architecture to keep the results comparable; and MNTDP-D (Veniat al, 2020).
We use the same set of hyperparameters, listed in Appendix D for our approaches across both benchmark suites, which suggests that this choice of hyperparameters robust and applicable to different problems and neural architectures. Moreover, we use a problem-specifte random seed to make the training process deterministic, so that the difference in performance can be accredited only to the LML algorithm, and not to randomness introduced during training. For each baseline, assess the performance on a held-out test dataset and report the average accuracy of the final model across all problems. . (see Eq. 14). as well as the amount of forward transfer on the last problem, Tr-1, computed as the difference in accuracy, compared to the standalone baseline (see Eq. 15
Compositional Benchmarks. We introduce benchmark suite consisting of different sequences which can diagnose different CL properties. We build on the CTrL benchmarks (Veniat et al. 2020) by instead using problems with compositional tasks, in which each problem combines image clas-

---

Under review as a conference paper at ICLR 2023
sification task with two-dimensional pattern recognition task. This allows to design sequences to evaluate specific desiderata, such as non-perceptual transfer, few-shot transfer, and scalability to long problem sequences. We have the following sequences of length 6. In S each problem is represented by a large dataset in order to evaluate an algorithm's 's plasticity. In 5 the last problem is the same as the first, but is represented with larger dataset, allowing for backward transfer to be evaluated. Conversely, in S the first and last problems are also the same but the first is represented by bigger dataset, thus, evaluating forward knowledge transfer to the same problem. We define three different sequences which evaluate perceptual transfer. In S the input domains of the first and last problems are the same. In S the input domains of the first, second and last problems are the same, making it harder to decide which problem to transfer from. In S the first and last problems are the same, however, perceptual knowledge needs to be transferred from the sec- ond problem, as it also has the same domain but is represented by larger dataset. Srew evaluates few-shot transfer, since the last problem is represented by 10 data points and solving it requires re-combining knowledge from problems 2 and 4 in a novel way. Finally, there are two sequences evaluating non non-perceptual transfer. In S the first and last problem share the same -wo-dimensional pattern, while having different input image domains. 5 is the same 5 except last problem's 's input space is also re-shaped from an image to a one-dimensional vector, which loses the structural information and requires different architecture for the first few modules in order to process the input. Therefore, this evaluate an algorithm's 's ability to transfer across different input spaces and similar but not identical neural architectures. Finally, Çlong a sequence of 60 randomly selected problems, which evaluates perceptual and non-perceptual transfer on long sequences.
The neural architecture which we use consists of L 8 modules, which leads to a large search space, since even for t = 6, there are O(t 1679616) 6) different possible paths. Full specification of the problems, sequences, neural architecture and training procedure can be found in Appendix (D) We create 3 realisations of each sequence by randomly selecting different compositional problems. Afterwards, for each sequence, we report the measurements, averaged over these 3 versions.
Results. Overall, CCL. has higher performance than the other methods, averaged across sequences (Tables [ and D. Our method is designed specifically to enhance perceptual, few-shot and non- other CL desiderata to be comparable to
perceptual transfer. Therefore, we expect its performance the baselines. We find that PeCL and CCL exhibit the same performance for sequences which do not involve non-perceptual transfer (Table 1). Our method demonstrates similar plasticity to the others, according to Sp', as well as similar performance on S" which evaluates backward transfer. Note that S enables our algorithms outperform the baselines on the sequences which require forward knowledge transfer, with the other scalable modular algorithm, MNTDP-D, being second. On Sour". our nerformance-basee prior helps our method identify the correct modules to be transferred, leading to 1003 higher accuracy on the last problem than MNTDP-D. On sfow our algorithm''s 's ability to perform few-shot transfer leads to +34.7 higher accuracy on the last problem than MNTDP-D.
For non-perceptual transfer (Table 2). we find that SRO- enables NoCL and CCL to transfer knowledge across different input distributions and input spaces. In S the different input space necessitates different modular architecture for the first 5 modules, resulting a much smaller search space. O = 216). This allows non-scalable approaches, namely MCL-RS and HOU- DINI, to also be effective on this sequence. Lastly, CCL's performance demonstrates that combining SpT and SBO-MAP leads to better performance on sequences that allow for both perceptual and non-perceptual transfer. Finally we evaluated our approach on glong, a sequence of 60 problems. PeCL achieved +7.37 higher average accuracy than the standalone baseline demonstrating its abil- ity to achieve perceptual transfer on a long sequence of problems. MNTDP-D achieved 8.83 higher average accuracy than SA which confirmed its scalability. Finally, CCL. performed the best, attain- ing +12.25 higher average accuracy than SA. This shows that our approach can successfully attain perceptual and nor-percep transfer across a long sequence of problems.
CTrL benchmarks. The CTrL benchmark suite (Veniat et al. 2020) defines fewer sequences to evaluate different CL properties. Namely, they specify SP. St. S SSuut, SuP which are defined identically to ours. In contrast, the sequences images from different domains. They also use a different modular architecture, based on R8 (He al. 2016), which is more complex than the architecture used in the previous benchmarks. Our experimental setup, detailed in Appendix [] mirrors the one used in Veniat et al. (2020), except that we make the training process deterministic, discussed above.
over multi-class classification tasks of coloured
8

---

Under review as a conference paper at ICLR 2023
SA
MNTDP-D
PeCL
NoCL
CCL
Sth
58.77
61.36
61.78
63.41
63.10
5
74.25
77.95
78.15
78.15
A
S
58.25
93.72
93.79
93.79
S- 56.28
81.67
81.92
81.92
S 73.61
74.54
74.49
74.49
Aggregated:
64.23
77.85
78.03
78.29
St
0.
22.12
24.67
32.57
32.57
S
0.
15.41
15.41
15.41
- 5Th
0.
00.2 20
00.20
00.20
5
0.
34.29
34.29
34.29
5
0.
0.
0.
0.
Aggregated:
0.
14.40
14.91
16.49
Table 3: The evaluations the CTrL sequences, except for Slong, For each sequence, we report average accuracy A and the amount of forward transfer on the last problem Ty-1
Our results (Table 3) demonstrate that our methods, namely PeCL and CCL, achieve similar per- formance to MNTDP-D on SPI, S* S Souut which evaluate plasticity, backward and perceptual transfer. On S NoCL and CCL successfully perform non-perceptual transfer, leading superior performance on the last problem of the sequence (+10.445 higher than MNTDP), CTrL also specifies Slung which has 100 problems but only evaluates perceptual transfer. PeCL was successfull in trans- ferring knowledge, achieving an average accuracy that was - 14.48 higher than SA. Overall, the results demonstrates that both SBO-MAP are also applicable to more complex modular architectures.
RELATED WORK
This work considers the data-incremental (De Lange et al.) 2021) supervised setting of continual learning. Other settings can involve overlapping problems Farquhar & Gal (2018) reinforcement learning (Khetarpal et al. 2020). CL desidarata is also presented in [Schwarz et al. (2018), Hadsell al. (2020); Delange al. (2021), which include other properties, e.g. selective forgetting of trivial information, but do not distinguish between different types of forward transfer. We list such types following Valkov al, (2018), but add the requirement of non-perceptual transfer over imput spaces.
Continual learning methods can be categorised into ones based on regularisation, replay or a dy- namic architecture (Parisi al. 2019). The first share the same set of parameters across all problems which limits their capacity and, in turn, their plasticity (Kirkpatrick et al. 2017). Dynamic architecture methods can share different parameters by learning a problem-speeiffe parameter masks (Mallya & Lazebnik 2018) adding more parameters (Rusu et al. 2016). This category includes modular approaches CL. which share and introduce new modules, allowing groups of parameters to be trained and always reused together. Modular approaches mainly differ by their search space and their search strategy. PathNet (Fernando et al., 2017) uses evolutionary scarch to search through paths that combine up modules per layer. Rajasegaran al. (2019) use random search the set of all paths. HOUDINI (Valkov et al. 2018) uses type-guided exhaustive search on the set of all possible modular architectures and all paths. This method attain the three types of forward transfer, but does not scale to large search spaces. Their results also show that exhaustive search leads to better performance than evolutionary search. MNTDP-D (Veniat al., 2020 is scalable approach which restrict its search space to perceptual transfer paths, derived from previous solutions. Similarly SeT MNTDP-D evaluates only not allow for novel combinations of pre-trained modules which prevents it from achieving few-shot transfer. In contrast, our approach can achieve three types of forward transfer. LMC (Ostapenko) et al., 2021) makes a soft selection over paths. For each layer, they compute linear combination of the outputs of all available pre-trained modules. do this, the authors model the distribution over each module's outputs using separate auto-encoder. In contrast, for SPT we use multivariate the projected inputs of a pre-trained module, which requires orders of magnitude fewer parameters. LMC can achieve perceptual and few-shot transfer, but is not capable of non non-perce porcppecltllll transfer. Finally, LMC requires that all modules be kept in memory which limits it applicability to larger libraries, thus, larger search spaces.
paths per problem, however, the approach does
Gaussian approximations

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:/8/2.017/6 2018.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. 2017 International Joint Conference on Neural Networks (IJCNN), 2017. doi: 10. 10.1109/ijenn.000 2017. 7.666776.....
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-3385, 2021.
Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
Norman R Draper and Harry Smith. Applied regression analysis, volume 326. John Wiley & Sons, 1998.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. arXiv preprint 18X0::8805009333 2018.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734 2017.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning, pp. 1920-1930, PMLR, 2019.
GPy. GPy: A gaussian process framework in python. GPy since 2012.
Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences, 24(12):1028-100 1040, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
William B Johnson. Extensions of lipschitz mappings into hilbert space. Contemp. Math., 26: 189-206, 1984.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforce- ment learning: review and perspectives. arXiv preprint arXiv:2012. 13490, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6998 6980, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barnkkakaaiikkaaa...am al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-356 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFF, (6(11):2277-2324 1998.
10

---

Under review as a conference paper at ICLR 2023
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http:/hyam.lecum. http://wwwe.co..o///bbbbbeee cww/exdbmmnisstt 2, 2010.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in Artificial Intelligence, pp. 367-377. PMLR, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05101, 2017.
Anastasia Makarova, Huibin Shen, Valerio Perrone, Aaron Klein, Jean Baptiste Faddoul, Andreas Krause, Matthias Seeger, and Cedric Archambeau. Automatic termination for hyperparameter optimization. In First Conference Automated Machine Learning (Main Track), 2022.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Vu Nguyen, Sunil Gupta, Santu Rana, Cheng Li, and Svetha Venkatesh. Regret for expected im- provement over the best-observed value and stopping condition. Asian conference on machine learning, pp. 279-294, PMLR, 2017.
Oleksiy Ostapenko, Pau Rodriguez, Massimo Caccia, and Laurent Charlin. Continual learning via local module composition. Advances in Neural Information Processing Systems, 34:30298- 30312, 2021.
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions on knowledge and data engineering, 22(10): 1355-13999999 2009.
German Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 11::5-771, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Detito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL http: /pwp:rs//pppppappeeeeeaeeee wew.ees...eurrrr...c.m cctpppper pdf
library.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
Jathushan Rajasegaran, Munawar Hayat, Salman Khan, Fahad Shahbaz Khan, and Ling Shao. Ran- dom path selection for incremental learning. Advances in Neural Information Processing Systems, 3, 2019.
Carl Edward Rasmussen and Christopher KI Williams. Gaussian processes for machine learning. 2006.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606.0466 2016.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska- Barwinska. Yee Whye Teh, Razvan Pascamu, and Raia Hadsell. Progress compress: A scalable framework for contin- ual learning. In International Conference Machine Learning, pp. 4528-45377 PMLR, 2018.

---

Under review as a conference paper at ICLR 2023
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1): 148-175, 2015.
Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. Robotics and autonomous systems, 15(1-2)225-6, 1995.
Lazar Valkov, Dipak Chaudhari, Akash Srivastava, Charles Sutton, and Swarat Chaudhuri. Houdini: Lifelong learning as program synthesis. Advances in Neural Information Processing Systems, 31, 2018.
Tom Veniat. http://////ggttaaaca..////////aa
2021.
Tom Veniat and Marc' Aurelio Ranzato. Continual transfer learning benchmark.
https://
2021.
Tom Veniat, Ludovic Denoyer, and Marc' Aurelio Ranzato, Efficient continual learning with modular networks and task-driven priors. arXiv preprint arXiv:2012. 12631, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist. a novel image dataset for benchmark- ing machine learning algorithms, 2017.
12

---

Under review as a conference paper at ICLR 2023
A DERIVING THE POSTERIOR FOR PT PATHS
In this section we show how we approximate the posterior distribution p(m- m²zr xi..... ..,.....y) To ease the presentation, and without loss of generality, we the number of pre-trained modules I = 3. Next, we express the joint distribution in terms of quantities which we can approximate.
p(M M2, M X1, XN. HI
H Hi H
N
= =P(M))((M (M Il p( H M DEE
2=1
Here, p( H, M can be expressed as:
p(X M¹) HJ p(H) X, M (XX (X(,)' Do(M)) (HH))((M))
p(H) (,,M' M M3 )p())A1))
D M1)
Emer DHH |m2 )p(=)))
Moreover, P(H3) Hy, 2 can be expressed as:
(3) p p(H), HJ,
p(H)) HJ (H) ( (M2)
p(H3.
p(H3)
J. AHH
Em p(H3 )m(m)
Therefore, the joint distribution can be expressed as:
p(M' MM M X1, m. XN, HI H HT, H2))
N p(H) M ((X,)) P(H MF) (HH
= (MM)))(MM))(MM)))))))
Em pl HH) m² )p(m²') Emv pl P(H mm )p(m3') p( p(H3))M2)
(11)
This expression contains three groups of distributions, which we need to define. First, we can define a prior over the choices of pre-trained modules for different module layers, p(M*). Second, we can approximate each module's training input distribution, leading p(H) (m = q (m)+()))) ) The third group of distributions contains p(Hj)13 " ,M') which is a distribution over the values of hidden layer H' given the previous hidden layer h and module m'. However, said hid- den layer is given by the deterministic transformation h; m (h5-)). Therefore, we can model p(HjHH MM) = 8(H) - h2) using the Dirac delta function ò. This function has the property that - (z)8(z c)d= f()), which we use next in order to simplify the posterior. We write:
p(m² ,m, ,x,,,,,,,,,, xN) p(m², m², m XI,.,,,, x p(mi m² m² xi, XN)
p(x1, xN)
p(m², m² m X1 XN, h hi hi2 hi,...,
Maz
p(h)) lm3)
= =p(m') )m
o p(h)) Im2 )p(m²") Emer p(h)) In (m(m3)) p(h mh3
N Σ am (b//lm² }p(m?') Em (h3) (h3|me m² )m(m") (h3lm²?
((x;lm²t
= (p(mm))(mm )m
13

---

Under review as a conference paper at ICLR 2023
In general, for 1 pre-trained modules, we approximate the numerator of the posterior using:
p(m²,... m pi XN) x p(mm, ,..,",,. m',
XN)
N
q(b) m²) Em ((bj/mmt )p(mett)) (nnmmttt)m(mmt
a(h' a(h//'|m²)) }
p(m')
II
To compute this, we need to define a prior distribution over modules and approximate a module's training input distribution.
B DEFINING THE PRIOR FOR PT PATHS
Computing Eq. 4 requires us to define a prior distribution over the choice of pre-trained module, p(M'). Say two modules m and mi, are trained using two different paths on two different problems. Now, say that the model trained on problem a achieved Acc(a) validation accuracy, while the model trained on problem b achieved higher validation accuracy Ace(b) = Acc(a) + 6, for 8 > We hypothesise that the module, whose model achieved the higher accuracy after training, is likely to compute a more useful transformation of its input. Therefore, if m and mi have similar likelihood for a given set of training data points, we would give preference to using mn. To this end, we define the prior distribution in terms of a module's original accuracy using the softmax function as follows:
plm3)
(12)
Em
Aecla
Here T is the temperature hyperparameter which compute follows. Suppose that, for given set inputs (x1,... X XN). we have selected the first - modules and have computed the inputs to the ith module, DH ( Moreover, suppose that the likelihood of module m is -ー
slightly higher than the likelihood of mi. i.e. that D(D) ( > p(D mm)). However, be- cause the model of mi was trained to higher accuracy ( (Acc(b) = Acc(a) + 6), we would like to give mú, preference over m Therefore, we would like to set the hyperparameter T so that the pos- terior of the path using mi is higher, i.e. p(mlke..., mi, Xi,..,,xy XN) < p(mp mg. x1,... XN) Using Eq. 4 we can express this as:
pas.. m² ..,, XN) < <pm my mi ,I.., xN) 11 (mm)) < p(m)))) (() mmm))
p(mm))
D(DH
m))
p(m²)
p(DH-
m
Andio:
D(DH
mm))
(13)
Acciur1a
P(DH--
m
Acc(a)
Acc(a)
+
D(D
mm))
log
T
T
p p(DH m)))
log p( D
(m;)) log p( DH
mm))
We can then use the inequality in Eq. 12 in order to determine the value T. To do this, one needs to decide how much difference in log likelihood should be overcome by a difference 8 in accuracy.
C BELL -ENNCHARRS FOR LIFELONG LEARNING
We now introduce BELL- - a suite of benchmarks for evaluating the aforementioned CL properties. As above, assume compositional tasks and then generate various lifelong learning sequences, with each evaluating one or two of the desired properties. Running an CL algorithm on all se- quences then allows to asses which properties are present and which are missing. This builds
14

---

Under review as a conference paper at ICLR 2023
Sequence
Sequence Pattern
Properties
Spi
Vi. 22 V
VI
V5 V
1,, 2.
5
V 12 W.
V
V v
3.
Sou
VT. V. 3 V 4. V5 V. = (D1, h1, [6)]
3.1
Sow
V 2/2 = (D1, 3/ 4 V 5
(1,)01
3.1
Sour
V 2/2 =(11.11.92 3/ V V.
3.1
Sth
VT. 22 V3. V1, ; VT V. = (Da, h6, 91
3.2.1
Sup
W 22 V3 44 5, V6 (Do, ho, 91)
3.2.2
Stow
V (D1, h1), V. = D2.h2) V. V h1, V5. Vo- = =(2,, h2 94)]
3.3
5+
V 2 3: VT VT VT]
4.
5long
V 100
5.
Table 4: A list of all of the different CL sequences in BELL, each of which evaluates different CL. properties. The first column contains the sequence's name, the second shows the sequence's s pattern and the third column lists the CL properties evaluated by this sequence.
upon the CTrL benchmark suite, presented in Veniat et al. (020)) which defines different sequences of image classification tasks, namely S S S S S and Slong, They evaluate plasticity, per- ceptual transfer, non-perceptual transfer, catastrophic forgetting, backward transfer and scalability. We define these sequences similarly but for problems with compositional tasks. This allows to introduce new sequences which evaluate new CL properties (S and S We also introduce new more challenging sequences (Sow* and (oou**).
We assume compositional tasks and represent each problem as a triple ; (D,, hg. g&) where D, is the distribution of the inputs and hy and 9n constitute the labelling function f. i.e. f= h. We refer to hy as the lower labelling sub-function and to Ok as the upper labelling sub-function. We use the indices and A indicate whether the corresponding labelling sub-function has occurred before in the sequence (j < t, k < t) or if it is new and randomly selected (j t, k t). For brevity, if = t and & we don't write out the whole triple but only v,, By repeating previously encountered domains or labelling sub-functions. we can control what knowledge can be transferred in each of the define sequences. turn, this allows us to evaluate different CL properties. We use T to indicate that the dataset generated for this problem is sufficient to learn well generalising approximation without transferring knowledge. On the other hand, V indicates that the CL algorithm cannot achieve good generalisation on this problem without transferring knowledge. Finally, - indicates that the generated training dataset consists of only a few datapoints, c.g. 10.
A complete list of the different sequences in BELL is presented in Table Following Veniat et al. (2020), we set the sequence length of most sequences to 6 which, as we show in the experiments sec- tion, is sufficient for evaluating different CL properties. Next, separately present each sequence, detailing which CL properties it evaluates.
Plasticity and Stability: The sequence Sti [ V V3 problems, each of which has a different input domain and a different task. Moreover, each of the necessitate transfer. Therefore,
consists of 6 distinct
generated datasets has sufficient number of data points as this sequence evaluates CL algorithm's ability to learn distinct problems, i.c. its plasticity (1.). Moreover, this sequence can be used evaluate an algorithm's 's stability (2.) by assessing its perfor- mance after training on all problems and checking for forgetting.
Forward Transfer: Most of our sequences are dedicated to evaluate different types of forward transfer. To begin with, in the sequence S = [wT V V. VT V5 VT] the first and the last datasets represent the same problem, however, the last dataset has fewer data points. Therefore, an CL algorithm would need to transfer the knowledge acquired from solving the first problem, thus, demonstrating its ability to perform overall forward transfer (3.).
Perceptual Forward Transfer: We introduce three different sequences for evaluating perceptual transfer (3.1). First, in S = [ , V, V. V V. V = (D1, h1,96]] the last problem has the same input domain and input-processing target function h in problem 1. However, the last problem's dataset is small, therefore, an CL algorithm needs to perform perceptual trans- fer from the first problem, which is described by large dataset. Second, Sow [VT v
(D1, hi, 92), V3 VT 17 Vo
(Di, ge)] shares the same input distributions and lower la-
15

---

Under review as a conference paper at ICLR 2023
belling sub- sub-function hı across problems 11. 22 and Ve Therefore, a CL algorithm needs to decide whether to transfer knowledge obtained from the first or from the second problem. Third, the se- quence Sour* [ T _ (D1, .2,2), V2 VT 455 V is similar to the preceding one, with the distinction that the last problem is the same as the first. In this sequence, an CL algorithm needs to decide between reusing knowledge acquired from solving the same problem 1V)), or transfer perceptual knowledge from a more different problem 22) Overall, these three sequences are de- signed to be increasingly more challenging in order to distinguish between different CL algorithms which are capable of perceptual transfer to a different extent.
Von-Perceptual Forward Transfer: Currently, we define two sequences to assess an algorithm's ability to transfer non-perceptual knowledge. Firstly, in Sth = [V.. V2 V3 VT V Vi (Da, has 91)] the last problem has the same upper labelling sub-function as the first problem. How- ever, the two problems' input distributions and lower labelling sub-functions are different. There- fore, an CL algorithm would need to transfer knowledge across different input domains (3.2.1). Secondly, the sequence S [VT 2 V Va V V = (De, hG, 91) is simiarly defined, however, the input distribution of the last problem is also defined on a different input space from the input space of the first problem. Therefore, algorithm would need transfer knowledge across different input spaces (3.2.2).
Few-shot Forward Transfer: In order to evaluate this property, we introduce the following se- quence, in which the first two problems are different from the rest of the sequences: Stew = |
(D1, h1), 3 = (D2, h2), V V = (D1, h1, 91), V V = (D2, h2, 9.)] The labelling func- tions of the first two problems are simpler, each consisting only of a lower labelling sub-function. This is done in order to provide CL algorithm with more supervision on how to approximate 11 and 91 more accurately. The fourth problem V in this sequence then shares the same input domain and lower lableling sub-function as the first problem, but introduces a new upper labelling sub- function g4. The last problem then shares the input domain and the lower labelling sub-function of 22, while also sharing the upper labelling sub-function of problem 44. Moreover, the last problem's 's training dataset consists of only a few data points. Therefore, a CL algorithm would need to reuse its ap- proximations of h2 and 91 in a novel manner in order to solve the last problem.
has the same first and last problem. However, the first dataset has significantly less data points than the last. Ideally, CL algorithm should use the knowledge acquired after solving the last problem in order to improve its performance on the first problem. While this sequence represents a starting point for evaluating backward transfer, it is possible to introduce other sequences, representing more elaborate evalua- tions. For instance, introducing sequences which evaluate perceptual and non -orepeeettl backward transfer separately. However, as backward transfer is not the focus of this thesis, this is left for future work.
Backward Transfer: The sequence S+ = [VT 22. V3 V VT]
Scalability: This property can be evaluated using long sequence of problems. For this purpose we define glong = ( 11 which consists of 60 problems, each randomly selected with replacement from set of problems. Most problems are represented by a small dataset, V, Each of the first 50 problems has a 1/2 probability of being represented by a large dataset, V. Each problem also has a TU probability of being represented by an extra small dataset, V
The definitions of these sequences rely on three sets of domains, lower labeling sub-functions and upper labelling sub-functions, respectively. In turn, these can be used to create a set of problems. Next, present set of problems which can be used together with the aforementioned sequence definitions in order to evaluate CL algorithms.
C.I COMPOSITIONAL PROBLEMS
To implement the sequences defined above, one needs to define a set of compositional problems. To this end, we define 9 different pairs of an input domain and a lower labelling sub-function, {(D,, h.) H Moreover, we define 16 different upper labelling sub-functions {g. 17, These can be combined into a total of 144 different compositional problems.
First, we define 9 image multi-class classification tasks, which all share input and output spaces 228x28 + R but each have a different input distribution D; and a sppaif- labelling function h4 Concretely, we start with the following image classification datasets: MNIST (Le- Cun et al.] 2010), Fashion MNIST (FMNIST) (Xiao et al., 2017). EMNIST (Cohen et al. 2017) and Kuzushiji49 (KMNIST) (Clanwwat et al. 2018). Since some of the classes in KMNIST have
16

---

Under review as a conference paper at ICLR 2023
significantly fewer training data points, only use the 33 classes with the following indices:
[0, 2,2,4 12, 15, 17 21, 24 - 28, 30, 34, 35, 37 - 41, 46, 47]. as they have a sufficient num- ber of associated data points. We split the image datasets into smaller 8-class classification datasets. We use i to denote the different splits of the same original dataset. For instance EMNIST, represents the third split of EMNIST, corresponding to classification task among the letters form 'i' to "p'- Using this we end up with the following 9 image datasets: M FMNIST1, {EMNIST; J [KMNIST.) ) For each of these image datasets, we aside 4800 validation images from the training dataset. We also keep the provided test images separate.
XOR
CIRCLE
X
SQUARE
12345678 12345678 12345678 12345678
Figure 3: An illustration of the four two-dimensional patterns which are used by the four g(2) func- tions to label the input coordinates. Green indicates a positive label, and red indicates a negative label.
Second, we define set of binary classification tasks, which map R + {0, 13. Each task's la- belling function Pi receives two concatenated 8-dimensional one-hot encodings and returns binary value, indicating the given combination of classes, represented by the input, fulfils a certain criteria. We further decompose the labelling function into gi (x) = -,22 (g," (xx: 8)), (x[8 :))). Here, 9, maps one-hot encoding to an integer between 1 and 8. For instance, of maps the first dimension to 1, the second to and so on. As result, we use (1) convert the initial input of two one-hot encodings to wwo-dimensional coordinates. We define 4 different (d mappings, where gi defined as above, and gi 9 and g each map the dimensions to a different randomly selected integer between and 8.
(2) (3)
At the same time, each g(2)) R {0, 1} outputs whether a given two-dimensional coordinate is part of certain pattern or not. We define different g( functions, each corresponding to one of two-dimensional patterns, shown in Fig In total, these functions need to label 8 * 8 = 64 different two-dimensional coordinates.
We fuse the 4 different g( functions with the 4 different g(2) functions to define 16 different functions:
(o (0(x-1)+++(x)) = =9k (0, "x] 8)), 90 (x(8 :)), k e 2,,,334),, (1,2,3,4))
S
Figure 4: An example input for v = (DANIST, /IMNIST,.9/ = (xXOR= 1 ( These images are classified by MNNIST and then are mapped to the coordinates (1, 6) by 9 (1) since they represent the first and sixth classes respectively. Afterwards, Ixor labels this input as O, using the XOR pattern, shown in
(2)
Finally, we can combine our 9 image classification datasets [(D;, 1=)) with our 16 binary clas- sification tasks, in order create 144 compositional problems { (M( = (Da,h hu, 95), k problem V (Dr, h4s ,,) two images sampled from
{1. 9}.j {1,... 16}]. }. The input D, Each image is labelled by h, each resulting in an -ige--iiie one-hot encoding of the corresponding image's 's class. The two one-hot encodings are then concatenated and labelled by gj.
17

---

Under review as a conference paper at ICLR 2023
(2) which results in binary label. An example for v (DMNIST, AMNNIST 9 (9XOR I )) is shown in Fig 4
Sequence S involves transferring across input spaces by having its last problem's 's input domain be defined over a different input space. To create this domain we flatten any randomly selected domain from IR 28 to 1784, This loses the images' spacial information and requires that a different neural architecture is applied to process those inputs.
C.2 REALISING THE SEQUENCES
To implement a sequence S of length I, we need to select concrete compositional problems which fit the pattern specified by said sequence. Let the sequence have (()) different pairs of image domain and lower labelling sub-funciion, and ((2) different upper labelling sub-functions. For all sequences, apart from glong, we select ((2) pairs of (D,, h4) by sampling from the set of all possible image classi- fication tasks, without replacement. Similarly, we select ((2) different upper labelling sub-functions by sampling without replacement from the set of available binary classification tasks {g: 1 For glons, we use sampling with replacement.
If a problem's training dataset needs to be large, VT, we generate according to the triple ni = -30000, Alle, All). The first value indicates that we generate 30000 data points in total. The second value indicates how many unique images from the ones set aside for training, when generating the inputs. In this case, we use all the available training images. The third value indicates how many out of the 64 unique two-dimensional coordinates, used by the upper labelling fub-function, are represented by the input images. In this case, we use all awo-dimensional coordi- nates.
used
Some of the problems' training datasets are required to be small and to necessitate transfer. For sequences S S S Suut* SSee , we generate the training datasets of each problem w using the triple ni = (10000, 100, All). This way, only 100 unique images are used to generate the training dataset, so solving the problem is likely be difficult without perceptual transfer. The subset of unique images is randomly sampled and can is different between two problems which share an input domain. For sequences 5tn and 5TP, which evaluate non-perceptual transfer, we use the triple ny (10000, Allg, 30). As result, the generated datasets will only represent 30/64 of the two-dimensional coordinates, which is not sufficient for learning the underlying two-dimensional pattern. Therefore, these problems will necessitate non-perceptual transfer. When generating dataset for a problem V in the sequence 5long, we randomly choose between the two, namely between (10000, 100, All) and (10000, All,, 30).
For the problems in which the training dataset needs to contain only a few data points, w-,, we use the triple "hr (10, 20, 10). This creates only 10 data points, representing 20 different images and 10 different two-dimensional patterns.
For problems with -ー we use the triple n (10, 20,20) 10) for generating the validation dataset. For the rest of the problems, we use the triple n.al (5000, All All). Finally, we generate all test datasets using the triple niest = (5000, Alless All).
D EXPERIMENTS
Across both benchmark suites, we use the same set of hyperparameters for our algorithms, which we initially selected for a different set of problems in our preliminary experiments. This suggests that this choice of hyperparameters is robust and applicable to different problems and neural archi- tectures. For PeCL and CCL we project the hidden states to 20 dimensions before modelling the resulting distribution. Moreover, we use 0.001 as the softmax temperature parameter for the prior distribution over pre-trained modules. For NoCL and CCL we store 40 of the training inputs of each of the pre-trained modules used in the (L- mmi + 1)th layer. The value for mmin is 3 for both the BELL and the CTrL benchmarks. The jitter of the expected improvement is 0.001 and the threshold used for early stopping, is 0.001. To calculate the value of the upper confidence bound (UCB), we set the hyperparameter 3 in order encourage exploration over exploitation.
18

---

Under review as a conference paper at ICLR 2023
We use two measurements to assess the performance of each LML algorithm. First, we compute the average accuracy, A, across all problems after the last problem is solved. An algorithm's s average accuracy after it is trained on some problem sequence 5 = 1W,, 22..., 1is)) can be defined as:
A(S)
Σ A(V.) S
1
(14)
where A(F;) denotes the algorithms' final accuracy on problem V, evaluated on held-out test dataset. Second, we compute the forward transfer on the last problem only, Tr for the sequences which the performance on the last problem diagnoses an LML property. We compute it as the difference between the final accuracy on the last problem by an LML algorithm, and this accuracy for a standalone baseline, ASA:
T (S) s() ASA ( (V)s))
(15)
We do not evaluate backward transfer, nor forgetting, since all the LML algorithms which eval- uate are immune to forgetting, and to backward transfer, by design as they freeze any previously trained parameters.
In our experiments, we make the training process deterministic, so that the difference in perfor- mance can accredited only to the L.I. algorithm, and not due to randomness introduced during training. For this purpose, we fix the random initialisation of new parameters to be problem and path-specific. In other words, for given problem, if a model with the same path instantiated twice, it will have the same initial values for its randomly initialised parameters. Moreover, we fix the sequence of randomly selected mini batches seen during training to be the same for given problem. Finally, command "torch.use deterministic algorithns(True). The overall results is that, for a given prob- lem and given library, evaluating the same path will always result in the same performance, even across different modular LMI. algorithms.
PyTorch for our experiments, we fix the random seed and use the
All experiments are implemented using PyTorch 1.11.0 (Paszke et al. 2019 We also GPy's Since
since 2012) implementation of a Gaussian process. We run each LML algorithm on a single sequence, on separate GPU. All experiments are run on single machine with Tesla P100 GPUs with 16 GB VRAM, 64-core CPU of the following model: "Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz", and 377 GB RAM.
D.1 BELL BENCHMARKS
We first evaluate algorithms and the baselines on the benchmark suite which we introduced in Section C
We create 3 versions of each sequence by randomly selecting different compositional problems. Then, for each sequence, we report the measurements, averaged over these 3 versions.
D.1.1 NEURAL ARCHITECTURE
Here, we present the minimal neural architecture which we have found to be suitable for solving said problems.
We first define a convolutional neural network SCNN _28x28 + R suitable for processing im- ages from the image classification datasets. We use a 5-layer architecture with ReLU hidden ac- tivations and a softmax output activation. The layers are as follows: Comv2d( channels=l, Connaelliuutccammessll output channels=64, kernel size=5, stride=2, padding=0), Conv2d(inptt channels=64, out- m41_chamell=66, Sizee, stride=2, padding=O), flatien, FC(4*4*64, 64), FC(64, 64), FC164, 10). Here, Conv2d specified a two-dimensional convolutional layer and FC specifies a fully-comnected layer.
Second, we define a fully-connected neural network for processing a concatenation of two 8- dimensional one-hot embeddings, (MIP 16 - R It consists of 2 FC hidden layers with 64 hidden units and RELU hidden activations, followed by an output FC layer with a sigmoid activa- tion.
For a compositional problem V = and the expected output is binary classification. We solve it using the architecture Coump
the input is a 2-tuple of images, xx)
19

---

Under review as a conference paper at ICLR 2023
Saur(concctteate((cwwW3 CCNN (x2))). This architecture processes each of the input images with the same ÇUNN model. Then the 2 outputs are concatenated and processed by a (MLP model.
We represent this as modular neural architecture by considering each of the 8 parameterised non- linear transformations to be a separate module. This increases the number possible paths for each problem. As result, for the 6th problem in sequence, the number of possible paths is upper bounded by O = 1679616). Therefore, in this setting, even sequences of length are challenging for modular LML approaches.
The input space of the last problem of sequence S is given by 8-dimensional vector: Therefore, only for this problem, we replace ÇoNN with different architecture, (rc. which consists of two fully connected layers, with a hidden size of 64, uses ReLU as hidden activation and softmax as its output activation.
We train new parameters to increase the log likelihood of the labels using the AdamW optimiser (Loshchilov & Hutter 2017) with 0.00016 learning rate, and 0.97 weight decay. The training is done with a mini batch size of 32 and across 1200 epochs. We apply early stopping, based on the validation loss. We stop after 6000 updates without improvement and return the parameters which were logged to have had the best validation accuracy during training.
D.2 CTRL BENCHMARKS
They define a number of se- quences, based seven image classfication tasks, namely: CIFAR10 and CIFAR100 (Krizhevsky et al. 2009), DTD (Cimpoi et al., 2014), SVHN (Netzer et al.
The CTrL benchmark suite was introduced in Veniat et al. (2020).
2011), MNIST (LeCun et al.) Xiao al. 2017). All images are rescaled to 32x32 pixels in the RGB color format. CTrL was first to introduce the following sequences: S St Stn S Sel and glung, which are defined similarly to our definitions. How- ever, the difference is that they are defined for and implemented by image classification tasks. The last task in Sin, which evaluates non-percepe con-perceptual transfer, is given by MNIST images with a different background color than the first task. The last task in Sour is given by shuffling the output labels of the first task. 5long has 100 tasks. For each task, they sample a random image dataset and a random subset of 5 classes to classify. The number of training data points is sampled according to distri- bution that makes it more likely for later tasks to have small training datasets. In contrast to us, they only 1 selection of tasks for each sequence, i.e. realisation of each sequence. To generate the sequences, we use the code provided by the authors (Veniat & Ranzato 2021).
1998. RainbowMNIST (Finn al. 2019), and Fashion MNIST
The neural architecture used is small variant of ResNet18 architecture which is divided into 6 modules, each representing a different ResNet block (He et al.) 2016).. While the paper presenting the CTrL benchmark states that modules used, used the authors' code (Veniat (20TT) for this method which specifies only 6 modules with the same total number of parameters. The difference from the architecture stated in the paper is that the output layer placed in the last module, instead of in a separate module.
All parameters are trained reduce the cross-entropy loss with an Adam optimiser Kingma Ba (0144) with 81 = 0.9, 31 = 0.999 and = 10-8. For each task, each path is evaluated 6 times with different combinations of values for the hyperparameters the learning rate (f10-2, 2, 10 10-3)) and of the weight decay strength (0, 10-5 10-4). The hyperameters which lead to the best validation per- formance are selected. Early stopping employed during training. no improvement is achieved in 300 training iterations, the parameters with the the best logged validation performance are selected. Data augmentation is also used during training, namely random crops (4 pixels padding and 32x32 crops) and random horizontal reflection.
E ABLATION EXPERIMENTS
E. BAYESIAN OPTIMISATION
We evaluate our search strategy SNT.1 (Eq. 10 which is used for non-perceptual transfer. For this purpose, we create a new sequence:
gin+
[VT
V
v 1s Vio (Da, ha, 91
20

---

Under review as a conference paper at ICLR 2023
Comperison with rancom search
0.45
0.90
0.70
12_1
0.65
B0
12
16
svollcted patlis
Figure 5: Comparing to randomly selected initial points (BOJ12 rnd rnd_nnit), and random search (RS).
which involves all 16 upper labelling sub-functions g of BELL. The last problem's 's dataset is gener- ated according to the triple nu = (10000, All,, 30), which states that only 30 out of the 64 possible two-dimensional patterns are represented in the dataset. As result, non-perceptual transfer is nec- essary order maximise the performance on the final problem. We create realisations of gin+ with different randomly selected problems.
Even though our method is deterministic, the baselines we compare to involve randomly selecting 2 or all random paths. For each such baseline, and each of the 5 realisations of the sequence Sin+, we run the baseline 10 times with 10 different random seeds. This results in 10 5 = 50 evaluations per method which we average
when reporting its performance. For each method, plot its maximum accuracy achieved per number paths evaluated.
NT.I We compare our search strategy $BO (which we denote BO_12) to an augmented version which randomly selects the initial 2 paths (BO 12.r), and random search baseline which recommends paths a random order (RS). The results, shown in Fig. 5 demonstrate our approach's is superior anytime performance.
E.2 RANDOM PROJECTIONS
The usage of SG (Eq. 5) relies approximations of the training input distributions of pre-trained modules. Instead of modelling a module's training input distribution directly, we proposed to first project samples from it to & dimensions using random projection, and then model the resulting distribution with a multivariate Gaussian. this section we would like to evaluate three aspects of this approach. First, would like to assess the usefulness of the resulting approximations for the purposes of selecting the correct input distribution. Second, we would like to assess the sensitivity of our approximations the hyperparameter k. Third, we would like compare our approach to Gaussian approximation of the original input space in order to determine whether we sacrifice performance.
To this end, we evaluate whether our approach is useful for distinguishing between set of input distributions. We compare the approximations resulting from different choices {10, 20, 40). The resulting methods are referred to as rp 10, rp 20 and rp 40 respectively. Moreover, we compare the method of computing Gaussian approximation of a module's 's training input distribution, without a random projection. Since this can lead to singular covariance matrix, we make use of diagonal loading (Draper & Smith 1998) which we add a small constant (10-8) 8) to the diagonal of the computed sample covariance matrix in order to make it positive definite. We refer to the resulting method as diag loading
We compare how well can these approaches distinguish between the 9 image datasets used in BELL. We chose to use the input images for our comparison since they have the highest dimension and should be the most diffieult approximate.
To evaluate one of the methods, we first use it to approximate all 9 input distributions using N data points, resulting in approximations, denoted as {q Second, for each input distribution descending
Pj. we sample 100 different data points and use them to order the approximations order of their likelihood. Ideally, if the data points sampled from the j-th distribution Pj. the
21

---

Under review as a conference paper at ICLR 2023
3.0
12 40
dagg lcading
25
loading
12_ 2.20
2.
19_10
average position
2.-
rp 10
15
15-
1.0
0.-
00
1000
10000
20000
30000
40000
training data points
# training data points
(a)
(b)
Figure 6: Comparison of different methods for modelling module's ' input distribution. The axis represents the number N of data points used to compute an approximation. The y axis represents the average position, as defined in the main text, which indicates how well a method can approximate the distributions. The lower the average position is, the better the model performs. Figure a) presents a plot across all choices of N. Figure b) focuses on the first few values of N.
corresponding approximation qy should have the highest likelihood, and thus should be the first in the list, i.c. should have index equal to O. We compute the index of qj in the ordered list and use as an indication how successfully the method has approximated pz- We compute this index for each of the 9 distributions and report the average index, also referred to the average position.
We evaluate each method for different choices of N, N = {50, 100, 500, 1000, 5000, 10000, 20000, 30000, 44000}. Moreover, repeat all evaluations times using different random seeds and report the mean and standard error of the average position. The results are reported in Fig. (6]
Our results show that directly modelling the original distribution with Gaussian leads sub- optimal performance. On the other hand, we observe that for 2 500, the methods which use random projection can always match the given data points with the correct distribution which they were sampled for. Surprisingly, for N = 50 and N = 100, loading outperforms the other methods and can successfully identify the correct distribution of the given data points. Furthermore, we observe that for these values of N, decreasing the dimension is that the data points are projected leads better performance of the methods that are based on random projection.
Overall, our results suggest that the approximations which we use for S are effective when the new modules are trained on more than 100 data points. This seems like reasonable requirement, as fewer points are likely to result in a sub-optimal performance.
22


Reviews:
[{"review_id": "SrlVtkg-pO9", "reviewer": "Reviewer_ifu1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a probabilistic framework to model modular architectures for the problem of continual learning (CL). They divide the problem of choosing modules in two parts: perceptual transfer (PT), and non-perceptual transfer (NT). In PT, the first l layers are assumed to be pre-trained and the model must choose between those modules to maximize performance when training new modules on top. NT works the other way around. From this, they derive 3 CL algorithms: (i) PeCL, which does perceptual and few-shot transfer, (ii) NoCL, which does non-perceptual transfer, and (iii) CCL, which combines both strategies. To evaluate their model, the authors introduce a compositional version of CTrL and evaluate plasticity, backward transfer, forward transfer, perceptual transfer, few-shot transfer, and non-perceptual transfer. In experiments they show that their models perform favourably compared to SA, MCL-RS, HOUDINI, MNTDP.", "summary_of_the_review": "The proposed method is sound and it tackles an interesting problem for continual learning. However, the lack of clarity of the text makes it difficult to properly asses the quality of this work. In its current form, this work does not meet the standards of ICLR.", "strength_and_weaknesses": "Strengths\n=======\n* The method is sound and it tries to approach modularity from first principles with a divide-and-conquer approach\n* The proposed method outperforms MNTDP\n\nWeaknesses\n=========\n* The model description is not clear, which makes it difficult to understand. Overall the submission looks unpolished. I highly encourage the authors to carefuly rework sections 3, 4, and 5, to make it clearer.\n* The authors cite LMC but do not compare with it, is there any reason?\n* The divide-and-conquer approach has a clear disadvantage: it introduces the additional complexity of establishing where to split the preceptual part from the non-perceptual part. Maybe the authors could include a limitations section to talk about that.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SlQl73QV6lp", "reviewer": "Reviewer_LPZE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends prior work on Continual Learning in two directions: 1) It introduces new learning sequences designed to evaluate non-perceptual transfer (transfer the last l layers) and few-shot transfer (transfer all layers) as well as more challenging sequences for the evaluation of perpetual transfer (transfer the first l layers). 2) Building on modular approaches to CL, e.g., (Venital et al. 2020), a probabilistic framework is proposed to tackle the exponential space of module combinations. One probabilistic model is proposed for perceptual (and few-shot) transfer and a separate model is proposed for non-perceptual transfer.\n\nExperiments compare the proposed approach to some recent methods and the results are somewhat favorable to the proposed method.\n", "summary_of_the_review": "The paper’s contribution is limited as outlined above and the quality of the presentation is very poor.", "strength_and_weaknesses": "## Strengths\n\n**S1.** The paper identifies limitations of prior work and seeks to address them.\n\n## Weaknesses\n\n**W1.** The contributions of the submission are very incremental. I see it as an extension of the work of (Venital et al. 2020) in the two directions outlined in the summary. Further, the theoretical contributions are obfuscated by the rough presentation (more on this below) and the validation is, in part, designed for the proposed method (i.e., some of the evaluation sequences are designed to not be handled properly by prior work).\n\n**W2.** The presentation has many issues:\n\n1. often symbols are introduced without definition and sometimes never get defined, e.g., Z in eq. (10), d in eq. (10) does not match the d previously defined; A at the end of page 4;\n\n2. evaluation metrics like those in Table 1 are never defined; etc.\n\n3. some explanations do not make much sense, e.g., “gives preference to modules which helped achieve a higher accuracy, on the problems which they have been trained on,” “minimum number of transfer which need to be transferred,” “can be explored by applying this search strategy to each sequentially,” etc.\n\n4. the probabilistic models proposed are not explained in detail, e.g., eq. (5) is simply stated with no explanation and the Gaussian Process in section 5.1 is not even specified.\n\n**W3.** The paper disregards other benchmarks in CL like (Lin et al. NeurIPS 2021.) Such a benchmark should be discussed, and results on it should either be included or there should be a discussion explaining why those would not be relevant.\n\n## References\n\nLin et al. The CLEAR Benchmark: Continual LEArning on Real-World Imagery. NeurIPS 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Rdt30v_NJk", "reviewer": "Reviewer_E5fq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a probabilistic framework for modular continual learning.\nThey built on the idea in [1] that modules can be chosen locally based on their input distribution.\nContrarily to [1], this paper opts for a Bayesian treatment of the module composition problem.\n\nThe paper proposes two transfer learning settings for compositionality: perceptual vs non-perceptual transfer.\nIn the former, we assume that most of the drift in p(x,y) happens in p(x), whereas the in the latter, it occurs in p(y|x).\nThey prescribe one probabilistic method for both settings.\n\nSome experiments in CTRL [2] and a modified version show that the method can outperform some baselines.\n\n[1] Continual Learning via Local Module Composition. Oleksiy Ostapenko, Pau Rodríguez, Massimo Caccia, Laurent Charlin\n[2] Efficient Continual Learning with Modular Networks and Task-Driven Priors. Tom Veniat, Ludovic Denoyer, Marc'Aurelio Ranzato", "summary_of_the_review": "I think the paper is worthy of a publication if the weakness mentioned above are addressed. ", "strength_and_weaknesses": "**Strengths**\n\nThe proposed method is sensible and adequately scales w.r.t. the number of training modules, which was a limitation of [1].\nThis work can inspire future work in continual learning.\n\n**Weaknesses**\n\nI wish the paper offered a more profound contrast with [1], both technically and empirically.\n\nThe idea of using the modules' input distribution as a signal for activating the modules was proposed in [1], but section 4 does not acknowledge that. Section 4 should explain how previous work has solved the problem above and then explain how their Bayesian treatment relates. The Section should further explain the pros and cons of such treatment compared to the previous solution.\n\nThe paper could instead (or additionally) benchmark their method against [1], which is the most relevant baseline for the paper.\nI understand that running new experiments during rebuttal is not ideal. \nI'm not expecting the authors to do that.\nAt least the others might want to add the reported results of [1] for the CTRL benchmark while mentioning that [1] operates in a much more challenging setting, i.e., **class**-incremental learning and not task-incremental.\n\n  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NM-8OR716yz", "reviewer": "Reviewer_ARaq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the idea of doing inference over a set of modules such that the resulting model can generalize as well as adapt quickly. Different approaches differ as to how they do search over an ensemble of modules. The proposed paper deals with both \"perceptual\" transfer and \"non-perceptual\" transfer. They introduce a probabilistic framework for selection of pre-trained modules. The key idea for driving the selection (i.e., which module to select) is  how well the module can transform the input. Each module is parameterized as a neural network. The paper define a posterior and prior, where the posterior is a function of how well the module can explain the input, and prior gives preference to modules which help achieve a higher accuracy. ", "summary_of_the_review": "The paper proposes a probabilistic framework to do inference over an ensemble of modules.", "strength_and_weaknesses": "Strengths:\n\n-  The paper tackles an important problem i.e., exploring better ways of searching through possible combinations of modules.\n\nWeaknesses:\n\n- It would be useful to study how the proposed method scales as a function of number of modules.\n- It would also be useful how the proposed method compares to methods which does a \"soft\" selection of the output of modules (like in LMC). \n- The paper seems a bit hard to parse. It may be useful to give a high level \"overview\"of the proposed method before jumping to describe how the paper works. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject