A Unified Bayesian Framework for Discriminative and Generative Continual Learning
Authors: Abhishek Kumar
Sunabha Chatterjee
Piyush Rai
Venue:  (2021)
Keywords: continual learning
bayesian learning

Abstract: Continual Learning is a learning paradigm where learning systems are trained on a sequence of tasks. The goal here is to perform well on the current task without suffering from a performance drop on the previous tasks. Two notable directions among the recent advances in continual learning with neural networks are (1) variational Bayes based regularization by learning priors from previous tasks, and, (2) learning the structure of deep networks to adapt to new tasks. So far, these two approaches have been orthogonal. We present a novel Bayesian framework for continual learning based on learning the structure of deep neural networks, addressing the shortcomings of both these approaches. The proposed framework learns the deep structure for each task by learning which weights to be used, and supports inter-task transfer through the overlapping of different sparse subsets of weights learned by different tasks. An appealing aspect of our proposed continual learning framework is that it is applicable to both discriminative (supervised) and generative (unsupervised) settings. Experimental results on supervised and unsupervised benchmarks shows that our model performs comparably or better than recent advances in continual learning.

Under review as a conference paper at ICLR 2021
A UNIFIED BAYESIAN FRAMEWORK FOR DISCRIMI- NATIVE AND GENERATIVE CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Continual Learning is a learning paradigm where learning systems are trained on sequence of tasks. The goal here is to perform well on the current task without suffering from a performance drop on the previous tasks. Two notable directions among the recent advances in continual learning with neural networks are (1) vari- ational Bayes based regularization by learning priors from previous tasks, and, (2) learning the structure of deep networks to adapt to new tasks. So far, these two approaches have been orthogonal. We present a novel Bayesian framework for continual learning based on learning the structure of deep neural networks, ad- dressing the shortcomings of both these approaches. The proposed framework learns the deep structure for each task by learning which weights to be used, and supports inter-task transfer through the overlapping of different sparse subsets of weights learned by different tasks. An appealing aspect of our proposed continual learning framework is that applicable both discriminative (supervised) and generative (unsupervised) settings. Experimental results on supervised and unsu- pervised benchmarks shows that our model performs comparably or better than recent advances in continual learning.
INTRODUCTION
Parisi et al. 2019) is the learning paradigm where a single model subjected to a sequence of tasks. At any point of time, the model expected to (i) make predictions for the tasks it has seen so far, (íz) if subjected to training data for new task, adapt to the new task leveraging the past knowledge possible (forward transfer) and benefit the previous tasks if possible (backward transfer). While the desirable aspects of more mainstream transfer learning (sharing of bias between related tasks (Pan & Yang 2009)) might reasonably be expected here too, the principal challenge is to retain the predictive power for the older tasks even after learning new tasks, thus avoiding the so-called f
Continual learning (CL) RRng 1997:
Real world applications in, for example, robotics time-series forecasting, rife with this chal- lenging learning scenario, the ability to adapt to dynamically changing environments or evolving data distributions being essential in these domains. Continual learning is also desirable in unsuper- vised learning problems as well (Smith et al. 2019 Rao et al. (2019b) where the goal is to learn the underlying structure or latent representation of the data. Also, as skill innate humans (Flesch) et al.) 2018). it is naturally an interesting scientific problem to reproduce the same capability in artificial predictive modelling systems.
Existing approaches to continual learning are mainly based on three foundational ideas. One of them is to constrain the parameter values to not deviate sigmificantly from their previously learned value by using some form of regularization or trade-off between previous and new learned weights
(Schwarz et al. 2018 Kirkpatrick et al. 2017 Zenke et al., 2017, Lee et al. 2017). A natural way to accomplish this is to train a model using online Bayesian inference, whereby the posterior of the parameters learned from task t serve as the prior for task t + 1 as in Nguyen et al. (2018) and Zeno et al. (2018). This new informed prior helps in the forward transfer, and also prevents catastrophic forgetting by penalizing large deviations from itself. In particular, VCL (Nguyen et al. 2018) achieves the state of the art results by applying this simple idea to Bayesian neural networks. The second idea is to perform an incremental model selection for every new task. For neural net- works, this is done by evolving the structure newer tasks encountered (Golkar al. 2019 Li

---

Under review as a conference paper at ICLR 2021
et al. 2019). Structural learning is very sensible direction in continual learning require a different network structure than old unrelated tasks and even if the tasks are highly related their lower layer representations can be very different. Another advantage of structural learning is that while retaining a shared set of parameters (which can be used to model task relationships) it also allow task-specifie parameters that increase the performance of the new task while avoiding catastrophic forgetting caused due to forced sharing of parameters. The third idea is to invoke a form of 'replay', whereby selected or generated samples representative of previous tasks, are used to retrain the model after new tasks are learned.
task may
In this work, we introduce a novel Bayesian nonparametric approach to continual learning that seeks to incorporate the ability of structure learning into the simple yet effective framework of online Bayes, particular, our approach models each hidden layer of the neural network using the Indian Buffet Process (Griffiths & Ghahramani] (011) prior, which enables us to learn the network structure as new tasks arrive continually. We can leverage the fact that any particular task & uses a sparse subset of the connections of neural network N,, and different related tasks share different subsets (albeit possibly overlapping). Thus, in the setting of continual learning, it would be more effective if the network could accommodate changes in its connections dynamically to adapt to newly arriving task. Moreover, in our model, we perform the automatic model selection where each task can select the number of nodes in each hidden layer. All this is done under the principled framework of variational Bayes and a nonparametric Bayesian modeling paradigm.
Another appealing aspect of approach is that in contrast some of the recent state-of-the- continual learning models (Yoon et al., 2018 Li et al. (019) that are specific to supervised learning problems, our approach applies to both deep discriminative networks (supervised learning) where each task can be modeled by a Bayesian neural network (Neal) Blundell deep generative networks (unsupervised learning) where each task can be modeled by a variational autoencoder (VAE) (Kingma & Welling 2013).
Blundell et al. (013)), as well as
2 PRELIMINARIES
Bayesian neural networks (Neal 2012) are discriminative models where the goal is to model the relationship between inputs and outputs via deep neural network with parameters w. The network parameters are assumed to have prior p(w) and the goal is to infer the posterior given the observed data D. The exact posterior inference is intractable in such models. One such approximate inference scheme is Bayes-by- by-Backrrop (Blundell et al.] 2015) that uses mean-field variational posterior ((w) over the weights. Reparameterized samples from this posterior are then used to approximate the lower bound via Monte Carlo sampling. Our goal in the continual learning setting is to learn such Bayesian neural networks for a sequence of tasks by inferring the posterior qe (w) for each task f, without forgetting the information contained in the posteriors of previous tasks.
Variational autoencoders (Kingma+ & Welling 2013) are generative models where the goal is to model a set of inputs {x), in terms of a stochastic latent variables {=] The mapping from each H., to xr is defined by a generator/decoder model (modeled by a deep neural network with pa- rameters 8) and the reverse mapping is defined by recognition/encoder model (modeled by another deep neural network with parameters Φ). Inference in VAES is done by maximizing the variational lower bound on the marginal likelihood. It is customary do point estimation for decoder parame- ters I and posterior inference for encoder parameters é. However, in the continual learning setting, it would be more desirable infer the full posterior qe ((ww) for each task's encoder and decoder parameters w (0, Φ), while not forgetting the information about the previous tasks as more and more tasks are observed. Our proposed continual learning framework address this aspect as well.
Variational Continual Learning (VCL) Nguyen et al. (2018) is a recently proposed approach to continual learning that combats catastrophic forgetting neural networks by modeling the network parameters w in a Bayesian fashion and by setting Pe(w) = q- (w)) that is, a task reuses the previous task's 's posterior as its prior. VCL solves the follow KL divergence minimization problem
qs (w) arg min KL. (<(w))] - Z, (
q
While offering a principled way that is applicable to both supervised (discriminative) and unsuper- vised (generative) learning settings, VCL assumes that the model structure is held fixed throughout,
2

---

Under review as a conference paper at ICLR 2021
R
4M
(a) Illustration on single hidden layer
(b) ())iierimiiattive Model (ii) Generative Model (VAE)
Figure I: (a) Evolution of network structure for 3 consecutive tasks. Weights used by a task are denoted with respective colors. Note that there be overlapping of structure between tasks. (b) Schematics representing our models, Os are parameters shared across all task, l² are task specific mask parameters, layer separate head parameters. In our exposition, we collectively denote these parameters by W = B O with the masks being B and other parameters being V.
0² are last
which can be limiting in continual learning where the number tasks and their complexity is usually unknown beforehand. This necessitates adaptively inferring the model structure, that can potentially adapt with each incoming task. Another limitation of VCL is that the unsupervised version, based on performing CL on VAEs, only does so for the decoder model's parameters (shared by all tasks). It uses completely task-specific encoders and, consequently, is unable to transfer information across tasks in the encoder model. Our approach addresses both these limitations in a principled manner.
BAYESIAN STRUCTURE ADAPTATION FOR CONTINUAL LEARNING
In this section, we present a Bayesian model for continual learning that can potentially grow and adapt its structure as more and more tasks arrive. Our model extends seamlessly for unsupervised learning as well. For brevity of exposition, in this section, we mainly focus on the supervised setting where a task has labeled data with known task identities (task-incremental), We then briefly discuss the unsupervised extension (based on VAEs) in Sec. 3.3 where task boundaries may or may not (task- agnostic) be available and provide further details in the appendix (See [I.
Our approach uses a basic primitive that models each hidden layer using nonparametric Bayesian prior (Fig. Ta shows an illustration and Fig. 16 shows a schematic diagram). We can use these hidden layers to model feedforward connections in Bayesian neural networks or VAE models. For simplicity, we will assume a single hidden layer, the first task activates as many hidden nodes as required and learns the posterior over the subset of edge weights incident each active node. Each subsequent task reuses some of the edges learned by the previous task and uses the posterior over the weights learned by the previous task the prior. Additionally, it may activate some new nodes and learn the posterior over some of their incident edges. It thus Iearns the posterior over subset of weights that may overlap with weights learned by previous tasks. While making predictions, task uses only the connections it has learned. More slack for later tasks in terms of model size (allowing it to create new nodes) indirectly lets the task learn better without deviating too much from the prior (in this case, posterior of the previous tasks) and further reduces chances of catastrophic forgettn Kirkpatrick et al. 2017).
3.1 GENERATIVE STORY.
Omitting the task id é for brevity, consider modeling 9 task using neural network having L hidden layers. We model the weights in layer as W B' o V a point-wise multiplication of a real- valued matrix V (with Gaussian prior [N0, o each entry) and task-specific binary matrix B'. This ensures sparse connection weights between the layers. Moreover, we model B ~ IBP(a.) using the Indian Buffet Process (IBP) Griffiths & Ghahramani (201T) prior, where the hyperparam- eter a controls the number of nonzero columns in B and its sparsity. The IBP prior thus enables learning the size of B' (and consequently of V') from data. As result, the number of nodes in the hidden layer is learned adaptively from data. The output layer weights are denoted as W Wout with each weight having a Gaussian prior N (0, ).) The outputs are
Yn Lik(Wo.. (N
1,...,N N
(2)
Here N is the function computed (using parameter samples) up to the last hidden layer of the network thus formed, and Lik denotes the likelihood model for the outputs.
3

---

Under review as a conference paper at ICLR 2021
Similar priors on the network weights have been used in other recent works to learn sparse deep neural networks (Panousis et al., 2019 Xu et al., 2019). However, these works assume a single focus here is leverage such priors in the continual learning
task to be learned. In contrast, setting where we need to learn a sequence of tasks while avoiding the problem of catastrophic forgetting. Henceforth, we further suppress the superseript denoting layer number from the notation for simplicity; the discussion will hold identically for all hidden layers. When adapting to new task, the posterior of V learned from previous tasks is used as the prior. A new B learned afresh, to ensure that a task only learns the subset of weights relevant to it.
Stick Breaking Construction. As described before, adaptively infer the number of nodes in each hidden layer, we use the IBP prior (Griffiths & Ghahramani, [201T). whose truncated stick-breaking process (Doshi al. 2009) construction for each entry 13 is as follows
Vk Beta(a, 1), TT& II V/: Ba Bernoulli(TT) 1=1
for E 1,.., D. where D denotes the number of input nodes for this hidden layer, and E 1,2, K, where K is the truncation level and a controls the effective value of K, i.e., the number of activie hidden nodes. Note that the prior probability TR of weights incident on hidden node being nonzero decreases monotonically with k, until, say, K nodes, after which no further nodes have any incoming edges with nonzero weights from the previous layer, which amounts to them being turned off from the structure. Moreover, due the cumulative product based construction of the π& 's, an implicit ordering is imposed on the nodes being used. This ordering is preserved across tasks, and allocation of nodes to task follows this, facilitating reuse of weights.
The truncated stick-breaking approximation is a practically plausible and intuitive solution for con- tinual learning since fundamental tenet of continual learning is that the model complexity should not increase in an unbounded manner as more tasks are encountered. Suppose we fix a budget on the maximum allowed size of the network (no. hidden nodes in layer) after it has seen, say, T tasks. Which exactly corresponds to the truncation level for each layer. Then for each task, nodes are allocated conservatively from this total budget, fixed order, conveniently controlled by the hyperparameter. In appendix (Sec. D we also discuss dynamic expansion scheme that avoids specifying truncation level (and provide experimental results).
3.2 INFERENCE
Exact inference is intractable in this model due to non-conjugacy. Therefore, resort to the vari- ational inference (Blei et al., (017).. We employ structured mean fiend approximation (Hoffmman) & Blei, 2015), which performs better than normally used mean-field approximation, mer captures the dependencies in the approximate posterior distributions of B and v. In partic- ular, we use qVV.B, v) = gg())( B ((u))(v), where, v(V) = -
the for-
(VV.. oa.
mean field Gaussian approximation for network weights. Corresponding to the Beta- Bernoulli hierarchy of (3), we use the conditionally factorized variational posterior family, that is, (B(o) n2_ IT Bern(Baxx0aar
where 02, opaa + logit(Tx)) and q(v)
II Beta uk M 2). Thus we have θ = {UK,1 V/.2, {pa,k Pa. P 1 1K as set of learn- able variational parameters.
Each column of B represents the binary mask for the weights incident to particular node. Note that although these binary variables (in single column of B) share common prior, the posterior for each of these variables are different, thereby allowing a task to selectively choose a subset of the weights, with the common prior controlling the degree of sparsity.
L= E (n. B,v) KL(A(V, B, B) B,
(4)
1 C= - D Is(v' - KLL - - (5)
Bayes-by-backprop Blundell et al. 2015)> is common choice for performing variational inference in this context. Eq. defines the Evidence Lower Bound (ELBO) in terms of detendepnnd likeli- hood and data-independent KL. terms which further gets decomposed using mean-field factorization.
4

---

Under review as a conference paper at ICLR 2021
The expectation terms optimized by unbiased gradients from the respective posteriors. All the KL divergence terms in (Eq. [5) have closed form expressions; hence using them directly rather than estimating them from Monte Carlo samples alleviates the approximation error well as the computational overhead, to some extent. The log-likelihood term can be decomposed as
F(V,B. log Lik(Y B, v) log Lik Y W WWW.WNNN (X; B,
(6)
where (x, Y) is the training data. For regression, Lik can be Gaussian with some noise variance, while for classification it can Bernoulli with a probit or logistic link. Details of sampling gradient computation for terms involving beta and Bernoulli r.v's is provided in the appendix. (Sec. F).
3.3 UNSUPERVISED CONTINUAL LEARNING
Our discussion thus far has primarily focused on continual learning where each task super- vised learning problem. Our framework however readily extends to unsupervised continual learn- ing (Nguyen et al. 2018 [Smith et al. 2019, Rao et al., 2019b) where we assume that each task involves learning deep generative model, commonly a VAE. In this case, each input observation In has an associated latent variable 2,. Collectively denoting all inputs as and all latent variables as Z, we can define ELBO similar to Eq. 4 aas
L Z VV,(ZVV,,,i) [m (( Z, V. B, v)] KL(g B, o)ll( D.)l://..V.BB.ll/...V.. B.))
Note that, unlike the supervised case, the above ELBO also involves an expectation over Z. Similar to Eq. 5 this can be approximated using Monte Carlo samples, where each 2,, is sampled from the amortized posterior (z,, V. B, v. Z.,.) In addition to learning the model size adaptively, as shown in the schematic diagram (Fig. 1b (ii)), our model learns shared weights and task-specifi masks for encoder and decoder models. contrast, VCL uses fixed-sized model with entirely task task-specific encoders, which prevents knowledge transfer across the different encoders.
3.4 OTHER KEY CONSIDERATIONS
Task Agnostic Setting Our framework extends to task-agnostic continual learning as well where the task boundaries are unknown. Based on Lee et al. (2020), we use gating mechanism (Eq. with t, represents the task identity of th sample v, and define marginal log likelihood
K
ptt, = kk(x)) = p(xnitn = k)p(tn = k)
Σ
p(an = k)p(t, = k)
(8)
log (()) = k) ix t K0)) + KL (q(t w)ppp( x|
(9)
where, g(l k) is the variational posterior over task identity. Similar E-step in Expectation
Maximization MMoon 1996). we can reduce the KL-Divergence term to zero and get the M-step as
arg max log p( X) arg max Bp(t=k) Bm) log ( k)
(10)
Here, log k) is intractable but can be replaced with its variational lower bound (Eq. 7 We use Monte Carlo sampling for approximating p(z, t., k)). Detecting samples from new task done using a threshold (Rao al., 2019a) the evidence lower bound (Appendix Sec.
Masked Priors Using the previous task's posterior the prior for current task (Nguyen et al. 2018) may introduce undesired regularization in case of partially trained parameters that do not contribute previous tasks and may promote catastrophic forgetting. Also, the choice of the initial prior as Gaussian leads to creation of more nodes than required due to regularization. To address this, mask the new prior for the next task & with the initial prior pt defined as
(11) where B is the overall combined mask from all previously learned tasks i.e., (B' "... 9t-1, Pr are the previous posterior and current prior, respectively, and po is the prior used for the first task. The standard choice of initial prior Po can be uniform distribution.
Pe (Vax) P'A49-1(Va.. (1 - B2, Po Va,k)
5

---

Under review as a conference paper at ICLR 2021
4 RELATED WORK
One of the key challenges in continual learning is to prevent catastrophic forgetting, typically ad- dressed through regularization of the parameter updates, preventing them from drastically changing from the value learnt from the previous task(s). Notable methods based on this strategy include EwC (Kirkpatrick al.) 2017). SI (Zenke et al. 2017). LP (Smola et al. (003). etc. Superseding these methods is the Bayesian approach, a natural remedy of catastrophic forgetting in that, for any task, the posterior of the model learnt from the previous task serves as the prior for the current task, which is the canonical online Bayes. This approach is used in recent works like VCL (Nguyen et al.) 2018) and task agnostic variational Bayes (Zeno et al.) 2018 for learning Bayesian neural networks in the CL setting. Our work is most similar in spirit to and builds upon this body of work.
Another key aspect in CL methods is replay, where some samples from previous tasks are used to fine-tune the model after learning new task (thus refreshing its memory in some sense and avoiding catastrophic forgetting). Some of the works using this idea include Lopez- et al. (2017). which solves constrained optimization problem at each task, the constraint being that the loss should decrease monotonically on a heuristically selected replay buffer; Hu et al. (2019), which uses a partially shared parameter space for inter-task transfer and generates the replay samples through a data-generative module; and Titsias et al. (020), which learns a Gaussian process for each task, with a shared mean function in the form a feedforward neural network, the replay buffer being the set of inducing points typically used to speed up GP inference. For VCL and our work, the coreset serves as a replay buffer (Appx. C): but we emphasize that it is not the primary mechanism to overcome catastrophic forgetting in these cases, but rather an additional mechanism to preventing it.
Recent work in CL has investigated allowing the structure of the model to dynamically change with newly arriving tasks. Among these, strong evidence in support of our assumptions can be found in Golkar et al. (2019), which also learns different sparse subsets of the weights of each layer of the network for different tasks. The spansity is enforced by a combination of weighted L₁ regularization and threshold baassed pruning. There are also methods that do not learn subset of weights but rather learn the subset of hidden layer nodes to be used for each task; such strategy is adopted by either using Evolutionary Algorithms to select the node subsets (Fernando et al. 2017) or by training the network with task embedding based attention masks (Serrà al. 2018). One recent approach (Adel eet al.) (2020), instead of using binary masks, tries to adapt network weights at different scales for different tasks; it is also designed only for discriminative tasks.
Among other related work, Li et al. (2019): Yoon et al. (2018); Xu & Zhu (2018) either reuse the pa- rameters of layer, dynamically grows the size of the hidden layer, or spawn new set of parameters (the model complexity being bounded through regularization terms or reward based reinforcements), Most of these approaches however tend be rather expensive and rely on techniques, such as neu- ral architecture search. In another recent work (simultaneous development with our work), Kessler jet al.] (2020) did a preliminary investigation on using IBP for continual learning. They however use IBP on hidden layer activations instead of weights (which they mention is worth considering), do not consider issues such as the ones we discussed in Sec. Scc.3.4 and only applics to supervised setting. Modelling number active nodes for given task has also been explored by (Serr et al. (018): Fer- nando et al. (2017): Ahn et al. 2019), but modelling posterior over connections weights between these nodes achieves more sparsity and flexibility in terms of structural learning at the cost of in- creased number of parameters, von Oswald et al. (2020) tries to amortize the network parameters directly from input samples which a promising direction and can be adapted for future research.
For non-stationary data, online variational Bayes is not directly applicable as assumes indepen- dently and identically distributed (i.i.d.) data. As a result of which the variance in Gaussian posterior approximation will shrink with an increase in the size of training data, Kurle et al. (2020) proposed use of Bayesian forgetting, which can be naturally applied to our approach enabling it to work with non-stationary data but it requires some modifications for task- setup. In this work, we have not explored this extension keeping it as future work.
5 EXPERIMENTS
We perform experiments on both supervised and unsupervised CL and compare our method with relevant state-of-the-art methods. In addition to the quantitative faccuracy/log-likeliin compar-

---

Under review as a conference paper at ICLR 2021
isons) and qualitative (generation) results, also examine the network structures learned by model. Some of the details (c.g., experimental settings) have been moved to the appendix
Spilt
Split
66.2
07.5
Tasks
Tasks
lasks
Tasks
Tasks
Nabre
Rebercal
EwC
VCL(nomeet]
(MM(mode)
nurs/coreett
DTN
Figure 2: Mean accuracies of tasks seen far as tasks observed on multiple benchmarks
5.1 SUPERVISED CONTINUAL LEARNING
We first evaluate our model standard supervised CL benchmarks. We experiment with different existing approaches such as, Pure Rehearsal (Robins) 1995). EwC (Kirkpatrick et al.] (017). IMM (Lee al., 2017), DEN (Yoon al.] 2018), RCL (Xu & Zhu 2018), and "Naïve" which learns a shared model for all the tasks. We perform our evaluations on five supervised CL benchmarks: SplitMNIST, Split notMNIST(small), Permuted MNIST, Split fashionMNIST and Split Cifar100. The last layer heads (Appx. E.I were kept separate for each task for fair baseline comparison.
For Split MNIST, Split notMNIST and Split fashionMNIST each dataset split into binary clas- sification tasks. For Split Cifar100 the dataset was split 10 multiclass classification tasks. For Permuted MNIST, each task is multiclass classification problem with fixed random permutation applied to the pixels of every image. We generated 5 such tasks for our experiments.
Task Not Mnist
Network Used 1%)
Sharing (%)
Task Permated Mnist
Network Used (56)
Sharing (%)
.2
43
41
9
10
22
660
60
Tasks
Tasks
Tasks
Tasks
Tasks
Tas
vd((rad)
vcllkcen)
ours
ours(rand)
(urs(kcen)
ours(growing)
Test accuracy variation for first tasks (left), network used (middle) as new tasks are observed and percentage of network connections shared (right) among different tasks in first hidden layer
Figure 3:
Performance evaluation Suppose we have a sequence of T tasks. To gauge the effectiveness of model towards preventing catastrophic forgetting, we report (i) the accuracy of first task after learning each of the subsequent tasks; and (iz) the average test accuracy over all previous tasks 2, / after learning each task I. For fair comparison, baselines (details in Appx.), except for DEN and RCL that grows the structure size. We also report results on some additional CL metrics Diaz-Rodriguez et al. (0188) in the Appx. (Sec. H.4).
the same architecture for each of the
Fig. 2 shows the mean test accuracies on all supervised benchmarks as new tasks are observed. As shown, the average test accuracy of our method (without well as with coresets) is better than the compared baseline (here, we have used random point selection method for coresets). Moreover, accuracy drops much more slowly than and other baselines showing the efficacy of our model in preventing catastrophic forgetting due to the adaptively learned structure. In Fig. 3 we show the accuracy on first task as new tasks arrive and compare specifically with VCL. In this case too, observe that our method yields relatively stable first task accuracies compared to VCL. We note that for permuted MNIST the accuracy of first task increases with training of new tasks which shows the presence of backward transfer, which is another desideratum of CL. We also report the performance with our dynamically growing network variant (for more details refer Appx. Sec. D).
TThe code for model he found this link:

---

Under review as a conference paper at ICLR 2021
Ours Naive
VCL
100
ABCR
2.3 5
1144
DE
ABCPEEG
Figure 4: Sequential generation for MNIST (left) and notMINIS (right) datasets. Here 1th row is generated after observing 1 task (Appendix contains more illustrations and zoomed-in versions)
Structural Observations An appealing aspect of work is that, the results reported above, which are competitive with the -hte--tf- are achieved with very sparse neural network struc- tures learnt by the model, which we analyze qualitatively here (Appendix Sec. H.T shows some examples of network structures learnt by our model).
As shown in Fig. 3 (Network Used) IBP prior concentrates weights very few nodes, and learns sparse structures. Also most newer tasks tend to allocate fewer weights and yet perform well, imply- ing effective forward transfer. Another important observation shown in Fig. is that the weight sharing between similar tasks like notMNIST is a higher than that of non-similar tasks like permuted MNIST, Note that new tasks show higher weight sharing irrespective of similarity, this an artifact induced by IBP (Sec 3.1) which tends to allocate more active weights on upper side of matrix.
We therefore conclude that although new task tend share weights learnt by old tasks, the new connections that it creates are indispensable for its performance. Intuitively, the more unrelated task to previously seen ones, the more new connections it will make, thus reducing negative transfer (an unrelated task adversely affecting other tasks) between tasks.
5.2 UNSUPERVISED CONTINUAL LEARNING
We next evaluate our model on generative tasks under CL setting. For that, we compare our model with existing approaches such as Naive, EwC and VCL. We do not include other methods mentioned in supervised setup as their implementation does not incorporate generative modeling. We perform continual learning experiments for deep generative models using a VAE style network. We consider two datasets, MNIST and notMNIST. For MNIST, the tasks are sequence of single digit generation from to Similarily, for noMMNIST each task one character generation from
Note that,
EWC
Methods
3-NN
5-NN
(MB15 Aug, log thatnood
Nalve
Naive
30.1%
33.1%
-80
Ours
EwC
16.6%
19.5%
VCL
16.0%
19.1%
-90-
Ours
0.37 %
0.40%
Ours (TA) 5.79%
5.32%
CURL (TA) 4.58%
4.35%
0
Figure 5: Avg. liceliodsss (left) for sequential generation (MNIST), confusion matrix (right) representing test samples mapped to each generative task learned in TA (lask-agnostic) setting
Table 1: MNIST K-NN test error rates obtained latent space both task-agnostic and know task setting.
unlike VCL and other baselines where all tasks have separate encoder and shared decoder, as we discuss in Sec. 3.3) our model uses shared encoder for all tasks, but with task-specific masks for each encoder (cf., Fig. Tb (ii)). This enables transfer of knowledge while the task-specific mask
effectively prevent catastrophic forgetting.
Generation: As shown Fig the modeling innovation we introduce for the unsupervised setting, results in much improved log-likelihood on held-out sets. In each individual figure in Fig 4 each row represents generated samples from all previously seen tasks and the current task. We see that the quality of generated samples in does not deteriorate compared other baselines more tasks are encountered. This shows that our model can efficiently perform generative modeling by reusing subset of networks and creating minimal number of nodes for each task.
8

---

Under review as a conference paper at ICLR 2021
Task-Agnostic Learning: Fig shows particular case where nine tasks inferred out of 10 class with high correlation among class and 9 due to visual similarity between them. Since each task uses set of network connection, this result enforees relations based on network sharing. Further the log-likelihood obtained for task-agnostic setting is comparable to model with known task boundaries, suggesting that approach can be used effectively in task-agnostic settings as well.
models ability to model task
Representation Learning: Table I represents the quality of the unsupervisedly learned represen- tation by our unsupervised continual learning approach. For this experiment, we use the learned representations to train a KNN classification model with different K values. We note that despite having task-specific encoders VCL and other baselines fail to learn good latent representation, while the proposed model learns good representations when task boundaries are known and comparable to state-of-the the-art baseline CURL (Rao et al. (019a) under task-agnostic setting.
CONCLUSION
We have successfully unified structure learning in neural networks with their variational inference in the setting of continual learning, demonstrating competitive performance with state-of-the-art models on both discriminative (supervised) and generative (unsupervised) learning problems. In this work, we have experimented with task-incremental continual learning for supervised setup and sequential generation task for unsupervised setting. we believe that our dask-agnostic setup can be extended class- maaseinnrremnttl learning scenario where sample points from set of classes arrives sequentially and model is expected to perform classification over all observed classes. It would also be interesting generalize this idea more sophisticated network architectures such as recurrent or residual neural networks, possibly by also exploring improved approximate inference methods. Few more interesting extensions would be in semi-supervised continual learning and continual learning with non- stationay data. Adapting other sparse Bayesian structure learning methods, e.g. Ghosh al.) (2018) the continual learning setting is also a promising avenue. Adapting the depth of the network is more challenging endeavour that might also be undertaken. We leave these extensions for future work.
REFERENCES
Tameem Adel, Han Zhao, and Richard E. Turner. Continual learning with adaptive weights (claw). In International Conference on Learning Representations, 2020. URL https:// openreview wwtprrvvvee.net//oooneeeee net//forum?ldetillooo2
Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. Uncertainty-based continual learning with adaptive regularization. In Advances in Neural Infor- mation Processing Systems, volume 32, pp. 3392-4402 Curran Associates, Inc.,
2019.
URL https:://proceedi: 2a3adf4bf 152011111111111115151111111 fo517ff
neun ips. ccc/paper/2111//1eee .ddf
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: review for statisti- cians. Journal of the American statistical Association, 112(518):55988777 877, 2017.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. ICML, 2015.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. In International Conference on Learning Representations, 2019.
http://rerreeeeeetoooooradddddda UtR.htttss://ppenrer//
Natalia Díaz Rodríguz,, Vincenzo Lomonaco, David Filliat, and Davide Maltoni. Don't 't forget. there is more than forgetting: new metrics for continual learning. arXiv preprint 1rXiv:1810.13160 2018.
Finale Doshi, Kurt Miller, Jurgen Van Gael, and Yee Whye Teh. Variational inference for the indian buffet process. In AISTATS, pp. 137-144, 2009.

---

Under review as a conference paper at ICLR 2021
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. CoRR, abs/1701.08734, 2017. URL httt:://axxiv.org/aaaa wtt./rggg/ab.//.//// 08734
Timo Flesch, Jan Balaguer, Ronald Dekker, Hamed Nili, and Christopher Summerfield. Comparing continual task learning minds and machines. Proceedings of the National Academy of Sciences, 2018.
Soumya Ghosh, Jiayu Yao, and Finale Doshi-Velez. Structured variational learning of bayesian neural networks with horseshoe priors. ICML, 2018.
Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning. CoRR, Abss/190.04446, 2019. URL httt::::/axxiv.or//aabbab wwwp://////aabaaa/ao/// 1903. 1490b/44764 04476
Teofilo F. Gonzalez. Clustering to minimize the maximum intercluster distance. Theor. Comput. Sci., 38:293-306, 1985.
Thomas L Griffiths and Zoubin Ghahramani. The indian buffet process: An introduction and review. JMLR, 12(Apr):1185-12 1224, 2011.
Matthew Hoffman and David Blei. Stochastic Structured Variational Inference. AISTATS, 38:361- 369, 2015.
Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao, Jinwen Ma, Dongyan Zhao, and Rui Yan. Overcoming catastrophic forgetting via model adaptation. In International Confer- ence on Learning Representations, 2019. URL http:/sss//oneeeei.....eee/////e hwwppeenwwwieew.eeeeeeeeeeee EyGvCoA5YX
Eric Jang. Shixiang Gu, and Ben Poole. Categorical reparameterizzation with gumbel-softmax. 2017. URL httpp:://aaxii.or//////o wtwpgg/abb/l1..../ 01144
Samuel Kessler, Vu Nguyen, Stefan Zohren, and Stephen Roberts. Hierarchical indian buffet neural networks for bayesian continual learning, 2020. URL ptttp:://axx.v.or///bb...
02290
Diederik Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2013.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A al. Overcom-
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):35113352, 2017.
Richard Kurle, Botond Cseke, Alexej Klushyn, Patrick van der Smagt, and Stephan Günnemann. Continual learning with bayesian neural networks for non-stationary data. In International Con- ference on Learning Representations, 2020. URL https: /tttps://openrevieew net. / forum?
id id=SJ1sFPVEDB
Sang-Wou Woo Lee, Jin-Hwa Kim, Jachyun Jun, Jung-Woo Ha, and Byoung- Tak Zhang, Overcoming Catastrophic Forgetting by Incremental Moment Matching. NIPS, Mar 2017.
Soochan Lec, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for fask-free continual learning. In International Conference Learning Representations, 2020.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. ICML, 2019.
David Lopez-Paz et al. Gradient episodic memory for continual learning. In NIPS, pp. 6467-6476, 2017.
Chris J Maddison, Andriy Mnih and Yee whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. ICLR, 2017.
10

---

Under review as a conference paper at ICLR 2021
T. K. Moon. The expectation maxemiiionmmaiiiizaaiii algorithm. IEEE Signal Processing Magazine, 13(6): 47-60, 1996.
Eric Nalisnick and Padhraic Smyth. arXiv:1605.06197 May 2017.
Stick-Breaking Variational Autoencoders.
ICLR, art.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard Turner. Variational continual learning. ICLR, 2018.
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, ((110):345-1350) 2009.
Konstantinos Panousis, Sotirios Chatzis, and Sergios Theodoridis. Nonparametric bayesian deep networks with local competition. In International Conference on Machine Learning, pp- 4980- 4988, 2019.
Genman Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell. Continual unsupervised representation learning. In Advances in Neural Information Processing Systems 32, pp. 7647-7657. Curran Associates, Inc., 2019a. http:///:p/pappe nips. ott.://w..oo.....o..ooooo.ee.eea representation- learning.
cc htppaapr//// /aaber//98110 8981 cont inual www.peuwiusueeeoemm
pdf
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Tch, and Raia Hadsell. Continual unsupervised representation learning. In Advances in Neural Information Processing Systems, pp. (645-76555 2019b.
Mark Ring. Child: first step towards continual learning. Machine Learning, 28(1):77-104, 1997.
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2): 123-146, 1995.
Jonathan Schwarz, Jelena Luketina, Wojciech M. Czarnecki, Agnieszka Brabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress &amp; Compress: A scalable framework for continual learning. ICML, May 2018.
Joan Serrà, Didac Surís, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. CoRR, abs/1801.00 001122,,,, 2018.
James Smith, Seth Baer, Zsolt Kira, and Constantine Dovrolis. Unsupervised continual learning and self-taught associative memory hierarchies. ICLR, 2019.
Alexander Smola, Vishy Vishwanathan, and Eleazar Eskin. Laplace propagation. In NIPS, pp- 441-448, 2003.
Michalis Titsias, Jonathan Schwarz, Alexander G G Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for continual learning using gaussian processes. ICIR, 2020.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. Continual learning with hypernetworks. In International Conference on Learning Representations, 2020.
Ju Xu and Zhanxing Zhu. Reinforced Continual Learning. NIPS, art. arXiv: 12369, May 2018.
Kai Xu, Akash Srivastava, and Charles Sutton. Variational russian roulette for deep bayesian non- parametries. In International Conference on Machine Learning, pp. 6963-6972, 2019.

---

Under review as a conference paper at ICLR 2021
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018. URL
httpps://opeureview.nee net/ http://aawaaaee.....oooooaaaaaa. ==
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML, pp. 3987-3995. JMLR. org, 2017.
Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using 10123, 2018.
online variational bayes. arXiv preprint
Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. WHAL: Weibull hybrid autoencoding
inference for deep topic modeling. In International Conference Learning Representations, 2018.
12

---

Under review as a conference paper at ICLR 2021
A
DATA
The data sets used in our experiments with train test split information are listed in table given below. MNIST dataset comprises 28 28 monochromatic images consisting of handwritten digits from to 9. notMNIST dataset comprises of glyph's 's of letters A to in different fonts formats with similar configuration as MNIST. fashion MNIST is also monochromatic comprising of 10 classes (T-shirt, Trouser, Pullover, Dress, Coat, Sandal, Shirt, Sneaker, Bag, Ankle boot) with similar to MNIST. C0 dataset contains RGB images with 600 images per class.
Dataset
Classes
Training size
Test size
MNIST
10
60000
10000
notMNIST
10
14974
3750
fashionMNIST
50000
20000
Cifar100
100
50000
10000
B MODEL CONFIGURATIONS
For permuted MNIST, split MNIST, split notMNIST and fashion MNIST experiments, we use fixed architecture of network for all the models with single hidden layer of 200 units except for DEN (which grows structure dynamically) which used two hidden layers initialized to 256, 128 units.
The VCL implementation was taken from its official repository at https: / //g1 huub. learning For DEN we used the of- IMM imple- BaM_MM_eeesooooooo cow wttp://///o......../...
ficial implementation https: httpps//gg.chbb.com///eeeoono.
mentation was taken from httppp://ggttubbb.om////jj//j/a1
RCL implementation was taken from https:
Reinfforead-CCntttnnal----errr http::://gtthbbb.cmm//o/esss
For EwC we used HAT's official implementation at
For others, used own implementations.
B. SUPERVISED CONTINUAL LEARNING: HYPERPARAMETER SETTINGS
For MNIST, notMNIST, fashionMNIST datasets, model uses single hidden layer neural network with 200 hidden units. For RCL (Xu & Zhu 2018) and DEN (Yoon et al. 2018). two hidden layers used with initial network size of 256, 128 units, respectively. For the Cifar 100 dataset an Alex-net like structure with three convolutional layers of 128, 256, 512 channels with 4 x 4, 3 x 3, x 2 channels followed by two dense layers of 2048, 2048 units each. For the convolutional layer, batch-norm layers were separate for each task. We adopt Adam optimizer for our model keeping learning rate of 0.01 for the IBP posterior parameters and 0.001 for others; this to avoid vanishing gradient problem introduced by sigmoid function. For selective finetuning, we use learning rate of 0.0001 for all the parameters. The temperature hyperparameter of the Gumbel- softmax reparameterization for Bernoulli gets annealed from 10.0 to minimum limit of 0.25. The value of a initialized to 30.0 for the initial task and maximum of the obtained posterior shape parameters for each of subsequent tasks. Similar to VCL, initialize likelihood training for the first task. For all datasets, we train our model for epochs. We selectively finetune model after that for 5 epochs. For experiments including coresets, we coreset size of 50. Coreset selection is done using random and k-center methods Nguyen et al. (2018). For our model with dynamic expansion, initialize network with 50 hidden units.
used
models with maximum-
B.2 UNSUPERVISED CONTINUAL LEARNING: HYPERPARAMETER SETTINGS
For all datasets, our model uses 2 hidden layers with 500, 500 units for encoder and symmetrically opposite for the decoder with a latent dimension of size 100 units. For other approaches like Naive, EwC and VCL (Kirkpatrick al. 2017 Nguyen al. 2018),
use task-specific encoders with hidden layers of 500, 500, 500 units respectively with latent size of 100 units, and a symmetrically reversed decoder with last two layers of decoder being shared among all the tasks and the first layer
13

---

Under review as a conference paper at ICLR 2021
being specific to each task. ration similar to that of supervised setting. Temperature for gumbel-softmax reparametrization gets annealed from 10 0.25. We initialize encoder hidden layers values as 40, 40, respectively, and symmetrically opposite in decoder for the first task. We update a's in similar fashion to supervised setting for subsequent tasks. For latent layers, intialize to 20. For the unsupervised learning experiments, we did not use coresets.
Adam optimizer for model keeping the learning rate configu-
C CORESET METHOD EXPLANATION
Proposed in Nguyen et al. 2018) as a method for cleverly sidestepping the issue catastrophic forgetting, the coreset comprises representative training data samples from all tasks. Let M(-1) denote the posterior state of the model before learning task t. With the t-th task's arrival having data Dr. a coresct C₁ created comprising choicest examples from tasks 1...t. t. Using data D, \ Ce and having prior (t-1),, new model posterior M is learnt. For predictive purposes at this stage (the test data comes from tasks 1...t), t),
posterior Mr pred is learnt with M prior and with data C.. Note that preed is used only for predictions at this stage, and does not have any role in the subsequent learning of, say, M(++), Such predictive model is learnt after every new task, and discarded thereafter. Intuitively makes sense as some new learnt weights for future tasks can help the older task to perform better (backward transfer) at testing time.
Coreset selection can be done either through random selection or K-center greedy algorithm Gon- zalez (1855). Next, the posterior is decomposed as follows:
Di.) x Di.) (C:))(CC))) =s (O)p((C))))
where, y(0) is the variational posterior obtained using the current task training data, excluding the current coreset data. Applying this trick in a recursive fashion, we can write:
p(0
U C.-1/C,))) a- (0)p() UC-
then approximate this posterior using variational approximation as q (0) = proj(q:- D C--1 (c())) Finally projection step is performed using coreset data before prediction follows: qe (0) = proj (0)p((())) This way of incorporating coresets into coresct data before prediction tries mitigate any residual forgetting. Algorithm summarizes the training procedure for model for setting with known task boundaries.
D DYNAMIC EXPANSION METHOD
Although our inference scheme uses truncation-basee approach for the IBP posterior, it is possible to do inference in concation-free manner. One possibility is to greedily grow the layer width until performance saturates. However we found that this leads to bad optima (low peaks of likelihood). We can leverage the fact that, given sufficiently large number of columns, the last columns of the IBP matrix tends to be all zeros. So we increase the number of hidden nodes after every iteration to keep the number of such empty columns equal constant value 7 in following manner.
D'
K'
C/ = II IB = 0), G'=T' Σ C
(12)
where represents current layer index, B' the sampled IBP mask for current task, C indicates all columns from jth column onward are empty. G¹ is the number of hidden units to expand in the current network layer.
E OTHER PRACTICAL DETAILS
E. SEGREGATING THE HEAD
has been shown in prior work on supervised continual learning Zeno et al. (2018) that using separate last layers (commonly referred "heads") for different tasks dramatically improves
14

---

Under review as a conference paper at ICLR 2021
Algorithm Nonparametric Bayesian CL
Input:Initial Prior Po(
Initialize the network parameters and coresets Initialize Phew <- (e)
i to do
Observe current task data D2;
Update coresets (Sec. C
Masked Training:
L: < ELBO with prior Pnew:
+ < arg min L,;
Selective Finetuning:
Fix the IBP parameters and learned mask;
θ f arg min Cj;
Pnew f- (9):
Pnew Mask(Pnew) using Eq 11
Perform prediction for given test set..
end for
performance in continual learning. Therefore, in the supervised setting, generalized linear model that uses the embeddings from the last hidden layer, with the parameters up to the last layer involved in transfer and adaptation. Although do report comparision of single head models available in Sec H.2
E.2 SPACE COMPLEXITY
The proposed scheme entails storing binary matrix for each layer of each task which results into 1 bit per weight parameter, which is not very prohibitive and can be efficiently stored as sparse matrices. Moreover, the tasks make use of very limited number of columns of the IBP matrix, and hence does not pose any significant overhead. Space complexity grows logarithmically with number of tasks T as O(M + logg(M)) where number of parameters.
E.3 ADJUSTING BIAS TERMS
The IBP selection acts the weight matrix only. For the hidden nodes not selected in task, their corresponding biases need to be removed as well. In principle, the bias vector for a hidden layer should be multiplied by binary vector u. with u, I[d Ba = 1]. In practice, we simply scale each bias component by the maximum reparameterized Bernoulli value in that column.
E.4 SELECTIVE FINETUNING
While training with reparameterization (fumbel-softmax), the sampled masks are close to binary but not completely binary which reduces performance bit with complete binary mask. So we fine- tune the network with fixed masks to restore performance. summarized version of Algorithm summarizes models training procedure. The method for update of coresets that used are similar to as it was proposed in Nguuen et al. (2018).
F ADDITIONAL INFERENCE DETAILS
Sampling Methods We obtain unbiased reparameterized gradients for all the parameters of the variational posterior distributions. For the Bernoulli distributed variables, employ the Gumbel- softmax trick Jang et al. (2017). also known as CONCRETE Maddison et al. (2017). For Beta dis- tributed v's, the Kumaraswamy Reparameterizztioon Gradient technique Nalisnick & Smyth (2017) used. For the real-valued weights, the standard location-scal trick of Gaussians is used.
Inference parameters that involves random stochastic node Z (i.e qo (Z)) cannot be done in a straightforward way, if the objective involves Monte Carlo expectation with respect that random variable (f. Eyy, (T+=)))) This is due to the inability to back-propagate through
15

---

Under review as a conference paper at ICLR 2021
random node. To overcome this issue, Kingma & Welling (013) introduced the reparametrization trick. This involves deterministically mapping the random variable Z = f(Ø, e) to rewrite the expectation in terms of new random variable where is now randomly sampled instead of Z (i.e Ege [I(e, +)). In this section, we discuss some of the reparameterization tricks we used.
F.1 GAUSSIAN DISTRIBUTION REPARAMETERIZZTTON
The weights of our Bayesian nueral network are assumed be distributed according a Gaus- sian with diagonal variances (i.e V₁ ~ (VVllvv V. We reparameterize our parameters using location-scall trick as:
V = LV, VV, X <~N(0,
where the index parameter that we are sampling. Now, with this reparameterization, the gradients over ;V,; VV, can be calculated using back-propagation.
F.2 BETA DISTRIBUTION REPARAMETERIZATIO
The beta distribution for parameters y' in the IBP posterior can be reparameterized using Ku- maraswamy distribution Nalisnick & Smyth (0077), since Kumaraswamy distribution and beta dis- tribution are identical if any one of rate or shape parameters are set to 1. The Kumaraswamy distri- bution defined as p(u ((11 0, a- (11 1a y which can be reparameterized as:
(1 //3/2/a
(0,
where represents a uniform distribution. The KL-Divergence between Kumaraswamy and beta distributions can be written as:
L (p((v,, a
(
(bb
+ log ab + log log(B(a,3))
I-
(8 1)b
20
1
B m a
(13)
m
ab
where y is the Euler constant, Vi is the digamma function and B is the beta function. As described in Nalisnick & Smyth (2017). we can approximate the infinite sum Eq 13 with finite sum using first 11 termss
F.3 BERNOULLI DISTRIBUTION REPARAMETERIZATION
For Bernoulli distribution over mask the IBP posterior, we employ the continuous relaxation of discrete distribution as proposed in Categorical reparameterization with Gumbel-softmax Jang et al. (017)), also known as the CONCRETE Maddison et al. (2017) distribution. We sample a concrete random variable from the probability simplex as follows:
exp((Ig((a)) g2)/A)
Bg K E exp((log(a)) + g)) 9 G(O,
where, À E oo) is a temperature hyper-parameter, discrete class probability for 4th class and gk is random sample from Gumbel distribution G. For binary concrete variables, the sampling reduces to the following form:
is posterior parameter representing the
log (ax) ) + log (ux /11 uz))
YE
UO)
X
then, Bg o({)) where is sigmoid function and ug is sample from uniform distribution U. To guarantee lower bound on the ELBO, both prior and posterior Bernoulli distribution needs to be replaced by concrete distributions. Then the KL--Diergence can be calculated as difference of log density of both distributions. The log density concrete distribution is given by:
log (B;; a, A) log (A) AY + log a 2 log (1 + exp (-XY, + log a4))
With all reparameterization techniques discussed above. imating the ELBO with sample size of 10 while training and a sample size of 100 while at test time.
Monte Carlo sampling for approx-
16

---

Under review as a conference paper at ICLR 2021
G IBP HYPERPARAMETER &
In this section, we discuss the approach to tune the IBP prior hyperparameter a. We found that using a sufficiently large value of without tuning performs reasonably well in practice. However, we experimented with other alternatives as well. For example, we tried adapting a with respect to previous posterior as a = max (a, mar(ay)) for each layer, where du Beta posterior shape parameter. Several other considerations can also be made regarding its choice.
G.1 SCHEDULING ACROSS TASKS
Intuitively, should be incremented for every new task according to some schedule. Information about task relatedness can be helpful in formulating the schedule. Smaller increments of a dis- courages creation of new nodes and encourages more sharing of already existing connections across tasks.
G.2 LEARNING
Although not investigated in this work, one viable alternative to choosing a by cross-validation could be to learn it. This can be accommodated into variational framework by imposing gamma prior on a and using a suitably parameterized gamma variational posterior. The only difference in the objective would be in the KL terms: the KL divergence of will then also have to estimated by Monte Carlo approximation (because of dependency on a in the prior). Also, since gamma distribution does not have analytic closed form KL divergence, the Weibull distribution can be suitable alternative Zhang et al. (2018).
H ADDITIONAL RESULTS: SUPERVISED CONTINUAL LEARNING
In this section, provide some additional experimental results for supervised continual learning setup. Table 2 shows final mean accuracies over 5 tasks with deviations, obtained by all the ap- proaches various datasets. It also shows that model performs comparably better than the baselines. We have included some more models in this comparison namely, HIBNN (Kessler et al.] 2018 and A-GEM (Chaudhry et 2019). Note
2020). UCL (Ahn al., 2019), HAT (Serrà that coreset based replay is not helping much in our case, In of VCL use of coresets performs better since it forces all parameters to be shared leading catastrophic forgetting. Our method has very less catastrophic forgetting hence the use of coresets does not improve performance significantly. Although in cases where we do not grow the model size dynamically and keep feeding tasks even after the model has reached its capacity (model will be forced to share more parameters), it will lead to forgetting and their use of coresets might help as it did for VCL.
H.1 LEARNED NETWORK STRUCTURES
In this section, we analyse the network structures that were learned after training our model. As
Talk3
Teal
Taal
Teat2
Taut
Talk
Tadh
Teat
Talk
TAM
TA0
Tea
TeaC4
(a) split MNIST
(b) not MNIST
(c) fashion MNIST
Figure 6: Learned masks for input to first hidden layer weights on split MIIST(left), not MNIST(iiddle) and fashion MNIST(right) datasets. Darker color represent active weights.
see in Fig. 6 aa), the masks captured the pixel values where the digits in MNIST
17

---

Under review as a conference paper at ICLR 2021
Method
S-MNIST
n-MNIST
P-MNIST
f-MNIST
s-Cifar100
Naive
79.615:10.7
72.33910.8
90.090:10.4
79.31910.6
47.082+10.7
Rehearsal
99.102:+0.3
95.203=0.5
97.
97.981+0.3
61.110+0.4
EwC
81.53010.4
90.297-0.6
95.392.1.55
86.57710.4
55.157
IMM (mode)
02.
84.442+0.4
96.433+0055
88.765+0.4
HIBNN
98.712:1..4
97.003++0.3
VCL
98.952:10.3
93.732+1.3
97.353+10.3
97,970
63.994
VCL(coreset)
98.731+0.4
94.993=
97.464+0.3
98.154+0.3
A-GEM
95.645
62.945
HAT
99.701
96.749
97.912
98.592
66.410
DEN
99.77910.11
06.
97.
98.580
UCL
99.791
97.112
97.883
98.896
64.32
RCL
0..
9..
98.005++022
98.698
64.814
Ours
99.81911.11
97.152-0.2
98.180
98.98611.22
70.105
Ours(coreset)
99.83410.1
97.061=0.2
98.163
98.
69.459+0.2
Table 2: Comparison of final mean accuracies on test set obtained using different methods over 10 runs except for some with zero deviations (1-2 runs). Deviations are rounded to decimal place, very small deviations kept as 0.1.
datasets have high value and zeros elsewhere which represents that our models adapts with respect to data complexity and only uses those weights that are required for the task. Due to the use of the IBP prior, the number of active weights tends to shrink towards the first few nodes of the first hidden layer. This observation enforces that our idea of using IBP prior to learn the model structure based on data complexity is indeed working. Similar behaviour can be seen in notMNIST and fashionMNIST in Fig. and c).
On the other hand Fig 7 (left) shows the sharing of weights between subsequent tasks of different datasets. It can be observed that the tasks that are similar at input level of representation have more overlapping/sharing of parameters (e.g split MNIST) in comparison those that are not very similar (e.g permuted MNIST). also shows Fig (right) that the amount of total network capacity used by our model differs for each task, which shows that complex tasks require more parameters as compared to easy tasks. Since the network size is fixed, the amount of network usage for all previous tasks tends converge towards 100 percent. This promotes parameter sharing but also introduces forgetting, since the network is forced to share parameters and is not able to learn new nodes.
H.2 ADDITIONAL PERMUTED MNIST RESULT
We have done experiments with separate heads for each task of permuted MNIST. Some ap- proaches use a single head permuted MNIST task and don't 't task labels at test-time. Here we compare some of the baselines (that supports single head) with our model (single head) on Permuted MNIST for 10 tasks. We also report number of epochs and average time to run for rough com- parision of time complexity taken by each model. To justify the choice of single hidden layer with
Method
Epochs/Task
Time/Task (sec)
Avg acc(10 tasks)
Ours
10
142
0.9794
VCL
100
380
0.9487
EwC
51
0.9173
200 units in MNIST like experiments, compare model Permuted MNIST experiment with multiple network depths and with separate heads, From table we can conclude that a single hidden layer is sufficient for obtaining good enough results. Further, to analyse the performance decrease and generality approach with number of tasks, we perform Permuted MNIST experiment with separate heads and single hidden layer of 200 units for different number of tasks. Table shows
18

---

Under review as a conference paper at ICLR 2021
35
50
38
40
40
39
Tasks (split MNISTI
60
50
Tasks
54
55
54
50
(fashion
59
53
30
64
60
59
63
42
43
43
41
35
(permuted MNIST)
9
10
10
Tasks (not MNIST)
30
60
49
49
46
Tasks
22
23
23
25
59
57
26
27
20
60
29
Figure 7: Percentage weight sharing between tasks (left), percentage of network capacity already used by previous triks()ight).
Network hidden layer sizes
Avg accuracy (5 tasks)
[000]
98. 180 + 0.187
100, 50]
988188 = 0.163 98.096 = 0.152
2500, 100, 50]
Table 3: Comparing performance on Permuted MNIST under different network configurations
that model quite stable and performance does not drop alot even with large number of tasks for fixed model size.
(Permuted MNIST) No. of tasks
Avg accuracy obtained
5
98. 180
10
98.062
20
97.874
Table 4: Comparison of model performance over different number of tasks for Permuted MNIST
experiment
H.3 ADDITIONAL CIFAR RESULT
MNIST data experiments are relatively casier to model and an approach might not generalize to more complex datasets like image textual data. This section includes extra results cifar- 10 and cifar-100 datasets with comparisons to some very strong baselines for observing performance under complex settings.
Method split Cifar- 100 (10 tasks)
split Cifar-100 (20 tasks)
split Cifar-10 (5-tasks)
VCL 63.994
68.398
81.161
HAT 67.410
72.851
89.356
Ours 70.105
73.534
89.021
Table 5: Avg. accuracy obtained after all tasks are obtained on cifar-10 10 and cifar-100 datasets
B that our approach is comparable to the some strong baselines like HAT, VCL on com- plex tasks like cifar- 10 and cifar- 100 classifications. Therefore. suggesting that it can be generalized to more complex task settings. For split cifar- 100 (20 tasks) each task is 5 class classification task and, split 10 has 2 class classification tasks.
19

---

Under review as a conference paper at ICLR 2021
H.4 OTHER METRICS
We quantified and observed the forward and backward transfer of our and VCL model, using the three metrics given in Díaz- Diaz-Rodriguez al. (2018) Permuted MNIST dataset as follows:
ACCURACY is defined as the overall model performance averaged over all the task pairs as follows:
Σizj Ras
Ace
N(N-1)
where, Rig obtained test classification accuracy of the model task 1y after observing the last sample from task t;.
FORWARD TRANSFER is the ability of previously learnt task perform on new task better and is give by:
Dis Raj
FWT
N(-1)
2
BACKWARD TRANSFER the ability of newly learned task to affect the performance of previous tasks. It can be defined as:
BWT Ef - Rí.))
N(N-1)
We compare our model with VCL and other baselines over these three metrics in Table
Method
Accuracy
FWT
BWT
Naive
90.090
0.1
-3.60e 2
EwC
95.392
0.1
-1.90
Rehearsal
97.565
0.1
+1.30e
VCL
97.353
0.1
-4.00 -3
Ours
98.180
0.1
+1.33e
Table 6: Comparison other metrics for permuted MNIST dataset
We can observe that backward transfer for our model more as compared to most baselines, which shows that our approach has suffers from less forgetting well. On the other hand forward transfer seems to give close to random accuracy (0.1) which is due to the fact that the model not trained on the correct class labels and is asked predict the correct label. So this metric is not very useful here; an alternative would be to train a linear classifier on the representations that are learned after each subsequent tasks for future task.
I UNSUPERVISED CONTINUAL LEARNING
Here we describe the complete generative model for our unsupervised continual learning approach. The generative story for unsupervised setting can be written follows (for brevity we have omitted the task id I):
B' IBP(a)
Vi d.k N(0,00)
W = B' の V
oout .O0, oz)
d.k
Zn N(pe,
X, Bernoulli
NNN Z,)))
20

---

Under review as a conference paper at ICLR 2021
where, Mz, of are prior parameters of latent representation; they either be fixed learned, and a the sigmoid function. The breaking process for the IBP prior remains the same here as well. For doing inference here, once again resort to structured mean-field assumption:
(Z, V, B) - (Z B.V X )9 ( (B)
where, q( B, V. D. X) II T (HuWN o &wx)): ) and NNN is IBP masked neural network used for amortization of Gaussian posterior parameters. Rest of variational posteriors are factorized in similar way in the supervised approach. Evidence lower bound calculation done explained in section 3.3.
1.1 ADDITIONAL EXPERIMENTAL RESULTS FOR UNSUPERVISED CONTINUAL LEARNING
In this section, we show further results for unsupervised continual learning. Fig TO shows, for MNIST and notMNIST datasets, how the likelihoods vary for individual tasks subsequent tasks arrive. It can be observed that the individual task accuracies learned by our model are better than
0
2
3
4
5
6
7
8
9
O
2
3
4
5
6
7
8
9
0
2
3
4
5
6
7
8
9
O
/
5
6
7
8
9
20
O
1
3
4
5
67
8
9
150
0
2
3
4
5
6
7
8
9
O
1
2
3
4
5
6
7
0
9
0
2
3
4
5
6
7
8
9
0
2
3
4
5
6
7
8 9
0
2
3
4
5
G
7
8
9
-00
Figure 8: On MNIST dataset (left) Reconstruction images after all tasks have been observed. (right) t-SNE plot of each class after all tasks have been observed.
A
ABCDEFGHIJ 60-
ABCDEFGHIJ
ABCDEFGHIJ
ABCDEFGHIJ
A F
ABCDEFGHT
209
ABCDEFGH
A ABCDGFGH G
-00
ABCDEFGH
Figure 9: On notMNIST dataset (left) Reconstruction of images after all tasks have been observed.
(right) i-SNE plot of each class after all tasks have been observed.
other baselines; this suggests that use of new weights when needed helps in retaining a better optima per task, and also the deterioration of our model is much less as compared to other model, repre- senting effective protection against catastrophic forgetting. Fig 12 shows the reconstructed images of MNIST and also the t-SNE plot of latent codes our model produces. can be observed that re- construction quality is good despite heavy constraints the model. Fig Tia shows the generated
21

---

Under review as a conference paper at ICLR 2021
1
2
3
-40
-110
-10
-95
-205
600
-120
-100
0123
6
12345 7.9
9
-7
-100
-80
-105
-115
-90
B
200
1150
X
-200
-200
200
-20
0123
6
4
3
H
200
-175
-150
-200
250
-00
-15
7
naive
eWC
vcl
ours
Figure 10: Generative Model Test likelihood decays of individual tasks after subsequent tasks
have been observed. (Top two) represents MNIST and (Bottom two) represents notMnist datasets.
0
2
3
4
5
6
7
8
ABCDEFGHTJ
0
2
3
5
6
8
8
8
ABCDEFGHIJ
0
a
3
4
6
6
2
8
9
ABCDEFGHI
0
1
2
3
4
5
6
7
0
9
ABCDERGHIJ
0120
1
2
3
5
6
7
8
9
ABCDEFGHOJ
158
150
01
2 2
3
4
5
6
7
8
9
01
1
3
5
6
7
8
9
ABCDEFGHIT
0
1
2 2
3
5
6
7
8
9
ABCDECGHID
0
1
3
4
5
6
7
0
9
ABCDEFGMTT
0
2
3
4
5
6
7
6
9
ABCDEFGHTJ
300
200
(a) Generated samples on MNIST dataset after all
(b) Generated samples on notMNIST dataset
tasks have been observed
after all tasks have been observed
Figure 11: Generated Samples
22

---

Under review as a conference paper at ICLR 2021
samples from the learned prior over latent space after all tasks are observed.
Similarily, Fig shows the reconstructed images of not MNIST dataset and the t-SNE plot of latent codes our model produces, and Fig Tlb shows the generated samples from the learned prior over latent space after all tasks are observed.
MNIST (error)
not MNIST (error)
Benchmarks
3-KNN
5-KNN
10-KNN
3-KNN
5-KNN
10-KNN
Naive
30.1%
33.1%
36.0%
20.6%
24.87%
30.8%
EwC
16.6%
19.48%
22.3%
11.7%
13.1%
17.8%
VCL 17.0%
19.02%
30.2%
12.3%
13.8%
16.5%
Ours 0.37%
0.40%
0.51%
0.08%
0.09%
0.21%
Table 7: Unsupervised learning benchmark comparison with sampled latents using multiple K-nearest neighbour errors obtained from each baseline
C922_ 0
C352
Clss
C022
clss_0 0
class
rlas_
Cos2_6
chss 7
Figure 12: t-SNE plot of latent space of VCL model on notMNIST (left) and MNIST (right) datasets
REPRESENTATION LEARNING In t-SNE plots, it can be observed that the latent space for MNIST dataset more clearly seperated as compared to notMNIST dataset. This can be attributed to the abundance of data and less variation in MNIST dataset as compared to notMNIST dataset. further analyzed the representations that were learned by our model by doing K-Nearest Neighbour classification on the latent space. Table 7 shows the KNN test error of our model and few other benchmarks on MNIST and notMNIST datasets. We performed the test with three different values for K. As shown in the table, the representations learned by other baselines are not very useful (as evidenced by the large test errors), since the latent space are not shared among the tasks, whereas our model uses shared latent space (yet modulated for each task based on the learned task-specific mask) which results in effective latent representation learning.
J TASK AGNOSTIC SETTING
We extended our unsupervised continual learning model to a generative mixture model, where each mixture component is considered as task distribution (i.e p(X) ΣK p(X k) k) with & representing the task identity). Here, p(t = k) can be assumed to a uniform distribution but it fails to consider the degree upto which each mixture is being used. Therefore, we keep count over the number of instances belonging to each task and use that as prior (i.e p(t = k) N, with N₂ being effective number of instances belonging to task k and = Ex
DETECTING BOUNDARIES Inspired from Rao al. (2019)), rely threshold to deter- mine if the data point an instance from a new task or not. During training, any instance with Eptt (..) (ELBO,,) less than threshold Tne is added buffer D Once the buffer D reaches
23

---

Under review as a conference paper at ICLR 2021
2
3
9
5
6
7
9
O
2
3
4
5
6
7
8
9
62
de33.5
O
2
3
9
5
6
2
8
9
2
3
9
5
6
7
8
9
2
3
9
6
7
8
9
3
0
3
3
4
5
6
7
5
9
2
3
4
6
7
8
9
0
2
3
9
5
6
7
8
9
-62
0
2
3
9
5
7
8 3
9 9
2
3
9
G
7
-50
-25
25
26
25
25
20
28
28
250
25
Figure 13: Reconstructed MNIST samples and T-SNE plots of our task agnostic setting
fixed size limit M, extend network with new task parameters and train network on Dnows with known task labels (i.e ply + 1) = 1 where T is total number of tasks learned)
SELECTIVE TRAINING Note that training this mixture model will require to have all task spe- cific variational parameters to be present at every time step unlike the case in earlier settings where we only need store the masks and can discard the variational parameters of previously seen tasks. This will result in storage problems since the number of parameters will grow linearly with the num- ber of tasks. To overcome this issue we fix the task specific mask parameters and prior parameters before the network is trained on new task instances. After the task specific parameters have been fixed, the arrival data belonging to a previously seen task (prey handled by training the network parameters with task specific masks B Bprev.
REPRESENTATION LEARNING It makes more sense do learn representations when we don't 't have target class labels task labels. As diseussed, trained model using gating mechanism with a threshold value of - 130. Fig [3] qualitatively shows the t-SNE plots and reconstruction for each class data points. Based on these results, understood and separated by our model.
conclude that the task boundaries are well
24


Reviews:
[{"review_id": "o8K36_tPmYq", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a new structure-learning approach to Continual Learning, by modelling each hidden layer using a nonparametric Bayesian prior, a technique inspired by recent work on learning sparse NNs. Using this technique, a sparse subset of available weights is used for each task, selectively allowing knowledge sharing between subsequent task while reducing catastrophic forgetting on the deactivated connections. \n\nThis makes for an overall very convincing submission, reflected by my minor criticism. Well placed among other recent publications on CL in top venues. \n\nPros (in no particular order):\n- The method is principled and follows naturally and elegantly in the VCL framework. The presentation follows a clear narrative that is easy to follow.\n- The method comes with a task detection mechanism.\n- Both discriminative and generative modelling are naturally supported in the same framework.\n- The Appendix discusses all necessary details to a level well-above standard in the literature. Analysis covers most of the interesting questions that can naturally be asked about this method. \n- Presented results are overall strong and cover standard evaluation in the literature. I was pleased to see experiments in an application other than supervised image classification (here unsupervised learning). \n- A simple heuristic for dynamic expansion is introduced, a worthwhile direction for future research. \n\nCons (in no particular order):\n- This method requires storage of a binary matrix for task and each layer in the network. However, the authors show that space complexity grows logarithmically with the number of tasks, which is likely to make this an acceptable trade-off. \n\nAuthor feedback:\n- IMHO the main contribution of this work is its ideas on structure learning in the context of Continual Learning. This is however not reflected in the title.\n- The paper would benefit from clearly stating why structure learning for CL is a worthwhile direction to pursue. Well-written papers clearly state why the proposed direction is a worthwhile method of investigation. \n- Figure 3 is missing axis labels", "extra_scores": {}, "extra_text": {}}, {"review_id": "_oHLrL7JdQl", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I found it difficult to evaluate this paper because the paper does not say which continual learning problem it is solving in the supervised learning case. Is it solving the class-incremental learning or task-incremental learning problem? Without knowing that, it is hard to make an assessment because the two problems are usually solved in very different ways and their evaluation protocols are different too. I tried to guess but get confused. \n\nThe paper writes in the evaluation section “To gauge the effectiveness of our model towards preventing catastrophic forgetting, we report (i) the test accuracy of first task after learning each of the subsequent tasks; and (ii) the average test accuracy over all previous tasks 1, 2, . . . t after learning each task.” And it also writes earlier “Omitting the task id t for brevity.” I am confused with these two statements. Do you need task id during training and testing? If you are solving the class-incremental learning, task id should not be used in training or testing, at least not testing. How do you do (i)? Do you only use the test instances of the first task? Do you restrict those instances to be classified into only the classes in the first task, or do you allow them to be classified to future classes in future tasks? For class-incremental learning, one should be getting the accuracy of all classes learned so far rather than each task. So, I am guessing that you are doing task-incremental learning. (ii) also gives me the same impression. If that is the case, the following systems are expected to be compared: Uncertainty-based Continual Learning with Adaptive Regularization (NIPS-2019) and Overcoming Catastrophic Forgetting with Hard Attention to the Task (Serrà et al., 2018). IBP is related to the mechanisms in (Serrà et al., 2018) and (Adel et al. 2020). It is desirable to have them compared. \n\nIf you are doing class-incremental learning, more recent baselines should be compared. The baselines used in your experiments are old. \n\nLearning a Unified Classifier Incrementally via Rebalancing. CVPR 2019. \nOvercoming catastrophic forgetting for continual learning via model adaptation. ICLR, 2019. \nLarge scale incremental learning. CVPR 2019\nRandom path selection for continual learning. NeurIPS 2019\nContinuous learning of context-dependent processing in neural networks. Nature Machine Intelligence, 2019.\nitaml : An incremental task-agnostic meta-learning approach, CVPR, 2020\nContinual learning with hypernetworks. ICLR, 2020\n\nIn the experiment, varying the number of tasks for each dataset is also desired to show the generality of the proposed approach. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "N6LbKTPB_H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes an online variational Bayesian (VB) approach to continual learning. The prior over neural network functions is both over the neural network structure and parameter values, where the structure is modelled by an Indian Buffet process (IBP) and the weights are drawn from a Gaussian.\nSimilarly, the approximate posterior is assumed to be IBP and factorised Gaussian as well and inference is performed through variational inference and reparametrization of the respective distributions.\n\nThe approach is similar to VCL in that it uses online VB for learning, however, the prior and approximate posterior is more general in that it also considers the neural network structure as a random variable (prior and posterior). \n\nTheory:\nThe approach is theoretically sound and well motivated; the paper is presented well and easy to follow.\nMy main concern is that that the second paragraph of the paper motivates with scenarios where the ability to *adapt to dynamically changing environments or evolving data distributions* is essential. However, online VB assumes iid data. That is, the online algorithm should infer the posterior over all tasks rather than adapting to dynamically changing data distributions. Inference is sequentially, but the ordering of the task should in theory not matter - it only matters in practice as we perform approximations. \nSee e.g. [1] (ICLR 2020) for an approach that explicitly adapts (through forgetting) the distribution over neural network weights. It could be possible to extend this work to similar adaptation mechanisms, although for multi-task learning such adaptation/forgetting may not be desirable. I would appreciate a few comments on this and I think it should also be discussed shortly in the paper. \n\nExperimental evaluation:\nThe experimental section considers scenarios that are very common in the CL literature. Unfortunately these are not the most interesting or insightful, as these are variants of MNIST. But since most related work considers these settings as well, the choice is justified. \nThe results are quite strong. I find the results on classification from the latent space of the unsupervised learning approach especially convincing and interesting (Table 1). \n\nRelated work:\nThe relation to [2] needs to be discussed in more detail. What exactly is the difference if the IPB is put on the activations rather than weights? What are pros and cons? I am aware that there are additional experiments in the supplementary material comparing to Kessler. Why do you think your approach outperformed the one of Kessler?\n\nAnother interesting aspect is that the coreset does not help much, which is in contrast to VCL. Do you think this is because the performance is already high? Or because the coreset selection algorithms (k-center, random) are unsuitable?\n\n[1] Continual Learning with Bayesian Neural Networks for Non-Stationary Data, ICLR 2020\n[2] Hierarchical indian buffet neural networks for bayesian continual learning\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4eVr-eStwC_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:  The paper proposes a continual learning framework based on Bayesian non-parametric approach.  The hidden layer is modeled using Indian Buffet Process prior.  The inference uses a structured mean-field approximation with a Gaussian family for the weights, and Beta-Bernoulli for the task-masks.  The variational inference is done with Bayes-by-backprop on a common ELBO setup.  The experiments show less diminishing accuracy on the increment of tasks on five datasets for the discriminative problem, and for generation the methods learn one digit or character at a time on MNIST and notMNIST datasets.\n\n\nPros:\n- The paper shows a structured mean-field approximation of the continual learning problem, and train a single hidden layer of 200 units on the discriminative and generative settings.\n- The paper is easy to follow.\n\nCons: \n- The structure of the network is rather shallow.  The authors also mentioned the challenge in their conclusion.\n- The experiments were done on rather simple datasets.  The paper introduces a proof of concept instead of showing the capabilities of the proposal in complex scenarios.\n- The mixture of the structured mean-field approximation seems straightforward and builds extensively on previous work.  The novelty may be on making the network train using this approach, but the proposal uses a rather simple layer configuration (one layer).\n\nComments:\n- In Section 5.1 you mention that you report the mean accuracies on the different tasks (Fig. 2).  How many versions of the tasks do you generate and average over?  Adding error bars would help to see the variance of the methods over the tasks.\n\nMinor comments:\n- Typo P6 par2 \"tries to adapts\"\n- Typo P7 par 3 \"comapred\"\n\nOverall rating:\nThe idea of applying Bayesian non-parametric for continuous learning is interesting, and the authors show a simple implementation on simple datasets.  The evaluated tasks are extremely related and in a general continuous learning setup this method may not work.  More extensive and complex experiments (i.e., more complex databases and setups for discrimination as well as for generation) may shed some light on the process.  Due to all these issues I rate the paper as a 5.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a Bayesian non-parametric method for task-incremental continual learning. It is more general than previous work in that it considers the network structure as a random variable and works for both supervised and unsupervised settings. Experimental results show that the proposed method outperforms prior work in the proposed tasks.

Pros:
- It is well motivated.
- It's theoretically sound.
- It can do task inference.
- It outperforms other methods in the proposed tasks.

Cons:
- The experimental setup was not very challenging, because the dataset(MNIST) was simple and the network was shallow.
- There was no ablation study to analyze the contributions of the algorithm to the performance.
- There is not enough experiments to support the advantage of task inference.
- The paper did not compare with the SOTA task-incremental learning algorithms HAT and DEN.

The main concerns of reviewers are on the experimental section as listed in cons and the difference from previous work. The authors explained that their method has the advantages over previous work
that it can do task inference. R3 agreed with the advantage and suggested more experiments in this direction should be performed.
The authors conducted additional experiments suggested by the reviewers including comparison with HAT. They also uploaded a revised version to incorporate the comments from the reviewers.

I think the paper is well motivated and the idea of applying Bayesian non-parametric for continuous learning is interesting. It could potentially motivate interesting future work on CL. However, the main advantages/contributions are not well presented and supported by the experiments. So at present time I believe there is much room for the authors to improve their paper before publication. I hope that the authors will be able to address the feedback they received to make this submission get where it should be.
Decision: Reject