A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning
Authors: Soochan Lee
Junsoo Ha
Dongsu Zhang
Gunhee Kim
Venue:  (2020)
Keywords: continual learning
task-free
task-agnostic

Abstract: Despite the growing interest in continual learning, most of its contemporary works have been studied in a rather restricted setting where tasks are clearly distinguishable, and task boundaries are known during training. However, if our goal is to develop an algorithm that learns as humans do, this setting is far from realistic, and it is essential to develop a methodology that works in a task-free manner. Meanwhile, among several branches of continual learning, expansion-based methods have the advantage of eliminating catastrophic forgetting by allocating new resources to learn new data. In this work, we propose an expansion-based approach for task-free continual learning. Our model, named Continual Neural Dirichlet Process Mixture (CN-DPM), consists of a set of neural network experts that are in charge of a subset of the data. CN-DPM expands the number of experts in a principled way under the Bayesian nonparametric framework. With extensive experiments, we show that our model successfully performs task-free continual learning for both discriminative and generative tasks such as image classification and image generation.

Published as a conference paper at ICLR 2020
A NEURAL DIRICHLET PROCESS MIXTURE MODEL FOR TASK-FREE CONTINUAL LEARNING
Soochan Lee, Junsoo Ha, Dongsu Zhang & Gunhee Kim
Department of Computer Science, Seoul National University, Seoul, Republic of Korea {soochan. 1ee, junsoo. ha @vi on. http://vision smu wttpaaoo...........ooeeeeetee...
on www..snu.cem ac.kac.kr, {961ives gunhee)
ABSTRACT
Despite the growing interest in continual learning, most its contemporary works have been studied in a rather restricted setting where tasks are clearly distinguish- able, and task boundaries are known during training. However, if our goal is develop an algorithm that learns as humans do, this setting is far from realistic, and it is essential to develop a methodology that works in a task-free manner. Mean- while, among several branches of continual learning, expansion-based methods have the advantage of eliminating catastrophic forgetting by allocating new re- crransion-basee approach
sources to learn data. In this work, propose for task-free continual learning. Our model, named Continual Neural Dirichlet Process Mixture (CN-DPM), consists of set of neural network experts that are in charge of a subset of the data. CN-DPM expands the number of experts principled way under the Bayesian nonparametric framework. With extensive experiments, we show that our model successfully performs task-free continual learning for both discriminative and generative tasks such as image classification and image generation.
INTRODUCTION
Humans consistently encounter new information throughout their lifetime. The way the information provided, however, vastly different from that of conventional deep learning where each mini- batch is iid-sampled from the whole dataset. Data points adjacent in time can be highly correlated, and the overall distribution the data can shift drastically as the training progresses. Continual learning (CL) aims at imitating incredible human's 's ability learn from non-iid stream of data without catastrophically forgetting the previously learned knowledge.
Lopez-Pazz Paz & Ranzato 2017 Kirkpatrick al. 2017 Rusu et al., 6 Shin et al. [017]] Yoon et 2018) assume that the data stream is explicitly divided into sequence of tasks that known training time. Since this assumption is far from realistic, task-free CL is more practical and demanding but has been largely understudied with only few exceptions of (Aljundi al. 2019a b). In this general CL, not only explicit task definition unavailable but also the data distribution gradually shifts without a clear task boundary.
Most CL approaches (Aljundi et al., 2018 2017: Lopez.
Meanwhile, existing CL methods can be classified into three different categories (Parisi al. 2019); regularization, replay, and expansion methods. Regularization and replay approaches address the catastrophic forgetting by regularizing the update of a specific set of weights or replaying the previ- ously seen data, respectively. On the other hand, the expansion methods are different from the two approaches in that it can expand the model architecture to accommodate new data instead of fixing it beforehand. Therefore, the expansion methods bypass catastrophic forgetting by preventing pre-existing components from being overwritten by the new information. The critical limitation of prior expansion methods, however, is that the decisions of when to expand and which resource to use heavily rely on explicitly given task definition and heuristics.
In this work, goal is to propose novel expansion-basse approach for task-free CL. Inspired by the Mixture of Experts (MoE) (Jacobs et al.] 1991), our model consists of a set of experts, each of which is in charge of subset of the data in stream. The model expansion (i.e., adding more

---

Published as a conference paper at ICLR 2020
experts) is governed by the Bayesian nonparametric framework, which determines the model com- plexity by the data, as opposed to the parametric methods that fix the model complexity before training. We formulate the frse CL online variational inference of Dirichlet process mix- ture models consisting of a set of neural experts; thus, we name our approach as the Continual Neural Dirichlet Process Mixture (CN-DPM) model.
We highlight the key contributions of this work as follows.
We are one of the first to propose an expansion-base based approach for task-free CL.. Hence, our model not only prevents catastrophic forgetting but also applies the setting where no task definition and boundaries are given at both training and test time. Our model named CN-DPM consists of set of neural network experts, which are expanded in principled way built upon the Bayesian nonparametrics that have not been adopted in general CL research.
Our model can deal with both generative and discriminative tasks of CL. With several benchmark experiments of CL literature on MNIST, SVHN, and CIFAR 10/100, we show that our model successfully performs multiple types of CL tasks, including image classifi- cation and generation.
2 BACKGROUND AND RELATED WORK
2.1 CONTINUAL LEARNING
Parisi et al. 22119) classify CL approaches into three branches: regularization (Kirkpatrick et al., 2017 Aljundi et al. 2018). replay (Shin al.) 2017 and expansion (Aljundi et al. 2017 Rusu et al. 2016 Yoon et al. 2018) methods. Regularization and replay approaches fix the model archi- tecture before training and prevent catastrophic forgetting by regularizing the change of a specific set of weights or replaying previously learned data. Hybrids of replay and regularization also exist, such as Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato 20177 Chaudhry et al. 201 On the other hand, methods based on expansion add new network components to learn new data. Conceptually, such direction has the following advantages compared to the first two: (i) catastrophic forgetting be eliminated since information is not overwritten pre-existing components and (ii) the model capacity is determined adaptively depending on the data.
Task-Free Continual Learning. All the works mentioned above heavily rely explicit task defi nition. However, in real-world scenarios, task definition is rarely given at training time. Moreover, the data domain may gradually shift without any clear task boundary. Despite its importance, task- free CL has been largely understudied; to the best of our knowledge, there are only a few works (Aljundi al. 2019a Rao al. 2019), each of which is respectively based on regularization, play, and bybrid of replay and expansion. Specifically.
et al. (2019a) extend MAS (Aljundi) 20.8) by adding heuristics to determine when to update the importance weights with no task definition. In their following work (Aljundi et al. 2019b), they improve the memory management al- gorithm of GEM (Lopez-Paz & Ranzato 2017) such that the memory elements are carefully selected to minimize catastrophic forgetting. While focused on unsupervised learning, |Ro al. (2019) is parallel work that shares several similarities with our method, e.g., model expansion and short-term memory. However, due to their model architecture, expansion is not enough to stop catastrophic forgetting; consequently, generative replay plays a crucial role in Rao et al. (2019). As such, it can be categorized hybrid of replay and expansion. More detailed comparison between our method and Rao et al. (019) is deferred to Appendix M
2.2 DIRICHLET PROCESS MIXTURE MODELS
We briefly review the Dirichlet process mixture (DPM) model (Antoniak) 1974. Ferguson 1983), and variational method to approximate the posterior of DPM models in online setting: Sequen- tial Variational Approximation (SVA) (Lin 2013). For a more detailed review, refer to Appendix
A
2

---

Published as a conference paper at ICLR 2020
Dirichlet Process Mixture (DPM). The DPM model is often applied to clustering problems where the number of clusters is not known in advance. The generative process DPM model is
Zn p(x;;,))) on G, G~ DP(a,
where Vy. the n-th data, and On is the n-th latent variable sampled from G, which itself is distribution sampled from a Dirichlet process (DP). The DP is parameterized by a concentration parameter a and base distribution Go. The expected number of clusters is proportional to a, and Go is the marginal distribution of when G is marginalized out. Since G' is discrete with probability (Teh 2010), same values can sampled multiple times for 9. If 0, = fm. the two data points In and T' belong to the same cluster. An alternative formulation uses the variable 2n that indicates to which cluster the n-th data belongs such that O,, = Φz., where Φy is the parameter the k-th cluster. In the context of this paper, Φu refers to the parameters of the k-th expert.
Approximation of the Posterior of DPM Models. Since the exact inference of the posterior of DPM models is infeasible, approximate inference methods are applied. Among many approximation methods, we adopt the Sequential Variational Approximation (SVA) (Lin 2013). While the data is given one by one, SVA sequentially determines Pn and Vk. which the variational approximation for the distribution of 2n and Φ respectively. Since Pr. satisfies Σx Pn.k = and Pa.k >= O, Pr.kk can be interpreted the probability of n-th data belonging k-th cluster and is often called responsibility. Pn+1 and (a+1)
at step n + 1 are computed as:
Ei=i Pi,k) ) Sg p(en+tll)) (do) a P(Ent( (o)
(do)
if k K f
Pm+1k
(d+) Uk
Go(dd) II p(as)))
if < k K if +
(3)
In practice, SVA adds a component only when PK-1 is greater than certain threshold If Go and p(2,10) are not a conjugate pair, stochastic gradient descent (SGD) is used to find the MAP estimation & with learning rate of A instead of calculating the whole distribution vx:
((++1) ok
+ >(V g(w) log Gu(ot
g(n) log p
(4)
DPM for Discriminative Tasks. DPM can be extended discriminative tasks where each data point is an input-output pair (x, y). and the goal is to learn the conditional distribution p(y|z). To use DPM, which is a generative model, for discriminative tasks. we first learn the joint distribution p(x, y) and induce the conditional distribution from it: p(y|z) p(z, u)/ Jy p(z, v). The joint
distribution modeled by each component be decomposed as p(x, y/2) p(yl x. 2 2)((=(2)
(Ras-
mussen & Ghahramani 2002 Shahbaba & Neal 22009).
DPM in Related Fields. Recent works Nagabandi al. 2019) and Jerfel et al. (2019) exploit the DPM framework to add new components without supervision in the meta-learning context. Naga bandi al. (2019) apply DPM to the model-based reinforcement learning to predict the next state from given state-action pair. When a new task appears, they add a component under the DPM framework to handle predictions in the new task. Jerfel et al. (2019) apply DPM online meta- learning. Extending MAML (Finn et al.) (017)), they assume that similar tasks can be grouped into a super-task in which the parameter initialization shared among tasks. DPM is exploited to find the super-tasks and the parameter initialization for each super-task. Therefore, it can be regarded as a meta-level CL. method. These works, however, lack generative components, which are often es- sential to infer the responsible component at test time, will be described in the next section. As consequence, it is not straightforward to extend their algorithms to other CL settings beyond model- based RL or meta-learning. In contrast, method implements DPM model that is applicable to general fask-free& CL.
3 APPROACH
We aim at general task-free CL, where the number of tasks and task descriptions are not available at both training and test time. We even consider the case where the data stream cannot be split into
3

---

Published as a conference paper at ICLR 2020
separate tasks in Appendix F All of the existing expansion methods not task-free since they require task definition at training (Aljundi et al. 2017) or even at test time (Rusu et al., 2016; Xu & Zhu 2018; Li et al. 2019). We propose novel expansion method that automatically determines when to expand and which component to use. We first deal with generative tasks and generalize them into discriminative ones.
3.1 CONTINUAL LEARNING AS MODELING OF THE MIXTURE DISTRIBUTION
We can formulate a CL scenario as a stream of data involving different tasks D1, D2.... where each task DE set of data sampled from a (possibly) distinct distribution p(æ|2 k). If K tasks are given so far, the overall distribution is expressed as the mixture distribution:
K
p(x)= = Σ p(z|2 = k)p(= k)) k),
(5)
R=1
Ex NA. The goal of CL learn the mixture distribution an online manner. Regularization and replay methods directly model the approximate distribution p(r; Φ) parameterized by single component é and update it to fit the overall distribution p(z). When updating Φ, however, they do not have full access to all the previous data, and thus the information of previous tasks is at risk of being lost as more tasks are learned. Another way to solve CL is to use a mixture model: approximating each p(z|= = k) with p(x; ox). If we learn a task distribution p(æ|2 K + 1) with parameter Φ and leave the existing parameters intact, we can preserve the knowledge of the previous tasks. The expansion-basee CL methods follow this idea.
where p(= k) can be approximated by N where N₂. ID& and -
Similarly, in the discriminative task, the goal of CL. is to model the overall conditional distribution, which is mixture of task-wise conditional distribution p(y]ær, k):
p(y|x) Σ plula, k)p(==>><>>>)) k((r).
Prior expansion methods use expert networks each of which models task-wise conditional distri- bution p(yla; db However, new problem arises in expansion methods: choosing the right expert given 2, i.e., p[2[2r in (). Existing methods assume that explicit task descriptor is given, which is generally not true in human-like learning scenarios. That is, we need a gating mechanism that can infer p(=|z)) only from (i.c., which expert should process x). With the gating, the model prediction naturally reduces to the sum of expert outputs weighted by the gate values, which the mixture of experts (MoE) (Jacobs et al. 1991) formulation: p(y]z) E. P(ylar; px)p(2 ke(xr).
However, it is not possible to use a single gate network as in Shazeer et al. (017) to model p(2[2) in CL,; since the gate network is classifier that finds the correct expert for a given data, training it in online setting causes catastrophic forgetting. Thus, one possible solution to replace a gating network to couple each expert with generative model that represents p(z|2 k) in Rasmussenn & Ghahramani (2002) and (Shahbaba & Neal (2009). As result, we can build a gating mechanism without catastrophic forgetting as
p(x; plx: ppz k) En pla; of b0)(== kk) k))
p(y|z)) Epullz
k/z) Σ Elylo: o
where p(= k) R Nt/N. We also differentiate the notation for the parameters of discriminative models for classification and generative models for gating by the superscript D and G.
If we know the assignment of z, which is the case of task-based CL. can independently train a discriminative model (i.e., p(y [z; xf) and a generative model (i.e., p(æ; Φf)) )) for each task
k. In task-free CL, however, z is unknown, so the model needs infer the posterior p(z|æ, v). Even worse, the total number of experts is unknown beforehand. Therefore, we propose to employ Bayesian nonparametric framework, specifically the Dirichlet process mixture (DPM) model, which can fit mixture distribution with no prefixed number of components. We use SVA described in
'The models with multiple output heads sharing the same base network (Rusu al. 2016 2018 can also fall into this category the expert correspond each subnetwork connected an output head.
Yoon et al.
4

---

Published as a conference paper at ICLR 2020
p v:
N1
$
Wake
Mic+1
Pk oc NxP (y
xi & (x +f)
N2
p(x,))
x
6
Sleep if STMis full
MAMA STM
x, y
Nk
p y: xx)
Train new expert
STM
p(12x)
Px+1 appy xx 40)pppppp))))))p))) (x; (8)
p k)
(a) Training
(b) Inference
Figure 1: Overview of our CN-DPM model. Each expert & (blue boxes) contains discriminative component for modeling pyy x; Φ and generative component for modeling p(z; ) jointly representing p(æ, y; o&). We also keep the assigned data count N₁. per expert. (a) During training, each sample (1c, y) coming in a sequence is evaluated by every expert to calculate the responsibility Pl: of each expert. If /**+ is high enough, i.e., none of the existing experts responsible, the data stored into short-term memory (STM). Otherwise, it is learned by the corresponding expert. When STM is full, a new expert is created from the data in STM. (b) Since CN-DPM is a generative model, first compute the joint distribution p y) for given x, from which it is trivial to infer p(ylæ:).
section 2.2 to approximate the posterior in an online setting. Although SVA is originally designed for the generative tasks, it is easily applicable discriminative tasks by making each component k: to model y12) = p(y| xr 2)p(r(z).
3.2
THE CONTINUAL NEURAL DIRICHLET PROCESS MIXTURE
MODEL
The proposed approach for task-free CL, named Continual Neural Dirichlei Process Mixture (CN- DPM) model, consists of set of experts, each of which is associated with discriminative model (classifier) and a generative model (density estimator). More specifically, the classifier models p(ylar, k),, for which we adopt any classifier regressor using deep neural networks, while the density estimator describes the marginal likelihood p(r|z = k). for which we can use any explicit density model such VAEs (Kingma & Welling 2014) and PixeIRNN (Oord et al., 2016). We respectively denote the classifier and the density estimator of expert ls as p(y [2; Φk) and o(); where df and of are the parameters of the models. Finally, the prediction p(ylær) can be obtained from Eq. (7) by plugging in the output of the classifier and the density estimator. Note that the number of experts is not prefixed but expanded via the DPM framework. Figure illustrates the overall training and inference process of our model.
Training. We assume that samples sequentially arrive one at a time during training. For a new sample, we first decide whether the sample should be assigned to an existing expert or a new expert should be created for it. Suppose that samples up to (In, Bn) are sequentially processed and K experts are already created when a new sample (In+1, fn+11 arrives. We compute the responsibility Pm+1, as follows:
(ie PE) P(un+1/ (un1: (P)p(((nt: oppynti 1n of |( ) where Φn Go(d) if K
if $ < K
Pn+1,
(8)
where Go is a distribution corresponding to the weight initialization. If arg max; Pa+ + K + 1, the sample is assigned to the existing experts proportional to P and the parameters of the experts are updated with the new sample by Eq. 4) such that de is the MAP approximation given the data assigned up the current time step. Otherwise, create
expert.
Short-Term Memory. However, it is not a good idea to create a new expert immediately and initialize it to be the MAP estimation given Ix+1- Since both the classifier and density estimator of an expert are neural networks, training the new expert with only a single example leads to severe overfitting. To mitigate this issue, we employ short-term memory (STM) to collect sufficient data
5

---

Published as a conference paper at ICLR 2020
before creating new expert. When data point is classified new, store it to the STM. Once the STM reaches its maximum capacity M, we stop the data inflow for a while and train a new expert with the data in the STM for multiple epochs until convergence. We call this procedure sleep phase. After sleep, the STM is emptied, and the newly trained expert is added to the expert pool. During the subsequent wake phase, the expert is learned from the data assigned to it. This STM trick assumes that the data in the STM belong to the same expert. We empirically find that this assumption acceptable in many CL settings where adjacent data are highly correlated. The overall training procedure is described in Algorithm I brevity.
Note that we use Pn.o instead of Pn,K+1 in the algorithm for
Inference. At test time, we infer p(y|z) from the collaboration of the learned experts as in Eq.
Techniques for Practicality. Naively adding new expert has two major problems: (i) the number of parameters grows unnecessarily large as the experts redundantly learn common features and (ii) there is no positive transfer of knowledge between experts. Therefore, we propose a simple method to share parameters between experts. When creating a new expert, we add lateral connections to the features of the previous experts similar to Rusu et al. (2016). To prevent catastrophic forgetting in the existing experts, we block the gradient from the new expert. In this way, we can greatly reduce the number of parameters while allowing positive knowledge transfer. More techniques such as sparse regularization in Yoon et al. (2018) can be employed to reduce redundant parameters further. As they are orthogonal to our approach, we do not use such techniques in our experiments. Another effective technique that we use in the classification experiments is adding a temperature parameter to the classifier. Since the range of log p(z is far broader than log p(y z),, the classifier has almost no effect without proper scaling. Thus, we can increase overall accuracy by adjusting the relative importance of images and labels. We also introduce an algorithm to prune redundant experts in Appendix D and discuss further practical issues of CN-DPM in Appendix B
Algorithm Training of the Continual Neural Dirichlet Process Mixture (CD-NDP) Model
Require: Data (x1, (11,9)), u1), (I yN), concen-
12:
M < {an) { UMM
tration a, base measure Go. short-term
13:
if [M| 2 M then {Add new expert)
memory capacity M, learning rate X
14:
òK+1 < FindMAP(M, Go)
1: MM < (Short-term memory}
15:
NK+1 < [M]: M +
2: K < 0 [Number of experts)
16:
K+ +1
3: No f at 6o f (ample((Go)
17:
end if
4: for n = 1 to N do
18:
else {Update existing experts}
5:
for k = 0 to do
19:
Pn.1:K + Pn.1:K/ Ek= Pn. for k = to K do
6:
1 + P(yn lzni ob )p)))) 62)
20:
7:
Pa.k f Nula
21:
N& - N₂ Pr,k
8:
end for
22:
du + 2u + Pu,XAV &y logla
9:
Pn.0:K + Pn,0:k/ Ex Pr,k
10:
if arg max; Pn.k = 0 then
23:
end for
11:
{Save on to short-term memory}
24:
end if
25:
end for
EXPERIMENTS
We evaluate the proposed CN-DPM model in task-free CL with four benchmark datasets. Appen- dices include more detailed model architecture, additional experiments, and analyses.
4.1 CONTINUAL LEARNING SCENARIOS
A CL scenario defines sequence of tasks where the data distribution for each task is assumed to be differrnnt from others. Below we describe the fise CL scenarios used in the experiments. At both train and test time, the model cannot access the task information. Unless stated otherwise, each task is presented for a single epoch (i.e., a completely online setting) with a batch size of 10.
Split-MNIST (Zenke al. 2017). The MNIST dataset (LeCun et al. 1998) is split into five tasks, each containing approximately I2K images of two classes, namely (0/1, 2/3, 4/5, 6/7, 8/9). We conduct both classification and generation experiments in this scenario.

---

Published as a conference paper at ICLR 2020
Table Test scores and the numbers of parameters in task-free CL on Split-MNIST, MNIST-SVHN, and Split- 100 scenarios. Note that iid- baselines are not CL methods.
Split-MNIST Acc. (%)
Split-MNIST (Gea.)
MNIST-SVHN Acc. (%)
Split-CIFAR100
Method
Param.
bits/dim
Param.
Param.
Acc. (%)
Param.
iid-offliee
98.63
478K
0.1806
988K
96.69
11.2M
73.80
11.2M
iid-onliee
96.18
478K
0.2156
988K
95.24
11.2M
20.46
11.2M
Fine-tune
19.43
478K
0.2817
988K
83.35
11.2M
2.43
11.2M
Reservoir
85.69
478K
0.2234
988K
94.12
11.2M
10.01
11.2M
CN-DPM
93.23
524K
0.2110
970K
94.46
7.80M
20.10
19.2M
Table 2: Performance comparison Split-IIFAR10 with various scenario length.
online Fin tine
GSS
Reservatr
Accuracy
30
DPN
Split-CIFAR10 Acc. (%)
25
Method
0.2 Epoch
Epoch
10 Epochs
Param,
20
iid-offline
93.17 36.65
93.17
93.17
11.2M
tid-online
62.79
83.19
11.2M
0K
4K
6K
8K
10K
Learned examples
Fine-tune
12.68
18.08
19.31
11.2M
Reservoir
37.00
44.00
43.82
11.2M
Figure 2: Split-CIFAR10 (0.2 Epoch).
GSS
33.56
11.2M
20
CN-DPM
41.78
45.21
46.98
4.60M
Accuracy (%)
Table 3: Dissecting the performance of CN-DPM.
Acc. Type
SplitCIIFARI0 Split-CIFAR100
Classifier (init)
88.20 20
55.42
0K
10K
20K
30K
40K
50K
Classifier (final)
88.20
55.24
Learned examples
Gating (VAEs)
48.18 18
31.14
Figure 3: Split-CIFAR
MNIST-SVHN (Shin et al. 2017). It is two-stage scenario where the first consists of MNIST, and second contains SVHN Netzer et al. (0TT). This scenario different from Split-MNIST; in Split-MNIST, new classes are introduced when transitioning into new task, whereas the two stages in MNIST-SVHN share the same set of class labels and have different input domains.
In Split-CIFAR1O, split CIFARIO (Krizhevsky & Hin- ton 2009) into five tasks the same manner as Split-MNIST. For Split-CIFAR 100, we build 20 tasks, each containing five classes according to the pre-defined superclasses in CIFAR100. The training sets of CIFAR1O and CIFAR 100 consist 50K examples each. Note that most of the pre- vious works (Rebuffi et 2017 Zenke al.! 2017: Lopez-Paz & Ranzato 2017 Aljundi al.) 2019c Chaudhry et al.) 2019a), except Maltoni & Lomonaco (2019), use task information at test time in Split-CIFAR 100 experiments. They assign distinct output heads for each task and utilize the task identity to choose the responsible output head at both training and test time. Knowing the right output head, however, the task reduces to 5-way classification. Therefore, our setting is far more difficult than the prior works since the model has to perform 100-way classification only from the given input.
Split- CIFAR10 and Split-
4.2 COMPARED METHOOS
All the following baselines use the same base network that will be discussed in section
iid-offline and iid-online. iid-offline shows the maximum performance achieved by combining standard training techniques such as data augmentation, learning rate decay, multiple iterations (up to 100 epochs), and larger batch size. iid-online is the model trained with the same number of epoch and batch size with other CL. baselines.

---

Published as a conference paper at ICLR 2020
Fine-tune. As popular baseline in the previous works, the base model is naively trained as data enters.
Reservoir. As Chaudhry et al. (2019)) show that simple experience replay (ER) can outperform most CL methods, we test the ER with reservoir sampling as a strong baseline. Reservoir sampling randomly chooses fixed number of samples with uniform probability from indefinitely long stream of data, and thus, it is suitable for managing the replay memory in task-free CL. At each training step, the model is trained using a mini-batch from the data stream and another one of the same sizes from the memory.
Cradient-Rased Sample Selection (GSS), Aljundi al. 2019b) propose sampling method called GSS that diversifies the gradients of the samples in the replay memory. Since it is designed to work in task-free settings, we report the scores in their paper for comparison.
4.3 MODEL ARCHITECTURE
Split-MNIST. Following Hsu al. 2018). we use simple two-hidden-layer MLP classifier with ReLU activation as the base model for classification. The dimension of each layer is 400. For genera- tion experiments, we use VAE, whose encoder and decoder have the same hidden layer configuration with the classifier. Each expert in CN-DPM has similar classifier and VAE with smaller hidden dimensions. The first expert starts with 64 hidden units per layer and adds 16 units when new expert is added. For classification, we adjust hyperparameter such that five experts are created. For generation, we set a to produce 12 experts since more experts produce a better score. We set the memory size in both Reservoir and CN-DPM to 500 for classification and 1000 for generation.
We use ResNet- (He et al., 2016) as the base model. In CN-DPM, we use a 10-layer ResNet for the classifier and CNN-based VAF. The encoder and the decoder of VAE have two CONV layers and two FC layers. We set a such that 2, 5, and 20 experts created for each scenario. The memory sizes in Reservoir, GSS, and CN-DPM are set to 500 for MNIST-SVHN and 1000 for Split-CIFAR 10/100. More details can be found in Appendix C
MNIST-SVHN and Split-
4.4 RESULTS OF TASK-FREE CONTINUAL LEARNING
All reported numbers in our experiments are the average of 10 runs. Table I and show our main experimental results. In every setting, CN-DPM outperforms the baselines by significant margins with reasonable parameter usage. Table 2 and Figure 2 shows the results of Split-CIFAR10 exper- iments. Since Aljundi et al. (2019b) test GSS using only 10K examples of CIFARIO, which is 1/5 of the whole train set, we follow their setting (denoted by 0.2 Epoch) for a fair comparison. We also test Split-CIle Colit-CIFARIOO variant where each task is presented for 10 epochs. The accuracy and the training graph of GSS are excerpted from the original paper, where the accuracy is the average of three runs, and the graph is from one of the runs. In Figure 2 the bold line represents the average of 10 runs (except GSS, which is a single run), and the faint lines are the individual runs. Surprisingly, Reservoir even surpasses the accuracy of GSS and proves be a simple but powerful CL method.
One interesting observation in Table 2 is that the performance of Reservoir degrades as each task extended up 10 epochs. This is due to the nature of replay methods: since the same samples are replayed repeatedly as representatives of the previous tasks, the model tends to be overfitted to the replay memory as training continues. This degradation is more severe when the memory size is small, as presented in Appendix ! Our CN-DPM, on the other hand, uses the memory to buffer recent examples temporarily, so there is no such overfitting problem. This is also confirmed by the CN-DPM's 's accuracy consistently increasing as learning progresses.
In addition, CN-DPM is particularly strong compared to other baselines when the number of tasks increases. For example, Reservoir, which performs reasonably well in other tasks, scores poorly in Split-CIFAR 100, which involves 20 tasks and 100 classes. Even with the large replay memory of size 1000, the Reservoir suffers from the shortage of memory (e.g., only 50 slots per task). In contrast, CN-DPM's accuracy is more than double of Reservoir and comparable to that of iid-online.
Table 3 analyzes the accuracy of CN-DPM in Aplit-CIFARIO/100 We assess the performance and forgetting of individual components. At the end of each task, we measure the test accuracy of the re- sponsible classifier and report the average of such task-wise classifier accuracies as Classifier (init).
8

---

Published as a conference paper at ICLR 2020
We report the average of the task-wise accuracies after learning all tasks Classifier (final). With little difference between the two scores, we confirm that forgetting barely occurs in the classifiers. In addition, report the gating accuracy measured after training as Gating (VAEs), which is the curacy of the task identification performed jointly by the VAEs. The relatively low gating accuracy suggests that CN-DPM has much room for improvement through better density estimates.
Overall, CN-DPM does not suffer from catastrophic forgetting. which a major problem in regu- larization and replay methods. As trade-off, however, choosing the right expert arises another problem in CN-DPM. Nonetheless, the results show that this new direction is especially promising when the number of tasks very large.
CONCLUSION
In this work, we formulated expansion- -assd task-free CL as learning Dirichlet process mix- ture model with neural experts. We demonstrated that the proposed CN-DPM model achieves great performance in multiple task-free settings, better than the existing methods. We believe there are several interesting research directions beyond this work: (i) improving the accuracy of expert selec- tion, which is the main bottleneck of our method, and (ii) applying our method to different domains such natural language processing and reinforcement learning.
ACKNOWLEDGMENTS
We thank Chris Dongjoo Kim and Yookoon Park for helpful discussion and advice. This work was supported by Video Analytics Center of Excellence in AIX center of SK telecom, Institute of Information & communications Technology Planning & Evaluation (IITP) grant (No.2019-0-0100) SW StarLab) and Basic Science Research Program through National Research Foundation Korea (NRF) funded by the Korea government (MSIT) (2017RIE1ALA0107733) Gunhee Kim is the corresponding author.
REFERENCES
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with network of experts. In CVPR, 2017.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Mareus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In ECCV, 2018.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-Free continual learning. In CVPR, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In NeurlPS, 2019b.
Rahaf Aljundi, Marcus Rohrbach, and Tinne Tuytelaars. Selfless sequential learning. In ICLR, 2019c.
Charles Antoniak. Mixtures of dirichlet processes with applications bayesian nonparametric problems. Ann. Stat., 2(6): 1974.
David Blei and Michael Jordan. Variational inference for dirichlet process mixtures. Bayesian Anal., 1(1):121-143, 2006.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. ICLR, 2015.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In ICLR, 2019a.
Arslan Chaudhry, Mareus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv, (1902. 10486v4), 2019b.

---

Published as a conference paper at ICLR 2020
Michael D. Escobar and Mike West. Bayesian density estimation and inference using mixtures. J. Am. Stat. Assoc., 90(430):577-588 1995.
Thomas S. Ferguson. Bayesian density estimation by mixtures of normal distributions. In Recent advances in statistics, pp. 287-302. Academic Press, 1983.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learn- ing scenarios: A categorization and case for strong baselines. In NeurlPS, Continual Learning Workshop, 2018.
Robert A. Jacobs, Michael Jordan, Steven J. Nowlan, and Geoffrey Hinton, Adaptive mixtures of local experts. Neural Comput., 3:79 3.79-87, 1991.
Ghassen Jerfel, Erin Grant, Thomas Griffiths, and Katherine Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. In NeurlPS, 2019.
Diederik P. Kingma and Max Welling. Auto-Encoding variational bayes. In ICLR, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. PNAS, 2017.
Demis Hass-
Krizhevsky and G Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2222 1998.
Xilai Li, Yingho Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: continual structure learning framework for overcoming catastrophic forgetting. In ICML, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. JEEE TPAMI, 40(12):2935-2447, 2017.
Dahua Lin. Online learning of nonparametric mixture models via sequential variational approxima- tion. In NeurlPS, 2013.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In NeurIPS, 2017.
Steven Maceachern. Estimating normal means with conjugate style dirichlet process prior. Com- mun. Stat. - Simul. Comput., 23(3):727-741, 1994.
Davide Maltoni and Vincenzo Lomonaco. Continuous learning in single-incremental-takk scenario Neural Networks, 116:56-73, 2019.
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via Meta-Learning: continual adaptation for Model-Based RL. In ICLR, 2019.
Radford M. Neal. Markov chain sampling methods for dirichlet process mixture models. J. Comput. Graph. Stat., 2000.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NeurIPS, Workshop Deep Learning and Unsupervised Feature Learning, 2011.
Aaron Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks, In ICML, 2016.
10

---

Published as a conference paper at ICLR 2020
Genman Parisi, Ronald Kemker, Jose L. Part, and Christopher Kanan. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Tch, and Raia Hadsell. Continual unsupervised representation learning. In NeurIPS, 2019.
Carl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of gaussian process experts. In NeurlPS, 2002.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph Lampert. iCaRL: in- cremental classifier and representation learning. In CVPR, 2017.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. In NeurIPS, 2016.
Jonathan Schwarz, Jelena Luketina, Wojciech Czarnecki, Agnieszka Crabska-Barwingg Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress compress: scalable framework for contin- ual learning. In ICMI,, 2018.
Babak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. J. Mach. Learn. Res., 2009.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The Sparsely- Mixture-of-Experts layer. In ICLR, 2017.
Hanul Shin, Jung Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay, In NeurIPS, 2017.
Yee whye Tch. Dirichlet process. Springer, Encyclopedia of Machine Lcarning:200-2877 2010.
Gido M. van de Ven and Andreas S. Tolias. Generative replay with feedback connections as general strategy for continual learning. arXiv, 10635v2),,,,), 2018.
Lianming Wang and David Dunson. Fast bayesian inference in dirichlet process mixture models. J.
Comput, Graph. Stat., 20(1):196 2269,6226666666 2011.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In NeurIPS, 2018.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In ICLR, 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML, 2017.

---

Published as a conference paper at ICLR 2020
A REVIEW OF DIRICHLET PROCESS MIXTURE MODEL
We review the Dirichlet process mixture (DPM) model and a variational method to approximate the posterior of DPM models in an online setting: Sequential Variational Approximation (SVA) (Lin 2013).
Dirichlet Process. Dirichlet process (DP) is distribution over distributions that defined over infinitely many dimensions. DP is parameterized by a concentration parameter a E R and a base distribution Go. For distribution G sampled from DP(a, Go), the following holds for any finite measurable partition (A1, A2.-.,, Ax] of probability space θ (Teh) 2010):
(G(A1), G(Ax)) ~ Dir(aGo(AA)
(o((A)))
The stick breaking process is often used as more intuitive construction of DP:
k-1 Uk (I -(-))) u1) 60 ug ~ Be)a(1,a),(4 ~ Go.
G G=
(10)
Initially, we start with a stick of length one, which represents the total probability. At each step k; we cut a proportion U) off from the remaining stick (probability) and assign it to the atom dy. sampled from the base distribution Go. This formulation shows DP is discrete with probability In our problem setting, G is distribution over expert's parameter space and has positive probability only at the countably many Φx. which are independently sampled from the base distribution.
2010).
Dirichlet Process Mixture (DPM) Model. The DPM model is often applied to clustering problems where the number of clusters is not known in advance. The generative process of DPM model is
In ~ p(0n), On ~ G. G~ DP(a, Go),
(1I)
where Iy the n-th data, and On the n-th latent variable sampled from G, which itself distribution sampled from Dirichlet process (DP). Since G is discrete with probability 1, the same values can be sampled multiple times for 0. If 0₂ = Om. the two data points on and um belong to the same cluster. An alternative formulation uses the indicator variable 2n that indicates to which cluster the nith data belongs such that O, o=.. where Φ% is the parameter of k-th cluster. The data In is sampled from distribution parameterized by 8,, For DP Gaussian mixture model as example, each
o²} parameterizes a Gaussian distribution.
The Posterior of DPM Models. The posterior of a DPM model for given 01...., On is also DP Teh 2010):
n 5(0.)
GI On ~ DP a a n, Go a +
(12)
i=1
The base distribution of the posterior, which is weighted average of Go and the empirical distribu- tion >m= 8((:), is in fact the predictive distribution of 6,+1 given 0m... (Teh 2010):
On 0 p 10
Go
8(0,).
(13)
If we additionally condition x,, and reflect the likelihood, we obtain Neal 2000
Bxt1 10,, In+1 Z + ((Zx+110)dGG((0) +
a
p(æ p(en+110.)8()))
(14)
where Z is the normalizing constant. Note that 0p+1 is independent from 2i.n given 0
Approximation of the Posterior of DPM Models. Since the exact inference of the posterior of DPM models is infeasible, approximate inference methods are adopted such as Markov chain Monte
Carlo (MCMC) (Maceachern) 1994 Escobar & West 1995; Neal 2000) or variational inference
(Blei & Jordan 2006 Wang & Dunson 20111 Lin (013). Among many variational methods, the
Sequential Variational Approximation (SVA) (Lin 2013) approximates the posterior as
p(G[ærm:)= = Σ p(zim (1m) 21m)
q(Gpp,v) = ( Pi.z: al (G) ([)
(15)
12

---

Published as a conference paper at ICLR 2020
where p(21: (1:m) is represented by the product of individual variational probabilities Ps. for 24, which greatly simplifies the distribution. Moreover, [GG Z1:m) is approximated by a stochastic process g0 (G)). Sampling from 9!²)) (G) is equivalent constructing a distribution
K B0D' Σ Brdser D' DP(aGo). 3x) Dir(a, (of(),//("))) (o.((.1 6b Mk. (16)
where (C+.) c. ) is the partition of #imm characterized by z.
The approximation yields the following tractable predictive distribution:
E Pilk
q(0' 1) E. [[(0)) a Go
vx")
(17)
a+n
SVA uses this predictive distribution for sequential approximation of the posterior of a and ø.
= 1((n=1122+11 o ((++1) 1) (g(2+1,, (n+1)) lpim, bl))). p(2n11, ((++1) [1m+1) (
x p(zn+ 12 (l (2(+)), )l ((n+1) (z1mm)
(18)
(19)
While the data is given one by one, SVA sequentially updates the variational parame- ters; the following Pn+1 and y(+1)) step + minimizes the KL divergence between (Zn+1, (((+1) p1m- ((442)) and the posterior:
ppn-
(E=1Pt,) (0) (d)
if Fisksl K if K +1
Pn+1.
(20)
(n+1) Vk
(d0)
Go(do) I ())L. Gold0)p( (( (0) P0t..
if k if = K+
(21)
In practice, SVA adds a new component only when Pn+1,K+1 is greater than threshold e. It uses stochastic gradient descent to find and maintain the MAP estimation of parameters instead of calculating the whole distribution Vyz
(n+1) + Bi + (n 300 log Gold()) ((m)) + ok
(6n log 1oxp(1)) ((1)),
(22)
where (n) is a learning rate of component at step n, which decreases as in the Robbins-Monro algorithm.
B PRACTICAL ISSUES OF CN-DPM
CN-DPN is designed based strong theoretical foundations, including the nonparametric Bayesian framework. In this section, we further discuss some practical issues of CN-DPM with intuitive explanations.
Bounded expansion of CN-DPM. The number of components in the DPM model is determined by the data distribution and the concentration parameter. If the true distribution consists of K clusters, the number of effective components converges to K under an appropriate concentration parameter a. Typically, the number of components is bounded by ((a log N) (Teh) 2010). Experiments in Appendix empirically show that CN-DPM does not blindly increase the number of experts.
The continued increase model capacity. Our model capacity keeps increasing as it learns new tasks. However, we believe this is one of the strengths of our method, since it may not make sense to use a fixed-capacity neural network to learn an indefinitely long sequence of tasks. The underlying assumption of using fixed-capacity model is that the pre-set model capacity is adequate (at least not insufficient) to learn the incoming tasks. On the other hand, CN-DPM approaches the problem in different direction: start small and add more needed. This property is essential in task-free settings where the total number tasks is not known. If there are too many tasks than expected, a fixed-capacity model would not be able learn them successfully. Conversely, if there are fewer
13

---

Generality the concentration parameter. The concentration parameter controls how sensitive the model is new data. In other words, it determines the level discrepancy between tasks, that makes the tasks modeled by distinct components, As an example, suppose that we are designing hand-written alphabet classifier that continually learns in the real world. In the development, we only have the character images for half of the alphabets, i.c. from 'a' to 'm'. If we can find good concentration parameter a for the data from 'a' 'm', the same can work well with novel Published as a conference paper at ICLR 2020
tasks than expected, resources would be wasted. We argue that expansion is promising direc- tion since it does not need to fix the model capacity beforehand. Moreover, we also introduce an algorithm to prune redundant experts in Appendix
alphabets (i.e., from 'n' to 'z') because the alphabets would have similar level of discrepancies between tasks. Therefore, we not need to access the whole data determine a if the discrepancy between tasks is steady.
C MODEL ARCHITECTURES AND EXPERIMENTAL DETAILS
C.I BASE MODELS
C.I.1 SPLIT-MNIST
Following Hsu et al. (2018), we use two-hidden-layer MLP classifier with 400 hidden units per layer. For generation tasks, we simple VAE with the two-hidden-layer MLP encoder and decoder, where each layer contains 400 units. The dimension of the latent code is set to 32. We use ReLU for all intermediate activation functions.
C.1.2 MNIST-SVHN AND SPLIT- SPLIT-CIFARIA CIFARIO/100
We use ResNet-18 (He et al. 2016). The input images are transformed to 32x32 RGB images.
C.2 CN-DPM
C.2.1 SPLIT-MNIST
For the classifiers in experts, use smaller version of the base MLP classifier. In the first expert, we set the number of hidden units per layer to 64. In the second or later experts, we introduce 16 new units per layer which connected the lower layers of the existing experts. For the encoder and decoder of VAEs, we use two-layer MLP. The encoder is expanded in the same manner as the classifier. However, we do not share the parameters beyond the encoders; with a latent code of dimension 16, we use the two-hidden-layer MIP decoder as done in the classifier. For generation tasks, we double the size; for example, we the size of initial and additional hidden units 128 and 32, respectively.
C.2.2 SPLIT-CIFARIO/I
The ResNet-18 18 base network has eight residual blocks. After passing through 2 residual blocks, the width and height of the feature are halved, and the number of channels is doubled. The initial number channels is set to 64.
For the classifiers in CN-DPM, we use a smaller version of ResNet that has only four residual blocks and resizes the feature every block. The initial number of channels set to 20 in the first expert, and added for the
four initial channels are added with new expert. Thus, 4, 8, 16, and 32 channels four blocks. The first layer of each block is connected to the last layer of the previous block of prior experts.
For the VAEs, we use a simple CNN-based VAEs. The encoder has two 3x3 convolutions followed by two fully connected layers. Each convolution is followed by 2x2 max-pool and ReLU activation. The numbers of channels and hidden units are doubled after each layer. In the first expert, the first convolution outputs 32 channels, while four channels are added with each new expert.
14

---

Published as a conference paper at ICLR 2020
As done for the VAE in Split-MNIST, each expert's VAE has an unshared decoder with 64- dimensional latent code. The decoder is the mirrored encoder where 3x3 convolution is replaced by 4x4 transposed convolution with stride 2.
C.2.3 MNIST-SVHN
For the classifier, we use ResNet-18 with 32 channels for the first expert and additional 32 channels for each new expert. We use the same VAE as in Split-CIFARIO
C.3 EXPERIMENTAL DETAILS
We use the classifier temperature parameter of 0.01 for Split-MNIST, Spli-CIFAR temperature parameter on MNIST-SVHN. Weight decay 0.00001 has been used for every model in the paper. Gradients are clipped by value with threshold of 0.5. All the CN-DPM models are trained by Adam optimizer. During the sleep phase, we train the new expert for multiple epochs with batch size of 50. In classification tasks, we improve the density estimation of VAFs by sampling 16 latent codes and averaging the ELBOs, following Burda et al. (2015).
and no
C.3.1 SPLIT-MNIST
The learning rate of 0.0001 and 0.0004 has been used for the classifier and VAE of each expert in the classification task. We learning rate 0.003 for the VAE of each expert generation task. In the generation task, we decay the learning rate of the expert by 0.003 before it enters the wake phase. Following the existing works in VAE literature, we use binarized MNIST for the generation experiments. VAEs are trained to maximize Bernoulli logliikeldoood in the generation task, while Gaussian log-likelihood is used for the classification task.
C.3.2 SPLIT-CIFAR10
The learning rate 0.005 and 0.0002 has been used for the classifier and VAE each expert in CIFAR1O. We decay the learning rate of the expert by 0.1 before it enters the wake phase. VAEs are trained to maximize Gaussian fog-likelihood
C.3.3 SPLIT-CIFAR10
The learning rate of 0.0002 and 0.0001 has been used for the classifier and VAE of each expert in CIFARIO. 10. We decay the learning rate of the expert by 0.2 before it enters the wake phase. VAEs trained to maximize Gaussian log-likell ligliikl
C.3.4 MNIST-SVHN
The learning rate of 0.0001 and 0.0003 has been used for the classifier and VAE of each expert in CIFARIO. We decay the learning rates classifier and VAE of each expert by 0.5 and 0.1 before enters the wake phase. VAEs trained to maximize Gaussian lideiio....od.
D PRUNING REDUNDANT EXPERTS
Lin (2013) propose a simple algorithm prune and merge redundant components in DPM models. Following the basic principle of the algorithm, we provide pruning algorithm for CN-DPM. First, we need to measure the similarities between experts to choose which expert to prune. We compute the log-likelihood Lnk = p(u.. yn )x) of each expert & for data (II:N (1:an. As result, we can obtain K vectors with N dimensions. We define the similarity s(k; k') between two experts k and k L. as the cosine similarity between the two corresponding vectors 1., and L.r,, i.e., s(k,k)= = [udلT the similarity is greater than a certain threshold c, we remove one of the experts with smaller N₂ Σn Pa.k. The N₃ data of the removed expert are added the remaining experts.
Figure 4 shows an example of an expert pruning. We test CN-DPM on Split-MNIST with an a higher than the optimal value such that more than five experts created. In this case, seven experts
15

---

Published as a conference paper at ICLR 2020
created. If we build similarity matrix shown in Figure 4b
which pair of experts
are similar. We then threshold the matrix at 0.9 in Figure 4c and choose expert pairs (2/3) and (5/6) for pruning. Comparing N& within each pair, we can finally choose prune expert and 6. After pruning, the test accuracy marginally drops from 87.07% % to 86.01%.
Accuracy (%)
1
2
3
4
5
6
7
1
2
60
1
40
2
20
3
0-
4
4
4-
5
6
6
0K
12K
24K
36K
48K
60K
7
7
Learned examples
(a) Training curve
(b) Similarity matrix
(c) Thresholded similarity
matrix
Figure 4: An example of the expert pruning in the Split-MNIST scenario.
COMPARISON WITH TASK-BASED METHODS ON SPLIT-MNIST
Table compares our method with task-based methods for Split-MNIST classification. All the numbers except for our CN-DPM are excerpted from Hsu et al. (2018), in which all methods are trained for four epochs per task with a batch size of 128. Our method is trained for four epochs task with batch size of 10. The model architecture used in compared methods is the same as our baselines: a two-hidden-layer MLP with 400 hidden units per layer. All compared methods single output head, and the task information is given at training time but not at test time. For CN-DPM, we test two training settings where the first one uses task information to select experts, while the second one infers the responsible expert by the DPM principle. Task information is not given at test time both cases.
Notice that regularization methods often suffer from catastrophic forgetting while replay methods yield decent accuracies. Even though the task-free condition is far more difficult setting, the performance of our method is significantly better than regularization and replay methods that exploit the task description. task information is available at train time, we can utilize to improve the performance even more.
Table 4: Comparison with task-based methods on Split-MNIST classification. We report the average of 10 runs with + standard error of the mean. The numbers except ours are from Hsu et 2018
Type
Method
Task labels Accuracy (%)
EWC (Kirkpatrick et al. 2017
19.80 + 0.05
Online LWC (SCh (Schwarz el 2018
19.77 1 0.04
Regularization
SI Zenke et al. 2017)
19.67 1 0.09
MAS Allundı et al 2018
19.52 1 0.04
LwF [i Hoıem 2017
24.17 + 0.33
GEM Lopez-Paz & Ranzato 2017
92,20 + 0.12
Replay
DGR (Shin al. 2017)
91.24 + 0.33
RtF (van de Ven & Tolias 2018
92.56 - 0.21
CN-DPM
/
93.81 + 0.07
Expansion
CN-DPM
93.70 + 0.07
Upper bound (iid)
97.53 + 0.30
16

---

Published as a conference paper at ICLR 2020
Table 5: Fuzzy Split-MNIST
task proportion
Cumulative
Task 5
Task
Task
Task
Task
Method
Acc. (%)
Param.
Fine-tune
28.41 0.52
478K
Reservoir
88.64 + 0.48
478K
12K
24K
36K
CN-DPM
93.22 0.07
524K
Learned examples
Figure 5:
Scenario configuration of
Fuzzy Split-MNIST
Label::
Label:
Label:
Label:
Labal:
Label:
Label: 1
Label:
Label:
Label:
3
a
2
3
2
3
Label:1 1
Label: D
Label: 1
Label: 3
Label: 2
Label: 2
Label:
Label: 3
Label: 2
Label:
5
5
5
S
5
4
5
4
Label: 5
Label:
Label: 5
Label: 4
Label: 5
Label 4
Label: 5
Label: 4
Label: 4
Label:
7
7
6
6
6
8
Label: 7
Label: 7
Label: 7
Label: 7
Label 6
Label 6
Label: 6
Label:
Label:
Label:
9
g
8
8
8
8
Label: 9
Label: 8
Label:
Label:
Label:
Label: 9
Label: g
Label: 8
Label:
Label:
Figure 6: Examples of generation samples by CN-DPM trained on Split-MNIST.
F FUZZY SPLIT-MNIST
In addition, experiment with the case where the task boundaries are clearly defined, which we call Fuczy-Split-MNIS7 Instead of discrete task boundaries, we have transition stages between tasks where the data of existing and new tasks are mixed, but the proportion of the new task linearly increases. This condition adds another level of difficulty since it makes the methods unable to rely on clear task boundaries. The scenario visualized Figure 5 As shown Table [S] CN-DPM can perform continual learning without task boundaries.
G GENERATION OF SAMPLES
Even in discriminative tasks where the goal is to model p(y|z), CN-DPM learns the joint distri- bution p(x, y). Since CN-DPM a complete generative model, it can generate (x, y) pairs. To generate sample, Cal( Ni N₂
first sample from p(2) which is modeled by the categorical distribution N : :.... ), i.e., choose expert. Given k, we first sample ær from the generator p(:; o and then sample y from the discriminator p(y| xf) Figure 6 presents 50 sample ex- amples generated from CN-DPM trained on Split-MNIST for single epoch. We observe that CN-DPM successfully generates examples of all tasks with no catastrophic forgetting.
H EXPERIMENTS WITH LONGER CONTINUAL LEARNING SCENARIOS
We present experiments with much longer continual learning scenarios on Split-MNIST, Split- CIFAR10 and Split-CIFAR with = standard error the mean. To compare with the default I-epoch scenario, we carry out ex- periments that repeat each task 10 times, which denoted 10 Epochs. In addition, also present
CIFAR 100 in Table 6 and respectively. We report the average of 10 runs
17

---

Published as a conference paper at ICLR 2020
the results of repeating the whole scenario 10 times, which are denoted Epoch x10. For example, in Split-MNIST, the 10 Epochs scenario consists of 10-epoch 0/1, 10-epoch 2/3, 10-epoch 8/9 tasks. On the other hand, the 1 Epoch x10 scenario revisits each task multiple times, ie., 1-pooch 0/1, 1-epoch 2/3, ..., l-epoch 8/9, lepocc 0/1, 1-epoch 8/9. We use the same hyperparameters tuned for the 1-epoch scenario.
We find that the accuracy of Reservoir drops as the length of each task increases. As mentioned in the main text, this phenomenon seems be caused by overfitting on the samples in the replay memory. Since only a small number of examples in the memory represent each task, replaying them for a long period degrades the performance. On the other hand, the performance of our CN-DPM improves as the learning process is extended.
In the 1 Epoch x10 setting, CN-DPM shows similar performance with 10 Epoch since the model sees each data point 10 times in both scenarios. On the other hand, Reservoir's 's scores in the / Epoch x10 largely increase compared to both Epock and 10 Epoch This difference can be explained by how the replay memory changes while training progresses. In the 10 Epoch setting, if task is finished, it is not visited again. Therefore, the examples the task in the replay memory monoton- ically decreases, and the remaining examples are replayed repeatedly. As the training progresses, the model is overfitted to the old examples in the memory and fails to generalize in the old tasks. In contrast, in Epoch x x10 setting, each task is revisited multiple times, and each time task is revis- ited, the replay memory is also updated with the new examples of the task. Therefore, the overfitting problem in the old tasks is greatly relieved.
Another important remark is that CN-DPM does not blindly increase the number of experts. If add new expert at every constant steps, would have 10 times more experts in the longer scenarios. However, this is not the case. CN-DPM determines whether it needs a new expert on a data-by-data basis such that the number of experts is determined by the task distribution, not by the length of training.
Table Experiments with longer training episodes on Split-MNIST
Epoch Acc. (%)
10 Epochs
Epnch x10
Method
Param.
Acc. (%)
Param.
Acc. (%)
Param.
iid-offline
98.63 0,01
478K
98.63 + 0.01
478K
98.63 0.01
478K
iid-online
96.18 + 0.19
478K
97.67 + 0.05
478K
97.67 + 0.05
478K
Fine-tune
19.43 0.02
478K
19.68 + 0.01
478K
20.27
0.26
478K
Reservoir
85.69 1 0.48
478K
78.82 1 0.71
478K
92.06 1: 0.11
478K
CN-DPM
93.23 0.09
524K
94.39 1 0.04
524K
94.15 0.04
616K
Table 7: Experiments with longer training episodes on Split-CIFARI0
Epoch
10 Epochs
Epoch x10 Acc. (%)
Method
Acc. (%)
Param.
Acc. (%)
Param.
Param.
iid-offline
93.17 + 0.03
11.2M
93.17 + 0.03
11.2M
93.17 + 0.03
11.2M
iid-online
62.79 + 1.30
11.2M
83.19 + 0.27
11.2M
83.19 + 0.27
11.2M
Fine-tune
18.08 1 0.13
11.2M
19.31 1 0.03
11.2M
19.33 1 0.03
11.2M
Reservoir
44.00 1 0.92
11.2M
43.82
1 0.53
11.2M
51.44 1: 0.42
11.2M
CN-DPM
45.21 + 0.18
4.60M
46.98 1 0.18
4.60M
47.10 + 0.16
4.60M
I EXPERIMENTS WITH DIFFERENT MEMORY SIZES
In experiments in the main text, we set the memory size of Reservoir and CN- DPM to 1000, following Aljundi et al. (2019b). Table 9 compares the experimental results with different memory sizes of 500 and 1000 on Split-IIFR 10/100. Compared to Reservoir, whose per- formance drops significantly with smaller memory, CN-DPM's accuracy drop is relatively marginal.
18

---

Published as a conference paper at ICLR 2020
Table Experiments with longer training episodes on Split-CIFAR100
Epoch Acc. (%)
10 Epochs
Epech x 10
Method
Param.
Acc. (%)
Param.
Acc. (%)
Param.
iid-offline
73.80 1: 0.11
11.2M
73.80 +: 0.11
11.2M
73.80 1: 0.11
11.2M
iid-online
20.46 0.30
11.2M
54.58 + 0.27
11.2M
54.58
0.27
11.2M
Fine-tune
2.43 + 0.05
11.2M
3.99 + 0.03
11.2M
4.30 1: 0.02
11.2M
Reservoir
10.01
0.35
11.2M
6.61 + 0.20
11.2M
14.53
0.35
11.2M
CN-DPM
20.10 0.12
19.2M
20.95 + 0.09
19.2M
20.67
0.13
19.2M
Table 9: Experiments with different memory sizes.
Acc. (%) 10 Epoch
Split-CIFAR100 Acc. (%) Epoch
Method
Memory
Epoch
10 Epoch
Reservoir
500
33.53
1.03
34.46 + 0.49
6.24 0.25
4.99 + 0.09 20.77 + 0.11
CN-DPM
500
43.07 + 0.16
47.01 + 0.22
19.17 + 0.13
Reservoir
1000
44.00
0.92
43.82 = 0.53
10.01 0.35
6.61 + 0.20
CN- CN-DPM
1000
45.21 + 0.18
46.98 = 0.18
20.10 + 0.12
20.95 + 0.09
THE EFFECT OF CONCENTRATION PARAMETER
IO]sows the results CN-DPM on Split-MNIST classification according to the concentration parameter which defines the prior of how sensitive CN-DPM is to new data. With higher an expert tends to be created more easily. In the experiment reported in the prior sections, we set log a -000. At log -600, too few experts created, and the accuracy is rather low. As increases, the number of experts grows along with the accuracy. Although the CN-DPM model task-free and automatically decides the task assignments experts, still need to tune the concentration parameter to find the best balance point between performance and model capacity, as all Bayesian nonparametric models require.
100
Accuracy
60
40
Table 10: The effects of concentration parameter
CN-DPM (loga 600)
0
CN-DPM (loga= - 400)
(M)
1.5
CN-DPM (loga 80)
1.0
log
Acc. (%)
Experts
Param.
0.5
-600
54.04 + 2.22
3.20 + 0.13
362K
0.0
15
400
93.23 + 0.09
5.00 + 0.00
524K
10
80
93.54 + 0.21
14.4 + 1.35
1.44M
5
0
24K
Learned examples
Figure/ 7: The effects of concentration pa-
rameter a.
K THE EFFECT OF PARAMETER SHARING
Table II compares when the parameters shared between experts and when they not shared. By sharing the parameters, we could reduce the number parameters by approximately 38% with- out sacrificing accuracy.
19

---

Published as a conference paper at ICLR 2020
Table 11: The effects of parameter sharing.
Model
Acc. (%)
Experts
Param.
CN-DPM
93.23 + 0.09
524K
CN-DPM w/o PS 93.30 + 0.24
5
839K
L TRAINING GRAPHS
Figure 8 shows the training graphs of our experiments. In addition to the performance metries, we present the number of experts in CN-DPM and compare the total number of parameters with the baselines. The bold lines represent the average of the 10 runs while the faint lines represent individual runs.
Figure 9 and Figure shos how the accuracy of each task changes during training. We also present the average accuracy of learned tasks at the bottom right.
M COMPARISON WITH THE CURL
Continual Unsupervised Representation Learning (CURL) (Rao et al. 2019) is a parallel work that shares some characteristics with CN-DPM in terms of model expansion and short-term memory. However, there are several key differences that distinguish our method from CURL, which will be elaborated in this section. Following the notations of Rao et al. (2019). here y denotes the cluster assignment, and 2 denotes the latent variable.
The Generative Process. The primary goal of CURL continually learn a unified latent rep- resentation which is shared across all tasks. Therefore, the generative model of CURL explicitly consists of the latent variable as summarized follows:
p(r; y, (()))=
where y Cat(z), ((
(p=))))l(((
The overall distribution of is the mixture of Gaussians, and includes the information of such that æ and y are conditionally independent given z. Then, z is fed into a single decoder network Me to generate the mean of which is modeled by Bernoulli distribution. On the other hand, the generative version of CN-DPM which does not include classifiers, has a simpler generative process:
p(a, v) p(y)p(xly) where y ~ Cat(x), x p(x
The choice of p(alyy) here is necessarily restricted to VAEs (Kinmma & Welling 2014): one may use other kinds of explicit density models such as PixeIRNN (Oord et al. 2016). Even we use VAEs to model p(r y). the generative process is different from CURL:
p(r, v: p(y)p(())))ppp))p (pppp(((ppppp)))))) (p)y),)))) 2) where Cat(a), N(0 I), Bernoull(p((z)).
Unlike CURL, CN-DPM generates and independently and maintains a separate decoder μnl for each cluster y/.
2. The Necessity for Generative Replay in CURL CURL periodically saves a copy of its pa- rameters and use generate samples of learned distribution. The generated samples are played together with new data such that the main model does not forget previously learned knowledge. This process is called generative replay. The generative replay is an essential element in CURL, unlike CN-DPM. CURL assumes factorized variational posterior q 2|2) (2)))))))2(z) y) where q(y(z) and q(2 xr. y) are modeled by separate output heads of the encoder neural network. However, the output head for q(y]x=) is basically gating network that could be vulnerable catastrophic for- getting, as mentioned in Section 3.1 Moreover, CURL. shares a single decoder (4 across all tasks. As consequence, expansion alone is not enough to stop catastrophic forgetting, and CURL needs another CL method to prevent catastrophic forgetting in the shared components. This is the main reason why the generative replay is crucial in CURL. As shown in the ablation of Rao et al. 2019), the performance of CURL drops without the generative replay. In contrast, the components of CN-DPM are separated for each task (although they may share low-level representations) such that no additional treatment is needed.
20

---

Published as a conference paper at ICLR 2020
100
Id-oniine
80
0.40
Accuracy
60
Reservair
0.35
CN-DPM
40
0.20
5.0
250
Experss Param,
0.5
0.0
Experts
12
0K
12K
24K
36K
48K
60K
0K
12K
24K
36K
48K
60K
Learned examples
examples
(a) Split-MNIST
(h) Split-MNIST (Gen.)
100
90
Experts Param. (M) Accuracy
70
Accuracy
60
30
50
20
40
12
10
Experts Param.
20K
40K
60K
80K
100K
120K
0K
10K
30K
50K
Learned examples
Learned examples
(c) MNIST-SVHN
(d) Split-CFFAR10
15
20
15
10
Experts
15
10
10K
30K Learned examples
(e) Split-CIFARION
Figure 8: Full training graphs.
21

---

Published as a conference paper at ICLR 2020
100
60
60
Task
60
20
20
0
0
T1
T2
T3
T4
T5
T1
T2
13
T4
T1
T2
T3
T4
T5
100
Finc-tune
100
100
Reservoir
60
Average
60
40
40
40
T2
T4
T1
T3
Figure Accuracy for each task in Split-IIFARIO.
60
60
Tack2
Task
0-
T5
710
T15
T10
T15
T20
T5
T10
T20
40
Task
40
Task
40
20
40
Task
40
Task
60
20
20
20
T5
T10
T15
T20
60
T10
T15
T20
T5
T10
T15
T20
11
T2
20
60
T5
710
T15
120
60
T10
T15
120
T5
T10
T15
T20
40
Task
40
Task
T20
40
T5sk17
40
Task
40
20
20
15
110
T15
110
115
120
Fine-ture
Task 20
Average
Reservoir
CN-OPM
20
20
0
T5
T10
T15
T20
T10
T15
120
T5
T10
T15
T20
Figure 10: Accuracy for each task Split-CIFARI00.
22


Reviews:
[{"review_id": "rygcWQYf9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an elegant method for task-free continual learning problems. It has nicely pointed out that the conventional continual learning algorithms had the limitation of knowing the task boundaries. Followings are my summary. \n\nSummary: \nBy applying DPM, the authors proposed a method of automatically determining whether to add a new expert for a new task or train the existing experts. While the Dirichlet Process Mixture (DPM) is not new, applying such nonparametric method to continual learning is new. The experimental results are impressive given the single-epoch setting. \n\nPros:\n1. Good experimental results for the task-free setting, in which no information about task boundaries is given. Particularly, even with smaller memory-usage than the experience replay (ER) methods, the proposed method achieves better results. \n2. Many past work should suffer from increased number of tasks due to the model capacity limit, but the proposed method efficiently expands the model capacity. \n3. Writing flow is good and is easy to follow. \n\nCons & Questions: \n1. I am not sure whether the proposed methods should work well for \"all\" cases. Is there any cases in which the proposed DPM would fail?\n2. What happens when you actually know the task boundaries? Would following the framework with known number of experts also excel other methods?\n3. Can you apply this to the RL setting? \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1x6ECnhYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper proposes to use a Bayesian nonparametric mixture model for task-free (without explicit task labels) continual learning. The main idea is to use an expansion-based model where the number of mixture components (experts) adapts to the training data/tasks. Specifically, a Dirichlet Process Mixture Model (DPMM) consisting of a set of neural network experts is used. Empirical results demonstrate improved performance on three different datasets over some of the baselines. \n\nI find the methodological contribution in the paper to be somewhat limited since the main idea of the model was initially proposed in the prior work (cited in the main paper): Dahual Lin - “Online Learning of Nonparametric Mixture Models via SVA”. In fact, the paper claims its contribution is expansion-based task-free continual learning. However, this “task-free characteristic” is the contribution of SVA based inference. Nevertheless, I do like how the existing SVA based inference has been adapted from an online learning setting to a more general continual learning setting by using various approximations/tricks (like short-term memory with wake-sleep training, point estimates). \n\nPros:\n- The idea of using a nonparametric model for CL is interesting and can lead to follow-up work.\n- Results show that the approach works well.\n- The code has been released.\n\nOverall I am inclining towards voting for acceptance if the authors could address my following questions:\n\n- Could you comment on the creation of test data? It is not clear to me how the model is evaluated if the task-boundaries are not known a priori. Shouldn’t the evaluation be based on tasks? How are you evaluating catastrophic forgetting? I am interested to know what was the test accuracy for the task for which the training data was seen early on during the training.\n\n- It seems to me that the method works on the assumption that the number of data points for each task is at least M (size of STM) and moreover, that these data points appear together sequentially. The method should be sensitive to the size of the STM. How are you choosing M? Would the framework work if data points for each task do not appear together?\n\n- Assuming you have clear task boundaries, how would you adapt this framework? Was the model compared to other methods that assume known task-boundaries like (VCL, EWC, Memory Replay)? \n\nOther comments:\n\n- The method is inspired by a Bayesian framework but calling it Bayesian wouldn’t be fair since only a point estimate is being learned for parameters. This is important to distinguish since there are other methods that are fully Bayesian like Nguyen et. al. “Variational Continual Learning”  (although such methods may have other pros and cons)\n\n- The samples from the base distribution of the posterior (v) are not iid anymore due to lateral connections b/w the representations. Do you think the theoretical result in Appendix B that the number of clusters is upper bounded by O(alpha*logN) is still valid?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Syx2wD4nKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes Continual Neural Dirichlet Process Mixture Model (CN-DPM) to solve task-free continual learning. The core idea is to employ Dirichlet process mixture model to create novel experts in online fashion when task distributions change.  The proposed method is validated on various tasks and demonstrated to perform well compared to the other baselines.\n\nOverall I find this paper to be well-written and the experiments are conducted thoroughly. The method is compared to proper baselines in various settings, and the paper describes detailed experimental settings and architectural choices to help readers willing to reproduce. I’ve gone through the appendix and they provide enough additional experiments to support the author’s claim.\n\nThe main algorithm itself cannot be considered to be novel. DPM or other Bayesian nonparametric models have been extensively used for the problems requiring to adapt the model size according to the change of data. Nevertheless, the application of DPM in task-free continual learning context seems to be considered as a contribution.\n\nI have much experience in implementing Bayesian nonparametric models with parametric distributions and compared various methods to conduct the posterior inference of them. In my experience, even for the low-dimensional parametric models, the posterior inference algorithms for DPM usually suffer from local optima, and the sequential methods such as SVA depends heavily on the data processing order. According to the experimental setting presented in the paper, the algorithm goes through a single pass over the data stream, yet still able to reasonably train (deep) neural networks and identify mixture components jointly. Do you have any intuition about how this becomes feasible?\n\nI don’t fully understand why generative modeling is required. In page 4 the authors stated that the generative model prevents catastrophic forgetting. But in my understanding, using expert-specific parameters is the part that prevents catastrophic forgetting, not the generative model itself. Learning generative model in online fashion may work well in simple structured data such as MNIST,  but I highly doubt that the generative model could be trained properly for CIFAR10 or CIFAR100, especially in online setting. My concern is that learning generative model part may even impede the discriminative learning. Could you elaborate more on this?\n\nAnother minor concern is the way the concentration parameter alpha is selected. The authors stated that they chose proper value of alpha according to the number of tasks known in advance. I think this does not make sense. Alpha should also be inferred along with other parameters, or fixed to non-informative value if the performance of the algorithm is not very sensitive to the choice of alpha.\n\nI think it would be more helpful to show how the task-assignment p(z=k|x) is learned. For instance, the clustering accuracy according to p(z=k|x) against the ground-truth task label can be measured, or at least qualitatively show what examples were assigned to each task. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an expansion-based approach for task-free continual learning, using a Bayesian nonparametric framework (a Dirichlet process mixture model).

It was well-reviewed, with reviewers agreeing that the paper is well-written, the experiments are thorough, and the results are impressive. Another positive is that the code has been released, meaning it’s likely to be reproducible.

The main concern shared among reviewers is the limited novelty of the approach, which I also share. Reviewers all mentioned that the approach itself isn’t novel, but they like the contribution of applying it to task-free continual learning. This wasn’t mentioned, but I’m concerned about the overlap between this approach and CURL (Rao et al 2019) published in NeurIPS 2019, which also deals with task-free continual learning using a generative, nonparametric approach. Could the authors comment on this in their final version?

In sum, it seems that this paper is well-done, with reproducible experiments and impressive results, but limited novelty. Given that reviewers are all satisfied with this, I’m willing to recommend acceptance. 


Decision: Accept (Poster)