A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning

1 INTRODUCTION . Humans consistently encounter new information throughout their lifetime . The way the information is provided , however , is vastly different from that of conventional deep learning where each minibatch is iid-sampled from the whole dataset . Data points adjacent in time can be highly correlated , and the overall distribution of the data can shift drastically as the training progresses . Continual learning ( CL ) aims at imitating incredible human ’ s ability to learn from a non-iid stream of data without catastrophically forgetting the previously learned knowledge . Most CL approaches ( Aljundi et al. , 2018 ; 2017 ; Lopez-Paz & Ranzato , 2017 ; Kirkpatrick et al. , 2017 ; Rusu et al. , 2016 ; Shin et al. , 2017 ; Yoon et al. , 2018 ) assume that the data stream is explicitly divided into a sequence of tasks that are known at training time . Since this assumption is far from realistic , task-free CL is more practical and demanding but has been largely understudied with only a few exceptions of ( Aljundi et al. , 2019a ; b ) . In this general CL , not only is explicit task definition unavailable but also the data distribution gradually shifts without a clear task boundary . Meanwhile , existing CL methods can be classified into three different categories ( Parisi et al. , 2019 ) : regularization , replay , and expansion methods . Regularization and replay approaches address the catastrophic forgetting by regularizing the update of a specific set of weights or replaying the previously seen data , respectively . On the other hand , the expansion methods are different from the two approaches in that it can expand the model architecture to accommodate new data instead of fixing it beforehand . Therefore , the expansion methods can bypass catastrophic forgetting by preventing pre-existing components from being overwritten by the new information . The critical limitation of prior expansion methods , however , is that the decisions of when to expand and which resource to use heavily rely on explicitly given task definition and heuristics . In this work , our goal is to propose a novel expansion-based approach for task-free CL . Inspired by the Mixture of Experts ( MoE ) ( Jacobs et al. , 1991 ) , our model consists of a set of experts , each of which is in charge of a subset of the data in a stream . The model expansion ( i.e. , adding more experts ) is governed by the Bayesian nonparametric framework , which determines the model complexity by the data , as opposed to the parametric methods that fix the model complexity before training . We formulate the task-free CL as an online variational inference of Dirichlet process mixture models consisting of a set of neural experts ; thus , we name our approach as the Continual Neural Dirichlet Process Mixture ( CN-DPM ) model . We highlight the key contributions of this work as follows . • We are one of the first to propose an expansion-based approach for task-free CL . Hence , our model not only prevents catastrophic forgetting but also applies to the setting where no task definition and boundaries are given at both training and test time . Our model named CN-DPM consists of a set of neural network experts , which are expanded in a principled way built upon the Bayesian nonparametrics that have not been adopted in general CL research . • Our model can deal with both generative and discriminative tasks of CL . With several benchmark experiments of CL literature on MNIST , SVHN , and CIFAR 10/100 , we show that our model successfully performs multiple types of CL tasks , including image classification and generation . 2 BACKGROUND AND RELATED WORK . 2.1 CONTINUAL LEARNING . Parisi et al . ( 2019 ) classify CL approaches into three branches : regularization ( Kirkpatrick et al. , 2017 ; Aljundi et al. , 2018 ) , replay ( Shin et al. , 2017 ) and expansion ( Aljundi et al. , 2017 ; Rusu et al. , 2016 ; Yoon et al. , 2018 ) methods . Regularization and replay approaches fix the model architecture before training and prevent catastrophic forgetting by regularizing the change of a specific set of weights or replaying previously learned data . Hybrids of replay and regularization also exist , such as Gradient Episodic Memory ( GEM ) ( Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2019a ) . On the other hand , methods based on expansion add new network components to learn new data . Conceptually , such direction has the following advantages compared to the first two : ( i ) catastrophic forgetting can be eliminated since new information is not overwritten on pre-existing components and ( ii ) the model capacity is determined adaptively depending on the data . Task-Free Continual Learning . All the works mentioned above heavily rely on explicit task definition . However , in real-world scenarios , task definition is rarely given at training time . Moreover , the data domain may gradually shift without any clear task boundary . Despite its importance , taskfree CL has been largely understudied ; to the best of our knowledge , there are only a few works ( Aljundi et al. , 2019a ; b ; Rao et al. , 2019 ) , each of which is respectively based on regularization , replay , and a hybrid of replay and expansion . Specifically , Aljundi et al . ( 2019a ) extend MAS ( Aljundi et al. , 2018 ) by adding heuristics to determine when to update the importance weights with no task definition . In their following work ( Aljundi et al. , 2019b ) , they improve the memory management algorithm of GEM ( Lopez-Paz & Ranzato , 2017 ) such that the memory elements are carefully selected to minimize catastrophic forgetting . While focused on unsupervised learning , Rao et al . ( 2019 ) is a parallel work that shares several similarities with our method , e.g. , model expansion and short-term memory . However , due to their model architecture , expansion is not enough to stop catastrophic forgetting ; consequently , generative replay plays a crucial role in Rao et al . ( 2019 ) . As such , it can be categorized as a hybrid of replay and expansion . More detailed comparison between our method and Rao et al . ( 2019 ) is deferred to Appendix M . 2.2 DIRICHLET PROCESS MIXTURE MODELS . We briefly review the Dirichlet process mixture ( DPM ) model ( Antoniak , 1974 ; Ferguson , 1983 ) , and a variational method to approximate the posterior of DPM models in an online setting : Sequential Variational Approximation ( SVA ) ( Lin , 2013 ) . For a more detailed review , refer to Appendix A. Dirichlet Process Mixture ( DPM ) . The DPM model is often applied to clustering problems where the number of clusters is not known in advance . The generative process of a DPM model is xn ∼ p ( x ; θn ) , θn ∼ G , G ∼ DP ( α , G0 ) , ( 1 ) where xn is the n-th data , and θn is the n-th latent variable sampled from G , which itself is a distribution sampled from a Dirichlet process ( DP ) . The DP is parameterized by a concentration parameter α and a base distribution G0 . The expected number of clusters is proportional to α , and G0 is the marginal distribution of θ whenG is marginalized out . SinceG is discrete with probability 1 ( Teh , 2010 ) , same values can be sampled multiple times for θ . If θn = θm , the two data points xn and xm belong to the same cluster . An alternative formulation uses the variable zn that indicates to which cluster the n-th data belongs such that θn = φzn where φk is the parameter of the k-th cluster . In the context of this paper , φk refers to the parameters of the k-th expert . Approximation of the Posterior of DPM Models . Since the exact inference of the posterior of DPM models is infeasible , approximate inference methods are applied . Among many approximation methods , we adopt the Sequential Variational Approximation ( SVA ) ( Lin , 2013 ) . While the data is given one by one , SVA sequentially determines ρn and νk , which are the variational approximation for the distribution of zn and φk respectively . Since ρn satisfies ∑ k ρn , k = 1 and ρn , k > = 0 , ρn , k can be interpreted as the probability of n-th data belonging to k-th cluster and is often called responsibility . ρn+1 and ν ( n+1 ) at step n+ 1 are computed as : ρn+1 , k ∝ { ( ∑n i=1 ρi , k ) ∫ φ p ( xn+1|φ ) ν ( n ) k ( dφ ) if 1 ≤ k ≤ K α ∫ φ p ( xn+1|φ ) G0 ( dφ ) if k = K + 1 , ( 2 ) ν ( n+1 ) k ( dφ ) ∝ { G0 ( dφ ) ∏n+1 i=1 p ( xi|φ ) ρi , k if 1 ≤ k ≤ K G0 ( dφ ) p ( xn+1|φ ) ρn+1 , k if k = K + 1 . ( 3 ) In practice , SVA adds a new component only when ρK+1 is greater than a certain threshold . If G0 and p ( xi|φ ) are not a conjugate pair , stochastic gradient descent ( SGD ) is used to find the MAP estimation φ̂ with a learning rate of λ instead of calculating the whole distribution νk : φ̂ ( n+1 ) k ← φ̂ ( n ) k + λ ( ∇φ̂ ( n ) k logG0 ( φ̂ ( n ) k ) +∇φ̂ ( n ) k log p ( x|φ̂ ( n ) k ) ) . ( 4 ) DPM for Discriminative Tasks . DPM can be extended to discriminative tasks where each data point is an input-output pair ( x , y ) , and the goal is to learn the conditional distribution p ( y|x ) . To use DPM , which is a generative model , for discriminative tasks , we first learn the joint distribution p ( x , y ) and induce the conditional distribution from it : p ( y|x ) = p ( x , y ) / ∫ y p ( x , y ) . The joint distribution modeled by each component can be decomposed as p ( x , y|z ) = p ( y|x , z ) p ( x|z ) ( Rasmussen & Ghahramani , 2002 ; Shahbaba & Neal , 2009 ) . DPM in Related Fields . Recent works of Nagabandi et al . ( 2019 ) and Jerfel et al . ( 2019 ) exploit the DPM framework to add new components without supervision in the meta-learning context . Nagabandi et al . ( 2019 ) apply DPM to the model-based reinforcement learning to predict the next state from a given state-action pair . When a new task appears , they add a component under the DPM framework to handle predictions in the new task . Jerfel et al . ( 2019 ) apply DPM to online metalearning . Extending MAML ( Finn et al. , 2017 ) , they assume that similar tasks can be grouped into a super-task in which the parameter initialization is shared among tasks . DPM is exploited to find the super-tasks and the parameter initialization for each super-task . Therefore , it can be regarded as a meta-level CL method . These works , however , lack generative components , which are often essential to infer the responsible component at test time , as will be described in the next section . As a consequence , it is not straightforward to extend their algorithms to other CL settings beyond modelbased RL or meta-learning . In contrast , our method implements a DPM model that is applicable to general task-free CL .

Summary: The paper proposes to use a Bayesian nonparametric mixture model for task-free (without explicit task labels) continual learning. The main idea is to use an expansion-based model where the number of mixture components (experts) adapts to the training data/tasks. Specifically, a Dirichlet Process Mixture Model (DPMM) consisting of a set of neural network experts is used. Empirical results demonstrate improved performance on three different datasets over some of the baselines.