Cognitively Inspired Learning of Incremental Drifting Concepts

Humans continually expand their learned knowledge to new domains and learn new concepts without any interference with past learned experiences . In contrast , machine learning models perform poorly in a continual learning setting , where input data distribution changes over time . Inspired by the nervous system learning mechanisms , we develop a computational model that enables a deep neural network to learn new concepts and expand its learned knowledge to new domains incrementally in a continual learning setting . We rely on the Parallel Distributed Processing theory to encode abstract concepts in an embedding space in terms of a multimodal distribution . This embedding space is modeled by internal data representations in a hidden network layer . We also leverage the Complementary Learning Systems theory to equip the model with a memory mechanism to overcome catastrophic forgetting through implementing pseudo-rehearsal . Our model can generate pseudo-data points for experience replay and accumulate new experiences to past learned experiences without causing cross-task interference . 1 INTRODUCTION . Humans continually abstract concept classes from their input sensory data to build semantic descriptions , and then update and expand these concepts as more experiences are accumulated Widmer & Kubat ( 1996 ) , and use them to express their ideas and communicate with each other Gennari et al . ( 1989 ) ; Lake et al . ( 2015 ) . For example , “ cat ” and “ dog ” are one of the first concept classes that many children learn to identify . Most humans expand these concepts as concept drift occurs , e.g. , incorporating many atypical dog breeds into the “ dog ” concept , and also incrementally learn new concept classes , e.g . “ horse ” and “ sheep , ” as they acquire more experiences . Although this concept learning procedure occurs continually in humans , continual and incremental learning of concept classes remains a major challenge in artificial intelligence ( AI ) . AI models are usually trained on a fixed number of classes and the data distribution is assumed to be stationary during model execution . Hence , when an AI model is trained or updated on sequentially observed tasks with diverse distributions or is trained on new classes , it tends to forget what has been learned before due to cross-task interference , known as the phenomenon of catastrophic forgetting in the literature French ( 1991 ) . Inspired by the Parallel Distributed Processing ( PDP ) paradigm McClelland et al . ( 1986 ) ; McClelland & Rogers ( 2003 ) , our goal is to enable a deep neural network to learn drifting concept classes Gama et al . ( 2014 ) incrementally and continually in a sequential learning setting . PDP hypothesizes that abstract concepts are encoded in higher layers of the nervous system McClelland & Rogers ( 2003 ) ; Saxe et al . ( 2019 ) . Similarly , and based on behavioral similarities between artificial deep neural networks and the nervous system Morgenstern et al . ( 2014 ) , we can assume that the data representations in hidden layers of a deep network encode semantic concepts with different levels of abstractions . We model these representations as an embedding space in which semantic similarities between input data points are encoded in terms of geometric distances Jiang & Conrath ( 1997 ) , i.e. , data points that belong to the same concept class are mapped into separable clusters in the embedding space . When a new concept is abstracted , a new distinct cluster should be formed in the embedding space to encode that new class . Incremental concepts learning is feasible by tracking and remembering the representation clusters that are formed in the embedding space and by considering their dynamics as more experiences are accumulated in new unexplored domains . We benefit from the Complementary Learning Systems ( CLS ) theory McClelland et al . ( 1995 ) to mitigate catastrophic forgetting . CLS is based on empirical evidences that suggest experience replay of recently observed patterns during sleeping and waking periods in the human brain helps to accumulate the new experiences to the past learned experiences without causing interference McClelland et al . ( 1995 ) ; Robins ( 1995 ) . According to this theory , hippocampus plays the role of a short-term memory buffer that stores samples of recent experiences and catastrophic forgetting is prevented by replaying samples from the hippocampal storage to implement pseudo-rehearsal in the neocortex during sleeping periods through enhancing past learned knowledge . Unlike AI memory buffers that store raw input data point , e.g. , samples of raw images , hippocampal storage can only store encoded representations after some level of abstraction which suggests a generative nature . Inspired by the above two theories , we expand a base deep neural classifier with a decoder network , which is amended from a hidden layer , to form an autoencoder with the hidden layer as its bottleneck . The bottleneck is used to model the discriminative embedding space . As a result of supervised learning , the embedding space becomes discriminative , i.e . a data cluster is formed for each concept class in the embedding space McClelland & Rogers ( 2003 ) . These clusters can be considered analogous to neocortical representations in the brain , where the learned abstract concepts are encoded McClelland et al . ( 1986 ) . We use a multi-modal distribution to estimate this distribution . We update this parametric distribution to accumulate new experiences to past learned experiences consistently . Since our model is generative , we can implement the offline memory replay process in the sleeping brain to prevent catastrophic forgetting McClelland et al . ( 1995 ) ; Rasch & Born ( 2013 ) . When a new task arrives , we draw random samples from the multi-modal distribution and feed them into the decoder network to generate representative pseudo-data points . These pseudo-data points are then used to implement pseudo-rehearsal for experience replay Robins ( 1995 ) . We demonstrate that the neural network can learn drifting conceptsincrementally while mitigating forgetting . 2 RELATED WORK . The problem of continual learning of incremental drifting concepts lies in the intersection of lifelong learning to encode drifting concept classes and incremental learning to incorporate new concept classes . In a lifelong learning setting , the number of classes are usually assumed to be fixed , but the distribution of sequential tasks is non-stationary . In an incremental learning setting , concept classes are learned sequentially while the conditional distribution for each concept class is stationary . Continual learning : the major challenge of continual learning is tackling catastrophic forgetting . Previous works in the literature mainly rely on experience replay Li & Hoiem ( 2018 ) . The core idea of experience replay is to implement pseudo-rehearsal by replaying representative samples of past tasks along with the current task data to retain the learned distributions . Since storing these samples requires a memory buffer , the challenge is selecting the representative samples to meet the buffer size limit . For example , selecting uncommon samples that led to maximum effect in past experiences has been found to be effective Schaul et al . ( 2016 ) . However , as more tasks are learned , selecting the effective samples becomes more complex . The alternative approach is to use generative models that behave more similar to humans French ( 1999 ) . Shin et al . ( Shin et al . ( 2017 ) ) use a generative adversarial structure to mix the distributions of all tasks . It is also feasible to couple the distributions of all tasks in the bottleneck of an autoencoder . The shared distribution then can be used to generate pseudo-samples Rannen et al . ( 2017 ) .Weight consolidation using structural plasticity Lamprecht & LeDoux ( 2004 ) ; Zenke et al . ( 2017 ) ; Kirkpatrick et al . ( 2017 ) is another approach to approximate experience replay . The idea is to identify important weights that retain knowledge about a task and then consolidate them according to their relative importance for past tasks in the future . Incremental learning : : forgetting in incremental learningstems from updating the model when new classes are incorporated , rather concept drifts in a fixed number of learned classes . Hence , the goal is to learn new classes such that knowledge about the past learned classes is not overwritten . A simple approach is to expand the base network as new classes are observed . Tree-CNN Roy et al . ( 2020 ) proposes a hierarchical structure that grows like a tree when new classes are observed . The idea is to group new classes into feature-driven super-classes and find the exact label by limiting the search space . As the network grows , the new data can be used to train the expanded network . Sarwar et al . Sarwar et al . ( 2019 ) add new convolutional filters in all layers to learn the new classes through new parameters . The alternative approach is to retain the knowledge about old classes in an embedding feature space . Rebuffi et al . Rebuffi et al . ( 2017 ) proposed iCarl which maps images into a feature space that remains discriminative as more classes are learned incrementally . A fixed memory buffer is used to store exemplar images for each observed class . Each time a new class is observed , these images are used to learn a class-level representative vector in the feature space such that the testing images can be classified using nearest neighbor with respect to these vectors . Contributions : We develop a unified framework that addresses challenges of both incremental learning and lifelong learning for the first time . Our idea is based on tracking and consolidating the multimodal distribution that is formed by the internal data representations of sequential tasks in a base neural network model hidden layers . We model this distribution as a Gaussian mixture model ( GMM ) with time-dependent number of components . Concept drifts are learned by updating the corresponding GMM component for a particular class and new concepts are learned by adding new GMM components . We also make the model generative to implement experience replay . We provide both theoretical and experimental results to justify why our algorithm is effective . 3 PROBLEM STATEMENT . Consider a learning agent which observes a sequence of observed tasks { Z ( t ) } Tt=1 Chen & Liu ( 2016 ) and after learning each task moves forward to learn the next task . Each task is a classification problem in a particular domain and each class represents a concept . The classes for each task can be new unobserved classes , i.e. , necessitating incremental learning Rebuffi et al . ( 2017 ) , or drifted forms of the past learned classes , i.e. , necessitating lifelong learning Chen & Liu ( 2016 ) , or potentially a mixture of both cases . Formally , a task is characterized by a dataset D ( t ) = 〈X ( t ) , Y ( t ) 〉 , where X ( t ) = [ xt1 , . . . , x t n ] ∈ Rd×nt and Y ( t ) ∈ Rkt×nt are the data points and one-hot labels , respectively . The goal is to train a time-dependent classifier function f ( t ) ( · ) : Rd →⊂ Rkt - where kt is the number of classes for the t-th task and is fixed for each task- such that the classifier continually generalizes on the tasks seen so far . The data points x ( t ) i ∼ q ( t ) ( x ) are assumed to be drawn i.i.d . from an unknown task distribution q ( t ) ( x ) . Figure 1 visualizes a high-level block-diagram of this continual and dynamic learning procedure . The agent needs to expand its knowledge about all the observed concepts such that it can perform well on all the previous learned domains . Learning each task in isolation is a standard supervised learning problem . After selecting a suitable parameterized family of functions f ( t ) θ : Rd → Rkt with learnable parameters θ , e.g . a deep neural network with learnable weight paramters θ , we can solve for the optimal parameters using the empirical risk minimization ( ERM ) : θ̂ ( t ) = arg minθ ê ( t ) θ = arg minθ ∑ i Ld ( f ( t ) θ ( x ( t ) i ) , y ( t ) i ) , where Ld ( · ) is a proper loss function . If nt is large enough , the empirical risk expectation would be a good approximation of the real expected risk function e ( t ) ( θ ) = Ex∼q ( t ) ( x ) ( Ld ( fθ ( t ) ( x ) , f ( x ) ) ) . As a result , if the base parametric family is rich and complex enough for learning the task function , then the ERM optimal model generalizes well on unseen test samples that are drawn from q ( t ) ( x ) . For the rest of the paper , we consider the base model fθ ( t ) to be a deep neural network with an increasing output size to encode incrementally observed classes . As stated , we rely on the PDP paradigm . Hence , we decompose the deep network into an encoder sub-network φv ( · ) : Rd → Z ⊂ Rf with learnable parameter v , e.g. , convolutional layers of a CNN , and a classifier sub-network hw ( · ) kt : Rf → Rkt with learnable parameters w , e.g. , fully connected layers of a CNN , where Z denotes the embedding space in which the concepts will be be formed as separable clusters . The concepts for each task are known a priori and hence new nodes are added to the classifier subnetwork output to incorporate the new classes at time t. We use a softmax layer as the last layer of the classifier subnetwork . Hence , we can consider the classifier to be a a maximum a posteriori ( MAP ) estimator after training . This means that the encoder network transforms the input data distribution into an internal multi-modal distribution with kt modes in the embedding space because the embedding space Z should be concept-discriminative for good generalization . Each concept class is represented by a single mode of this distribution . We use a Gaussian mixture model ( GMM ) to model and approximate this distribution ( see Figure 1 , middle panel ) . Catastrophic forgetting is the result of changes in this internal distribution when changes in the input distribution leads to updating the internal distribution heuristically . Our idea is to track changes in the data distribution and update and consolidate the internal distribution such that the acquired knowledge from past experiences is not overwritten when new experiences are encountered and learned in the future . Figure 1 : Block-diagram visualization of the proposed Incremental Learning System including the learning procedure steps . ( Based viewed enlarged on screen and in color . Enlarged version is included in the Appendix ) The main challenge is to adapt the network f ( t ) θ ( · ) and the standard ERM training loss such that we can track the internal distribution continually and accumulate the new acquired knowledge consistently to the past learned knowledge with minimum interference . For this purpose , we form a generative model by amending the base model with a decoder ψu : Z → Rd , with learnable parameters u . This decoder maps back the internal representations to reconstruct the input data point in the input space such that the pair ( φu , ψu ) forms an autoencoder . According to our previous discussion , a multi-modal distribution would be formed in the bottleneck of the autoencoder upon learning each task . This distribution encodes the learned knowledge about the concepts that have been learned from past experiences so far . If we approximate this distribution with a GMM , we can generate pseudodata points that represent the previously learned concepts and use them for pseudo-rehearsal . For this purpose , we can simply draw samples from all modes of the GMM and feed these samples into the decoder subnetwork to generate a pseudo-dataset ( see Figure 1 ) . After learning each task , we can update the GMM estimate such that the new knowledge acquired is accumulated to the past gained knowledge consistenly to avoid interference . By doing this procedure continually , our model is able to learn drifting concepts incrementally . Figure 1 visualizes this repetitive procedure in this lifelong learning setting .

Summary: In this paper, the authors propose ICLA, an approach to tackle the problem of incremental and continual learning. By keeping track of a model's "internal" representations of data, they identify and overcome the "drift" issue (as in continual learning). Particularly, they adopt an encoder-decoder style architecture to learn an embedding space that serves two purposes. (i) The embeddings from the encoder serve as an "internal" representation of data whose distribution is explicitly estimated using a Gaussian Mixture Model. (ii) By sampling embeddings from this GMM and passing them through the decoder, they turn this into a generative framework to support memory replay to overcome forgetting. When new classes are incorporated (as in incremental learning), additional components are added to the parameterization of the GMM. The authors provide some theoretical guarantees for error bounds and conduct empirical evaluations on MNIST and FMNIST.