Contextual Transformation Networks for Online Continual Learning

Continual learning methods with fixed architectures rely on a single network to learn models that can perform well on all tasks . As a result , they often only accommodate common features of those tasks but neglect each task ’ s specific features . On the other hand , dynamic architecture methods can have a separate network for each task , but they are too expensive to train and not scalable in practice , especially in online settings . To address this problem , we propose a novel online continual learning method named “ Contextual Transformation Networks ” ( CTN ) to efficiently model the task-specific features while enjoying neglectable complexity overhead compared to other fixed architecture methods . Moreover , inspired by the Complementary Learning Systems ( CLS ) theory , we propose a novel dual memory design and an objective to train CTN that can address both catastrophic forgetting and knowledge transfer simultaneously . Our extensive experiments show that CTN is competitive with a large scale dynamic architecture network and consistently outperforms other fixed architecture methods under the same standard backbone . Our implementation can be found at https : //github . com/phquang/Contextual-Transformation-Network . 1 INTRODUCTION . Continual learning is a promising framework towards building AI models that can learn continuously through time , acquire new knowledge while being able to perform its already learned skills ( French , 1999 ; 1992 ; Parisi et al. , 2019 ; Ring , 1997 ) . On top of that , online continual learning is particularly interesting because it resembles the real world and the model has to quickly obtain new knowledge on the fly by levering its learned skills . This problem is important for deep neural networks because optimizing them in the online setting has been shown to be challenging ( Sahoo et al. , 2018 ; Aljundi et al. , 2019a ) . Moreover , while it is crucial to obtain new information , the model must be able to perform its acquired skills . Balancing between preventing catastrophic forgetting and facilitating knowledge transfer is imperative when learning on a stream of tasks , which is ubiquitous in realistic scenarios . Thus , in this work , we focus on the continual learning setting in an online learning fashion , where both tasks and data of each task arrive sequentially ( Lopez-Paz & Ranzato , 2017 ) . In the literature , fixed architecture methods employ a shared feature extractor and a set of classifiers , one for each task ( Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2019a ; b ; Aljundi et al. , 2019a ) . Although using a shared feature extractor has achieved promising results , the common and global features are rather generic and not well-tailored towards each specific task . This problem is even more severe when old data are limited while learning new tasks . As a result , the common feature extractor loses its ability to extract previous tasks ’ features , resulting in catastrophic forgetting . On the other hand , while dynamic architecture methods such as Rusu et al . ( 2016 ) ; Li et al . ( 2019 ) ; Xu & Zhu ( 2018 ) alleviate this problem by having a separate network for each task , they suffer from the unbounded growth of the parameters . Moreover , the subnetworks ’ design is not trivial and requires extensive resource usage ( Rusu et al. , 2016 ; Li et al. , 2019 ) , which is not practical in many applications . These limitations motivated us to develop a novel method that can facilitate continual learning with a fixed architecture by modeling the task-specific features . To achieve this goal , we first revisit a popular result in learning multiple tasks that each task ’ s features are centered around a common vector ( Evgeniou & Pontil , 2004 ; Aytar & Zisserman , 2011 ; Pentina & Lampert , 2014 ; Liu et al. , 2019b ) . This result motivates us to develop a novel framework of Contextual Transformation Networks ( CTN ) , which consists of a base network that learns the common features of a given input and a controller that efficiently transforms the common features to become task-specific , given a task identifier . While one can train CTN using experience replay , it does not explicitly aim at achieving a good trade-off between stability and plasticity . Therefore , we propose a novel dual memory system and a learning method that encapsulate alleviating forgetting and facilitating knowledge transfer simultaneously . Particularly , we propose two distinct memories : the episodic memory and the semantic memory associated with the base model and the controller , respectively . Then , the base model is trained by experience replay on the episodic memory while the controllers is trained to learn task-specific features that can generalize to the semantic memory . As a result , CTN achieves a good trade-off between preventing catastrophic forgetting and facilitating knowledge transfer because the task-specific features can generalize well to all past and current tasks . Figure 1 gives an overview of the proposed Contextual Transformation Network ( CTN ) . Interestingly , the designs of our CTN and dual memory are partially related to the Complementary Learning Systems ( CLS ) theory in neuroscience ( McClelland et al. , 1995 ; Kumaran et al. , 2016 ) . Particularly , the controller acts as a neocortex that learns the structured knowledge of each task . In contrast , the base model acts as a hippocampus that performs rapid learning to acquire new information from the current task ’ s training data . Following the naming convention of memory in neuroscience , our CTN is equipped with two replay memory types . ( i ) the episodic memory ( associated with the hippocampus ) caches a small amount of past tasks ’ training data , which will be replayed when training the base networks . ( ii ) the semantic memory ( associated with the neocortex ) stores another distinct set of old data only used to train the controller such that the task-specific features can generalize well across tasks . Moreover , the CLS theory also suggests that the interplay between the neocortex and the hippocampus attributes to the ability to recall knowledge and generalize to novel experiences ( Kumaran & McClelland , 2012 ) . Our proposed learning approach closely characterizes such properties : the base model focuses on acquiring new knowledge from the current task while the controller uses the base model ’ s knowledge to generalize to novel samples . In summary , our work makes the following contributions . First , we propose CTN , a novel continual learning method that can model task-specific features while enjoying neglectable complexity overhead compared to fixed architecture methods ( please refer to Table 4 ) . Second , we propose a novel objective that can improve the trade-off between alleviating forgetting and facilitating knowledge transfer to train CTN . Third , we conduct extensive experiments on continual learning benchmarks to demonstrate the efficacy of CTN compared to a suite of baselines . Finally , we provide a comprehensive analysis to investigate the complementarity of each CTN ’ s component . 2 METHOD . Notations . We denote φ as parameter of the base model that extracts global features from the input and θ as the parameter of the controller which modifies the features from φ given a task identifier t. The task identifier can be a set of semantic attributes about objects of that task ( Lampert et al. , 2009 ) or simply an index of the task , which we use in this work as a one-hot vector . A prediction is given as gϕt ( hφ , θ ( x , t ) ) , where gϕt ( · ) is the task Tt ’ s classifier with parameter ϕt such as a fully connected layer with softmax activation . And hφ , θ ( x , t ) is the final feature after transformed by the controller . We denoteDtrt as the training data of task Tt , Memt andMsmt as the episodic memory and the semantic memory of task Tt respectively . The episodic memory and semantic memory maintains two distinct sets of data obtained from task Tt . The episodic memory of task T1 , . . . , Tt−1 is denoted asMem < t ; similarly , Msm < t denotes the semantic memory of the first t− 1 tasks . Remark . Both theMemt andMsmt are obtained from Dtrt through the learner ’ s internal memory management strategy and contains distinct samples from each other such that their combined sizes do not exceed a pre-defined budget . 2.1 LEARNING TASK-SPECIFIC FEATURES FOR CONTINUAL LEARNING . Given a backbone network , one can implement the task-specific features by employing a set of task-specific filters and applying them to the backbone ’ s output . However , this trivial approach is not scalable , even for small networks . In the worst case , it results in storing an additional network per task , which violates the fixed architecture constraint . Since we want to obtain task-specific features with minimal parameter overhead , we propose to use a feature-wise transformation ( Perez et al. , 2018 ) to efficiently extract the task-specific features h̃ ( x , t ) from the common features ĥ ( x ) as follows : h̃ ( x ; t ) = γt ‖γt‖2 ⊗ ĥ ( x ) + βt ‖βt‖2 and { γt , βt } = cθ ( t ) , ( 1 ) where ⊗ denotes the element-wise multiplication operator , and cθ ( t ) is the controller implemented as a linear layer with parameter θ that predicts the transformation coefficients { γt , βt } given the task identifier t. Since the task identifiers are one-hot vectors , which are sparse and make training the controller difficult , we also introduce an embedding layer to map the task identifiers to dense , low dimensional vectors . For simplicity , we will use θ to refer to both the embedding and the linear layer parameters . In addition , instead of storing a set of coefficients { γt , βt } for each task , we only need a fixed set of parameter θ to predict these coefficients , which results in the fixed parameters in the controller . The coefficients { γt , βt } are ` 2-normalized and then transforms the common features ĥ ( x , t ) to become task-specific features h̃ ( x ; t ) . Finally , both feature types are combined by a residual connection before passing to the corresponding classifier gt ( · ) to make the final prediction : gϕt ( σ ( h ( x , t ) ) ) , and h ( x , t ) = ĥ ( x , t ) + h̃ ( x , t ) , ( 2 ) where σ ( · ) is a nonlinear activation function such as ReLU . Importantly , when the task-specific features are removed , i.e. , h̃ ( x , t ) = 0 , Eq . 2 reduces to the traditional experience replay . Lastly , for each incoming task , CTN has to allocate a new classifier , which is the same for all continual learning methods , and a new embedding vector , which is usually low dimensional , e.g . 32 or 64 . Therefore , CTN enjoys almost the same parameter growth as existing continual learning methods . 2.2 TRAINING THE CONTROLLER . While one can train CTN with experience replay ( ER ) , it does not explicitly address the trade-off between facilitating knowledge transfer and alleviating catastrophic forgetting . This motivates us to develop a novel training method that can simultaneously address both problems by leveraging the controller ’ s task-specific features . First , we introduce a dual memory system consisting of the semantic memoryMsmt associated with the controller and the episodic memoryMemt associated with the base model . We propose to train only the base model using experience replay with the episodic memory to obtain new knowledge from incoming tasks . The controller is also trained so that the task-specific features can generalize to unseen samples to the base model stored in the semantic memory . As a result , the task-specific features can generalize to both previous and current tasks , which simultaneously encapsulate both alleviating forgetting and facilitating knowledge transfer . Formally , given the current batch of data for task Tt as Bt , the training of CTN can be formulated as the following bilevel optimization problem ( Colson et al. , 2007 ) : Outer problem : minθ Lctrl ( { φ∗ , θ } ; Msm < t+1 ) Inner problem : s.t φ∗ = arg min φ Ltr ( { φ , θ } , Bt ∪Mem < t ) , ( 3 ) where φ∗ denotes the optimal base model corresponding to the current controller θ . Since every CTN ’ s prediction always involves both the controller and the base model , we use Ltr ( { φ , θ } , Bt ∪Mem < t ) to denote the training loss of the pair { φ , θ } on the data Bt ∪Mem < t . Similarly , Lctrl ( · ) denotes the controller ’ s loss . For simplicity , we omitted the dependency of the the loss on the classifiers ’ parameters and imply that the classifiers are jointly updated with the base model . Since we do not know the optimal transformation coefficients of any task , the controller is trained to minimize the classification loss of the samples via φ . We implement both the training and controller ’ s losses as the cross-entropy loss . Notably , Eq . 3 characterizes two nested optimization problems : the outer problem , which trains the controller to generalize , and each controller parameter θ parameterizes an inner problem that trains the base model to acquire new knowledge via experience replay . Moreover , only φ is trained in the inner problem , while only θ is updated in the outer problem . The bilevel optimization objective such as Eq . 3 has been successfully applied in other machine learning disciplines such as hyperparameter optimization , meta learning ( Franceschi et al. , 2018 ; Finn et al. , 2017 ) , and AutoML ( Liu et al. , 2019a ) . In this work , we extend this framework to continual learning to train the controller . However , unlike existing works ( Franceschi et al. , 2018 ; Finn et al. , 2017 ; Liu et al. , 2019a ) , our Eq . 3 has to be solved incrementally when a new data sample arrives . Therefore , we consider Eq . 3 as an online learning problem and optimize it using the follow the leader principle ( Hannan , 1957 ) . Particularly , we relax the optimal solutions of both the inner and outer problems to be solutions from a few gradient steps . When a new training data arrives , we first train the base model φ using experience replay for a few SGD steps with an inner learning rate α , each of which is implemented as : φ← φ− α∇φLtr ( { φ , θ } , Bt ∪Mem < t ) , ( 4 ) Then , we optimize the controller θ such that it can improve φ ’ s performance on the semantic memory : θ ← θ − β∇θLctrl ( { φ , θ } , Msm < t+1 ) , ( 5 ) where β is the outer learning rate . As a result , Eq . 3 is implemented as an alternative update procedure involving several outer updates to train θ , each of which includes an inner update to trainφ . Moreover , performing several updates per incoming sample does not violate the online assumption since we will not revisit that sample in the future , unless it is stored in the memories .

Summary: This paper tackles online continual neural network learning (following Lopez-Paz & Ranzato, 2017) with a combination of techniques: (1) a controller (or base parameter modulator, or hypernetwork) is introduced which produces task-specific scale and shift parameters, which modulate the feature maps of a base model (Perez et al., 2017); (2) as training data set samples are only provided once, the authors maintain experience replay buffers using soft label targets (by now standard techniques in online learning); (3) a bilevel optimization scheme where controller parameters are updated in an outer loop and base parameters adapted in an inner loop is employed. Task identities are available both during learning and at test time. This information is used to pick the correct task embeddings and the correct task-specific final classifier ("head").