Continual Learning via Explicit Structure Learning

1 INTRODUCTION . Learning different tasks continuously is a common and practical scenario that happens all through the course of human learning . The learning of new skills from new tasks usually does not have negative impact on the previously learned tasks . Furthermore , with learning multiple tasks that are highly related , it often helps to advance all related skills . However , this is commonly not the case in current deep learning models . When presented a sequence of learning tasks , the model experiences so called “ catastrophic forgetting ” problem ( McCloskey & Cohen , 1989 ; Ratcliff , 1990 ) , where the model “ forgets ” the previous learned task while learning the new task . This is an interesting phenomenon that has attracted lots of research recently . Many efforts have been tried to overcome catastrophic forgetting . Common approaches such as Elastic weight consolidation ( EWC Kirkpatrick et al. , 2017 ) and synaptic intelligence ( Zenke et al. , 2017 ) focuses on alleviating catastrophic forgetting by applying constraints on the update of the parameters . However , forgetting is still non-negligible with these approaches , especially when the number of tasks increases . Forgetting could also be alleviated with memory-based methods , where certain information regarding learned tasks are stored to help retaining the performance of the learned tasks ( see Lopez-Paz et al. , 2017 ; Sener & Savarese , 2018 , for example ) . Additionally , there are also methods ( Mallya & Lazebnik , 2018 ; Rebuffi et al. , 2017a ; 2018 ; Mancini et al. , 2018 ) that could learn multiple domains and completely avoid forgetting by adding a small portion of parameters while the original weights is kept fixed . However , these kind of models rely on a strong base network and knowledge can only be transferred from one task to another . Most of the current approaches in continual learning employ a model with a fixed structure , and apply the same model to learn all tasks . We believe that a more intuitive and sensible approach is to learn task specific model structure explicitly along with learning of the model parameters1 . Since each task may require different architecture . In case two tasks are not quite relevant , it may not make much sense to employ the same structure for learning . For example , in case of learning digit and face recognition models , the lower level features required is likely to be drastically different , and similarly for the required structure . Forcing the same structure for these tasks is likely to prevent 1The structure that referred here is more fine-grained , such as number of layers , type of operations at each layer , etc . It does not refer to generic structure names like convolutional nerual networks or recurrent neural networks . sharing , since very less information is relevant between the tasks . On the other hand , if tasks were allowed to use different structure , it has the potential to encourage information sharing . This is because now the irrelevant part can be handled by the new/different structure . Therefore , in this work we propose a framework that explicitly separate the learning of model structure and model parameters . In particular , we employ architecture search for each of the sequential tasks to find the optimal structure for the current task . The search considers various options , such as share previous layer ’ s parameter , spawn new parameters , and so on . The model parameters are then learned correspondingly . We found that 1 ) qualitatively , the learned structure is sensible – similar tasks tends to share more parameters and structure , whereas distant tasks share less ; 2 ) quantitatively , separating the structure and parameter learning significantly reduced catastrophic forgetting . Additionally , we propose a new approach for doing continuous learning with architecture search . 2 EXPLICIT STRUCTURE LEARNING . The continual learning task that we are concerned with in this work is defined as follows . N tasks T1 , T2 , . . . , TN forms a task sequence T = ( T1 , T2 , ... , TN ) . Each task Ti is composed of dataset Di = { ( x ( i ) 1 , y ( i ) 1 ) , ( x ( i ) 2 , y ( i ) 2 ) , . . . , ( x ( i ) NTi , y ( i ) NTi ) } , where each dataset is of size NTi . The model gets to observe tasks from 1 to N sequentially . After the model finished learning on task Ti , the model can no longer access this task , i.e . all data from Di will not be available when learning tasks Ti+1 to TN . Ideally , we would like to minimize the following loss function in this continuous learning setting L ( θ ) = N∑ i=1 Li ( θ ) ( 1 ) Li ( θ ) = 1 NTi NTi∑ n=1 ` i ( fθ ( x ( i ) n ) , y ( i ) n ) ( 2 ) where fθ is the model and ` i is the loss function for task Ti . However , since we do not have access to all dataset at the same time , the above loss in equation 1 is very hard to minimize . In case we ignore the access issue and train each task separately , this will result in catastrophic forgetting . Additionally , the above loss definition clearly does not account for structure changes explicitly in the formulation . As motivated earlier , we believe explicit learning of structure is important for continuous learning . The key intuition for explicit learning structures for new tasks is that in case there is high dissimilarity between the current and previously seen tasks , optimizing model parameter may inevitably cause forgetting . This is because the model is asked to function very differently , and it is very unlikely that the new optimal parameters will also be a good solution for the previous tasks . Even in case where the tasks are similar , it may not be ideal to share all of the structure , as there might be fine grained details in tasks that make the part of the model focus on extracting different types of representations . Therefore , we introduce here si ( θ ) to indicate the structure for task Ti . The updated loss for individual task is then Li ( θ ) = 1 NTi NTi∑ n=1 ` i ( fsi ( θ ) ( x ( i ) n ) , y ( i ) n ) ( 3 ) Now the structure is explicitly taken into consideration when learning all the tasks . When optimizing the updated loss in equation 3 , one needs to determine the optimal parameter based on the structure si . This loss can be viewed in two ways . One can interpret the above loss as selecting a task specific network from a ‘ super network ’ that has parameter θ using si , or for each task we train a new model with parameter si ( θ ) . There is a subtle difference between this two views . The former one has an constraint on the total model size , where as the latter one does not . So in worst case scenario , the model size will grow linearly as we increase the number of tasks . This would lead to a trivial solution – training completely different models for different tasks and is no longer continuous learning ! To address this problem , we propose the following loss Li ( θ ) = 1 NTi NTi∑ n=1 ` i ( fsi ( θ ) ( x ( i ) n ) , y ( i ) n ) + βiRsi ( si ) + λiR p i ( θ ) ( 4 ) where βi > 0 , λi ≥ 0 , Rsi andR p i indicate regularizer for structure and parameter , respectively . For instance , one can use ` 2 regularization for Rpi when optimizing model parameters , and Rsi can be as simple as the ( log ) number of parameters . In this way , the total number of parameters are upper bounded , and the degenerate case is thus avoided . 3 OUR IMPLEMENTATION . It is a challenging problem to optimize the loss described in equation 4 , since it involves explicit optimization of the structure of the model . In our implementation , we choose to separate this optimization problem to two steps : structure optimization and model parameter learning . In particular , we employ neural architecture search to deal with structure optimization , and then gradient based method for learning model parameters after the structure is fixed . We explain our implementation in detail in the following section . 3.1 STRUCTURE OPTIMIZATION . We employ neural architecture search for structure optimization . Before we move on to further details , we make a further simplification , where we assume that one already have in mind a global structure that may work for all tasks , and we are only selecting connectivity pattern between layers and their corresponding operator . It is straight forward to adapt this to more complicated cases , we make the simplification because : 1 ) it is common in a multi-task continual learning scenario that one has some rough clue regarding the overall model structure ; 2 ) this simplifies the optimization problem significantly . Let ’ s define a certain network with L shareable layers and one task-specific layer ( i.e . last layer ) for each task . A super network S is maintained so that all the new task-specific layers and new shareable layers will be stored into S. The goal of search is trying to find out the optimal choice for each of the L layers , given the current task data Di and all the shareable layer ’ s weights stored in S. The candidate choices for each layer could be “ reuse ” , “ adaptation ” and “ new ” . The reuse choice will make new task use the same parameter as the previous task . The adaptation option adds a small parameter overhead that trains an additive function to the original layer output . The new operator will spawn new parameters of exactly the size of the current layer parameters . Here , we denote the size of the lth layer in super network S as |Sl| . The total number of choices in the lth layer Cl is 2|Sl|+1 , because we will have |Sl| ” reuse ” , |Sl| ” adaptation ” and 1 ” new ” . Thus , the total search space is ∏L l Cl . One potential issue here is that , in worst case , the search space may grow exponentially with respect to the number of tasks . One way of dealing with this is to limit the total number of possible choices , and make use a priority queue for managing the options . However , we do not find this necessary in all of our experiments . Similar to DARTS ( Liu et al. , 2018 ) , to make the search space continuous , we relax the categorical choice of the lth layer as a softmax over all possible Cl choices : xl+1 = Cl∑ c=1 exp ( αlc ) ∑Cl c′=1 exp ( α l c′ ) glc ( xl ) ( 5 ) Here , the vector αl of dimension Cl is the architecture weights that are used for mixing the choices for each sharable layer . And glc here is the operator for the choice c at layer l which is expressed as : glc ( xl ) = Slc ( xl ) if c ≤ |Sl| , Slc ( xl ) + γ l c−|Sl| ( xl ) if |S l| < c ≤ 2|Sl| , ol ( xl ) if c = 2|Sl|+ 1 ( 6 ) Here , γ is the adapter operator and o is the new operator training from scratch . After this relaxation , the task of discrete search become optimizing a set of continuous weights α = { αl } . After finishing the searching , the optimal architecture could be obtained by taking the index with the largest weight αlc for each layer l , i.e . cl = argmaxα l. Adopting the training strategy from DARTS . We use validation loss Lval to update the architecture weights α , while the operator weights are optimized by the training loss Ltrain . The architecture weights and operator weights are updated alternately during the search process . Because it is a nested bi-level optimization problem , the original DARTS provide a second-order approximation for more accurate optimization . In this work , we find it is sufficient to use the simple alternately update approach , which was referred as the first-order approximation in ( Liu et al. , 2018 ) . To make it clear how “ reuse ” , “ adaptation ” and “ new ” operator works , we walk through a concrete example in the following . Let us take a convolutional neural network ( CNN ) with all the layers using 3x3 kernel size as an example . The choice of “ reuse ” is just using the existing weight and keep it fixed during learning , thus there is no additional parameter cost . For “ adaptation ” , it could be a 1x1 conv layer added to the original 3x3 conv layer in parallel ( i.e . similar to the adaptor used in paper ( Rebuffi et al. , 2017a ) ) . During training , the weight of the original 3x3 conv is kept fixed , while the parameters of the 1x1 conv adapter is modified . In this case , the additional parameter cost is only 1/9 of the original parameter size . In case of “ new ” , it is literally start with a new operator that initialized randomly and train from scratch . We make use of the loss function Lval to implement the regularizer Rsi ( si ) . The value of the regularizer is set proportional to the product of the additional parameter size zlc and its corresponding weight α l c ( i.e . Rsi ( si ) = ∑ c , l α l cz l c ) . The architecture weights α is optimized in terms of both accuracy and parameter efficiency at the same time .

Summary: The proposed approach aims to mitigate catastrophic forgetting in continual learning (CL) problems by structure learning: determining whether to reuse or adapt existing parameters, or initialise new ones, when faced with a new task. This is framed as an architecture search problem, applying ideas from Differentiable Architecture Search (DARTS). The approach is verified on the Permuted MNIST dataset and evaluated on the Visual Decathlon, showing an improvement.