Continual learning using hash-routed convolutional neural networks

Continual learning could shift the machine learning paradigm from data centric to model centric . A continual learning model needs to scale efficiently to handle semantically different datasets , while avoiding unnecessary growth . We introduce hash-routed convolutional neural networks : a group of convolutional units where data flows dynamically . Feature maps are compared using feature hashing and similar data is routed to the same units . A hash-routed network provides excellent plasticity thanks to its routed nature , while generating stable features through the use of orthogonal feature hashing . Each unit evolves separately and new units can be added ( to be used only when necessary ) . Hash-routed networks achieve excellent performance across a variety of typical continual learning benchmarks without storing raw data and train using only gradient descent . Besides providing a continual learning framework for supervised tasks with encouraging results , our model can be used for unsupervised or reinforcement learning . 1 INTRODUCTION . When faced with a new modeling challenge , a data scientist will typically train a model from a class of models based on her/his expert knowledge and retain the best performing one . The trained model is often useless when faced with different data . Retraining it on new data will result in poor performance when trying to reuse the model on the original data . This is what is known as catastrophic forgetting ( McCloskey & Cohen , 1989 ) . Although transfer learning avoids retraining networks from scratch , keeping the acquired knowledge in a trained model and using it to learn new tasks is not straightforward . The real knowledge remains with the human expert . Model training is usually a data centric task . Continual learning ( Thrun , 1995 ) makes model training a model centric task by maintaining acquired knowledge in previous learning tasks . Recent work in continual ( or lifelong ) learning has focused on supervised classification tasks and most of the developed algorithms do not generate stable features that could be used for unsupervised learning tasks , as would a more generic algorithm such as the one we present . Models should also be able to adapt and scale reasonably to accommodate different learning tasks without using an exponential amount of resources , and preferably with little data scientist intervention . To tackle this challenge , we introduce hash-routed networks ( HRN ) . A HRN is composed of multiple independent processing units . Unlike typical convolutional neural networks ( CNN ) , the data flow between these units is determined dynamically by measuring similarity between hashed feature maps . The generated feature maps are stable . Scalability is insured through unit evolution and by increasing the number of available units , while avoiding exponential memory use . This new type of network maintains stable performance across a variety of tasks ( including semantically different tasks ) . We describe expansion , update and regularization algorithms for continual learning . We validate our approach using multiple publicly available datasets , by comparing supervised classification performance . Benchmarks include Pairwise-MNIST , MNIST/Fashion-MNIST ( Xiao et al. , 2017 ) and SVHN/incremental-Cifar100 ( Netzer et al. , 2011 ; Krizhevsky et al. , 2009 ) . Relevant background is introduced in section 2 . Section 3 details the hash-routing algorithm and discusses its key attributes . Section 4 compares our work with other continual learning and dynamic network studies . A large set of experiments is carried out in section 5 . 2 FEATURE HASHING BACKGROUND . Feature hashing , also known as the hashing trick ( Weinberger et al. , 2009 ) is a dimension reduction transformation with key properties for our work : inner product conservation and quasi-orthogonality . A feature hashing function φ : RN → Rs , can be built using two uniform hash functions h : N→ { 1 , 2 ... , s } and ξ : N→ { −1 , 1 } , as such : φi ( x ) = ∑ j∈ [ [ 1 , N ] ] j : h ( j ) =i ξ ( j ) xj where φi denotes the ith component of φ . Inner product is preserved as E [ φ ( a ) Tφ ( b ) ] = aTb . φ provides an unbiased estimator of the inner product . It can also be shown that if ||a||2 = ||b||2 = 1 , then σa , b = O ( 1s ) . Two different hash functions φ and φ′ ( e.g . h 6= h′ or ξ 6= ξ′ ) are orthogonal . In other words , ∀ ( v , w ) ∈ Im ( φ ) × Im ( φ′ ) , E [ vTw ] ≈ 0 . Furthermore , Weinberger et al . ( 2009 ) details the inner product bounds , given v ∈ Im ( φ ) and x ∈ RN : Pr ( |vTφ′ ( x ) | > ) ≤ 2 exp ( − 2/2 s−1 ‖v‖22 ‖x‖ 2 2 + ‖v‖∞ ‖x‖∞ /3 ) ( 1 ) Eq.1 shows that approximate orthogonality is better when φ′ handles bounded vectors . Data independent bounds can be obtained by setting ‖x‖∞ = 1 and replacing v by v ‖v‖2 , which leads to ‖x‖22 ≤ N and ‖v‖∞ ≤ 1 , hence : Pr ( |vTφ′ ( x ) | > ) ≤ 2 exp ( − 2/2 s−1 ‖x‖22 + ‖v‖∞ /3 ) ≤ 2 exp ( − 2/2 N/s+ /3 ) ( 2 ) Better approximate orthogonality significantly reduces correlation when summing feature vectors generated by different hashing functions , as is done in hash-routed networks . 3 HASH-ROUTED NETWORKS . 3.1 STRUCTURE . A hash-routed network maps input data to a feature vector of size s that is stable across successive learning tasks . An HRN exploits inner product preservation to insure that similarity between generated feature vectors reflect the similarity between input samples . Quasi-orthogonality of different feature hashing functions is used to reduce correlation between the output ’ s components , as it is the sum of individual hashed feature vectors . An HRN H is composed of M units { U1 , ... , UM } . Each unit Uk is composed of : • A series of convolution operations fk . It is characterized by a number of input channels and a number of output channels , resulting in a vector of trainable parameters wk . Note that fk can also include pooling operations . • An orthonormal projection basis Bk . It contains a maximum of m non-zeros orthogonal vectors of size s. Each basis is filled with zero vectors at first . These will be replaced by non-zero vectors during training . • A feature hashing function φk that maps a feature vector of any size to a vector of size s. The network also has an independent feature hashing function φ0 . All the feature hashing functions are different but generate feature vectors of size s . 3.2 OPERATION . 3.2.1 HASH-ROUTING ALGORITHM . H maps an input sample x to a feature vectorH ( x ) of size s. In a vanilla CNN , x would go through a series of deterministic convolutional layers to generate feature maps of growing size . In a HRN , the convolutional layers that will be involved will vary depending on intermediate results . Feature hashing is used to route operations . Since feature hashing preserves the inner product in the hashed features space , similar samples will be processed similarly . Intermediate features are hashed and projected upon the units ’ projection bases . The unit where the projection ’ s magnitude is the highest is selected for the next operation . Operations continue until a maximum depth d is reached ( i.e . there is a limit of d − 1 chained operations ) , or when the projection residue is below a given threshold τd . H ( x ) is the sum of all residues . Let { Ui1 , Ui2 , ... , Uid−1 } be the ordered set of units involved in processing x ( assuming the final projection residue ’ s magnitude is greater than τd ) . Operation 0 simply involves hashing the ( flattened ) input sample using φ0 . Let xik = fik ◦ fik−1 ◦ ... ◦ fi1 ( x ) be the intermediate features obtained at operation k. The normalized hashed features vector after operation k is computed as such : hik = φik ( xik ‖xik‖∞ ) ∥∥∥∥φik ( xik‖xik‖∞ ) ∥∥∥∥ 2 ( 3 ) For operation 0 , hi0 is computed using x and φ0 . pik = Bik+1hik and rik = hik − pik are the projection vector and residue vector over basis Bik+1 resp . As explained earlier , this means that : ik+1 = argmax j∈I\ { i1 , ... , ik } ‖Bjhik‖2 ( 4 ) where I is the subset of initialized units ( i.e . units with bases containing at least one non-zero vector ) . Finally , H ( x ) = ∑ j∈ { i0 , ... , id−1 } rj ( 5 ) The full inference algorithm is summarized in Algorithm 1 and an example is given in Figure 1 . 3.2.2 ANALYSIS . The output of a typical CNN is a feature map with a dimension that depends on the number of output channels used in each convolutional layer . In a HRN , this would lead to a variable dimension output as the final feature map depends on the routing . In a continual learning setup , dealing with variable dimension feature maps would be impractical . Feature hashing circumvents these problems by generating feature vectors of fixed dimension . Similar feature maps get to be processed by the same units , as a consequence of using feature hashing Algorithm 1 : Hash-routed inference Input : x Output : H = H ( x ) h0 = φ0 ( x ) ; J = ∅ H ← 0 ; h← h0 ; y ← x for j = 1 , ... , d− 1 do ij = argmaxk∈I\J ‖Bkh‖2 ; // select the best unit r← h−Bijh ; // compute new residue H ← H + r ; // accumulate residue for output J ← J ∪ { ij } ; // update set of used units if ‖r‖2 < τd then break ; // stop processing when residue is too low else y ← fij ( y ) ; // compute feature map h← φij ( y ‖y‖∞ ) ∥∥∥φij ( y‖y‖∞ ) ∥∥∥2 ; // new hash vector using flattened feature map end end for routing . In this context , similarity is measured by the inner product of flattened feature maps , projected onto different orthogonal subspaces ( each unit basis span ) . Another consequence is that unit weights become specialized in processing a certain type of features , rather than having to adapt to task specific features . This provides the kind of stability needed for continual learning . For a given unit Uk , rank ( Bk ) ≤ m < < s. Hence , it is reasonable to consider that the orthogonal subspace ’ s contribution to total variance is much more important than that of Bk . This is whyH ( x ) only contains projection residues . Note that in Eq.3 , hik ∈ Im ( φik ) and ‖hik‖2 = 1 . The operand under φik has an infinite norm of 1 , which under Eq.2 leads to inner product bounds independent of input data when considering orthogonality . Moreover , due to the approximate orthogonality of different feature hashing functions , summing the residues will not lead to much information loss as each residue vector rik is in Im ( φik−1 ) but this also explains why each unit can only be selected once . The residues ’ ` 1-norms are added to the loss function to induce sparsity . Denoting LT the specific loss for task T ( e.g . KL-divergence for supervised classification ) , the final loss L is : L = LT + λ ∑ j∈ { i0 , ... , id−1 } ‖rj‖1 ( 6 )

Summary: For a learning model to learn continuously, it needs to handle new datasets without catastrophic forgetting or requiring the model to grow larger. This paper proposes a hash-routed convolution neural network where a different set of convolution filters are used depending on the data. New convolution filters can be added to the network without increasing the computational cost of the network.