Continual learning using hash-routed convolutional neural networks

Continual learning could shift the machine learning paradigm from data centric to model centric . A continual learning model needs to scale efficiently to handle semantically different datasets , while avoiding unnecessary growth . We introduce hash-routed convolutional neural networks : a group of convolutional units where data flows dynamically . Feature maps are compared using feature hashing and similar data is routed to the same units . A hash-routed network provides excellent plasticity thanks to its routed nature , while generating stable features through the use of orthogonal feature hashing . Each unit evolves separately and new units can be added ( to be used only when necessary ) . Hash-routed networks achieve excellent performance across a variety of typical continual learning benchmarks without storing raw data and train using only gradient descent . Besides providing a continual learning framework for supervised tasks with encouraging results , our model can be used for unsupervised or reinforcement learning . 1 INTRODUCTION . When faced with a new modeling challenge , a data scientist will typically train a model from a class of models based on her/his expert knowledge and retain the best performing one . The trained model is often useless when faced with different data . Retraining it on new data will result in poor performance when trying to reuse the model on the original data . This is what is known as catastrophic forgetting ( McCloskey & Cohen , 1989 ) . Although transfer learning avoids retraining networks from scratch , keeping the acquired knowledge in a trained model and using it to learn new tasks is not straightforward . The real knowledge remains with the human expert . Model training is usually a data centric task . Continual learning ( Thrun , 1995 ) makes model training a model centric task by maintaining acquired knowledge in previous learning tasks . Recent work in continual ( or lifelong ) learning has focused on supervised classification tasks and most of the developed algorithms do not generate stable features that could be used for unsupervised learning tasks , as would a more generic algorithm such as the one we present . Models should also be able to adapt and scale reasonably to accommodate different learning tasks without using an exponential amount of resources , and preferably with little data scientist intervention . To tackle this challenge , we introduce hash-routed networks ( HRN ) . A HRN is composed of multiple independent processing units . Unlike typical convolutional neural networks ( CNN ) , the data flow between these units is determined dynamically by measuring similarity between hashed feature maps . The generated feature maps are stable . Scalability is insured through unit evolution and by increasing the number of available units , while avoiding exponential memory use . This new type of network maintains stable performance across a variety of tasks ( including semantically different tasks ) . We describe expansion , update and regularization algorithms for continual learning . We validate our approach using multiple publicly available datasets , by comparing supervised classification performance . Benchmarks include Pairwise-MNIST , MNIST/Fashion-MNIST ( Xiao et al. , 2017 ) and SVHN/incremental-Cifar100 ( Netzer et al. , 2011 ; Krizhevsky et al. , 2009 ) . Relevant background is introduced in section 2 . Section 3 details the hash-routing algorithm and discusses its key attributes . Section 4 compares our work with other continual learning and dynamic network studies . A large set of experiments is carried out in section 5 . 2 FEATURE HASHING BACKGROUND . Feature hashing , also known as the hashing trick ( Weinberger et al. , 2009 ) is a dimension reduction transformation with key properties for our work : inner product conservation and quasi-orthogonality . A feature hashing function φ : RN → Rs , can be built using two uniform hash functions h : N→ { 1 , 2 ... , s } and ξ : N→ { −1 , 1 } , as such : φi ( x ) = ∑ j∈ [ [ 1 , N ] ] j : h ( j ) =i ξ ( j ) xj where φi denotes the ith component of φ . Inner product is preserved as E [ φ ( a ) Tφ ( b ) ] = aTb . φ provides an unbiased estimator of the inner product . It can also be shown that if ||a||2 = ||b||2 = 1 , then σa , b = O ( 1s ) . Two different hash functions φ and φ′ ( e.g . h 6= h′ or ξ 6= ξ′ ) are orthogonal . In other words , ∀ ( v , w ) ∈ Im ( φ ) × Im ( φ′ ) , E [ vTw ] ≈ 0 . Furthermore , Weinberger et al . ( 2009 ) details the inner product bounds , given v ∈ Im ( φ ) and x ∈ RN : Pr ( |vTφ′ ( x ) | > ) ≤ 2 exp ( − 2/2 s−1 ‖v‖22 ‖x‖ 2 2 + ‖v‖∞ ‖x‖∞ /3 ) ( 1 ) Eq.1 shows that approximate orthogonality is better when φ′ handles bounded vectors . Data independent bounds can be obtained by setting ‖x‖∞ = 1 and replacing v by v ‖v‖2 , which leads to ‖x‖22 ≤ N and ‖v‖∞ ≤ 1 , hence : Pr ( |vTφ′ ( x ) | > ) ≤ 2 exp ( − 2/2 s−1 ‖x‖22 + ‖v‖∞ /3 ) ≤ 2 exp ( − 2/2 N/s+ /3 ) ( 2 ) Better approximate orthogonality significantly reduces correlation when summing feature vectors generated by different hashing functions , as is done in hash-routed networks . 3 HASH-ROUTED NETWORKS . 3.1 STRUCTURE . A hash-routed network maps input data to a feature vector of size s that is stable across successive learning tasks . An HRN exploits inner product preservation to insure that similarity between generated feature vectors reflect the similarity between input samples . Quasi-orthogonality of different feature hashing functions is used to reduce correlation between the output ’ s components , as it is the sum of individual hashed feature vectors . An HRN H is composed of M units { U1 , ... , UM } . Each unit Uk is composed of : • A series of convolution operations fk . It is characterized by a number of input channels and a number of output channels , resulting in a vector of trainable parameters wk . Note that fk can also include pooling operations . • An orthonormal projection basis Bk . It contains a maximum of m non-zeros orthogonal vectors of size s. Each basis is filled with zero vectors at first . These will be replaced by non-zero vectors during training . • A feature hashing function φk that maps a feature vector of any size to a vector of size s. The network also has an independent feature hashing function φ0 . All the feature hashing functions are different but generate feature vectors of size s . 3.2 OPERATION . 3.2.1 HASH-ROUTING ALGORITHM . H maps an input sample x to a feature vectorH ( x ) of size s. In a vanilla CNN , x would go through a series of deterministic convolutional layers to generate feature maps of growing size . In a HRN , the convolutional layers that will be involved will vary depending on intermediate results . Feature hashing is used to route operations . Since feature hashing preserves the inner product in the hashed features space , similar samples will be processed similarly . Intermediate features are hashed and projected upon the units ’ projection bases . The unit where the projection ’ s magnitude is the highest is selected for the next operation . Operations continue until a maximum depth d is reached ( i.e . there is a limit of d − 1 chained operations ) , or when the projection residue is below a given threshold τd . H ( x ) is the sum of all residues . Let { Ui1 , Ui2 , ... , Uid−1 } be the ordered set of units involved in processing x ( assuming the final projection residue ’ s magnitude is greater than τd ) . Operation 0 simply involves hashing the ( flattened ) input sample using φ0 . Let xik = fik ◦ fik−1 ◦ ... ◦ fi1 ( x ) be the intermediate features obtained at operation k. The normalized hashed features vector after operation k is computed as such : hik = φik ( xik ‖xik‖∞ ) ∥∥∥∥φik ( xik‖xik‖∞ ) ∥∥∥∥ 2 ( 3 ) For operation 0 , hi0 is computed using x and φ0 . pik = Bik+1hik and rik = hik − pik are the projection vector and residue vector over basis Bik+1 resp . As explained earlier , this means that : ik+1 = argmax j∈I\ { i1 , ... , ik } ‖Bjhik‖2 ( 4 ) where I is the subset of initialized units ( i.e . units with bases containing at least one non-zero vector ) . Finally , H ( x ) = ∑ j∈ { i0 , ... , id−1 } rj ( 5 ) The full inference algorithm is summarized in Algorithm 1 and an example is given in Figure 1 . 3.2.2 ANALYSIS . The output of a typical CNN is a feature map with a dimension that depends on the number of output channels used in each convolutional layer . In a HRN , this would lead to a variable dimension output as the final feature map depends on the routing . In a continual learning setup , dealing with variable dimension feature maps would be impractical . Feature hashing circumvents these problems by generating feature vectors of fixed dimension . Similar feature maps get to be processed by the same units , as a consequence of using feature hashing Algorithm 1 : Hash-routed inference Input : x Output : H = H ( x ) h0 = φ0 ( x ) ; J = ∅ H ← 0 ; h← h0 ; y ← x for j = 1 , ... , d− 1 do ij = argmaxk∈I\J ‖Bkh‖2 ; // select the best unit r← h−Bijh ; // compute new residue H ← H + r ; // accumulate residue for output J ← J ∪ { ij } ; // update set of used units if ‖r‖2 < τd then break ; // stop processing when residue is too low else y ← fij ( y ) ; // compute feature map h← φij ( y ‖y‖∞ ) ∥∥∥φij ( y‖y‖∞ ) ∥∥∥2 ; // new hash vector using flattened feature map end end for routing . In this context , similarity is measured by the inner product of flattened feature maps , projected onto different orthogonal subspaces ( each unit basis span ) . Another consequence is that unit weights become specialized in processing a certain type of features , rather than having to adapt to task specific features . This provides the kind of stability needed for continual learning . For a given unit Uk , rank ( Bk ) ≤ m < < s. Hence , it is reasonable to consider that the orthogonal subspace ’ s contribution to total variance is much more important than that of Bk . This is whyH ( x ) only contains projection residues . Note that in Eq.3 , hik ∈ Im ( φik ) and ‖hik‖2 = 1 . The operand under φik has an infinite norm of 1 , which under Eq.2 leads to inner product bounds independent of input data when considering orthogonality . Moreover , due to the approximate orthogonality of different feature hashing functions , summing the residues will not lead to much information loss as each residue vector rik is in Im ( φik−1 ) but this also explains why each unit can only be selected once . The residues ’ ` 1-norms are added to the loss function to induce sparsity . Denoting LT the specific loss for task T ( e.g . KL-divergence for supervised classification ) , the final loss L is : L = LT + λ ∑ j∈ { i0 , ... , id−1 } ‖rj‖1 ( 6 )

Summary: This paper studies the problem of continual learning and proposes a new learning framework named hash-routed convolutional neural networks (HRN). HRN has a set of convolutional units and hashes similar data to the same unit. With this design, the paper claims three key contributions. (1) HRN provides excellent plasticity and more stable features. (2) HRN achieves excellent performance on a variety of benchmarks. (3) HRN can be used for unsupervised or reinforcement learning.