Pay Attention to Features, Transfer Learn Faster CNNs

1 Introduction . Despite recent successes of CNNs achieving state-of-the-art performance in vision applications ( Tan & Le , 2019 ; Cai & Vasconcelos , 2018 ; Zhao et al. , 2018 ; Ren et al. , 2015 ) , there are two major shortcomings limiting their deployments in real life . First , training CNNs from random initializations to achieve high task accuracy generally requires a large amount of data that is expensive to collect . Second , CNNs are typically compute-intensive and memory-demanding , hindering their adoption to power-limited scenarios . To address the former challenge , transfer learning ( Pan & Yang , 2009 ) is thus designed to transfer knowledge learned from the source task to a target dataset that has limited data samples . In practice , we often choose a source dataset such that the input domain of the source comprises the domain of the target . A common paradigm for transfer learning is to train a model on a large source dataset , and then fine-tune the pre-trained weights with regularization methods on the target dataset ( Zagoruyko & Komodakis , 2017 ; Yim et al. , 2017 ; Li et al. , 2018 ; Li & Hoiem , 2018 ; Li et al. , 2019 ) . For example , one regularization method , L2-SP ( Li et al. , 2018 ) , penalizes the L2-distances of pretrained weights on the source dataset and the weights being trained on the target dataset . The pretrained source weights serves as a starting point when training on the target data . During fine-tuning on the target dataset , the regularization constrains the search space around this starting point , which in turn prevents overfitting the target dataset . Intuitively , the responsibility of transfer learning is to preserve the source knowledge acquired by important neurons . The neurons thereby retain their abilities to extract features from the source domain , and contribute to the network ’ s performance on the target dataset . ∗Equal contribution , corresponding authors . †Work partially done during an internship at Baidu Research . Moreover , by determining the importance of neurons , unimportant ones can further be removed from computation during inference with network pruning methods ( Luo et al. , 2017 ; He et al. , 2017 ; Zhuang et al. , 2018 ; Ye et al. , 2018 ; Gao et al. , 2019 ) . The removal of unnecessary compute not only makes CNNs smaller in size but also reduces computational costs while minimizing possible accuracy degradations . As the source domain encompasses the target , many neurons responsible for extracting features from the source domain may become irrelevant to the target domain and can be removed . In Figure 1 , a simple empirical study of the channel neurons ’ activation magnitudes corroborates our intuition : as deeper layers extract higher-level features , more neurons become either specialized or irrelevant to dogs . The discussion above hence prompts two questions regarding the neurons : which neurons should we transfer source knowledge to , and which are actually important to the target model ? Yet traditional transfer learning methods fail to provide answers to both , as generally they transfer knowledge either equally for each neuron with the same regularized weights , or determine the strength of regularization using only the source dataset ( Li et al. , 2018 ) . The source domain could be vastly larger than the target , giving importance to weights that are irrelevant to the target task . Recent years have seen a surge of interest in network pruning techniques , many of which induce sparsity by pushing neuron weights or outputs to zeros , allowing them to be pruned without a detrimental impact on the task accuracies . Even though pruning methods present a solution to neuron/weight importance , unfortunately they do not provide an answer to the latter question , i.e . whether these neurons/weights are important to the target dataset . The reason for this is that pruning optimization objectives are often in conflict with traditional transfer learning , as both drive weight values in different directions : zero for pruning and the initial starting point for transfer learning . As we will see later , a näıve composition of the two methods could have a disastrous impact on the accuracy of a pruned CNN transferlearned on the target dataset . In this paper , to tackle the challenge of jointly transferring source knowledge and pruning target CNNs , we propose a new method based on attention mechanism ( Vaswani et al. , 2017 ) , attentive feature distillation and selection ( AFDS ) . For the images in the target dataset , AFDS dynamically learns not only the features to transfer , but also the unimportant neurons to skip . During transfer learning , instead of fine-tuning with L2-SP regularization which explores the proximity of the pre-trained weights , we argue that a better alternative is to mimic the feature maps , i.e . the output response of each convolutional layer in the source model when images from the target dataset are shown , with L2-distances . This way the fine-tuned model can still learn the behavior of the source model . Additionally , without the restriction of searching only the proximity of the initial position , the weights in the target model can be optimized freely and thus increasing their generalization capacity . Therefore , we present attentive feature distillation ( AFD ) to learn which relevant features to transfer . To accelerate the transfer-learned model , we further propose attentive feature selection ( AFS ) to prune networks dynamically . AFS is designed to learn to predictively select important output channels in the convolution to evaluate and skip unimportant ones , depending on the input to the convolution . Rarely activated channel neurons can further be removed from the network , reducing the model ’ s memory footprint . From an informal perspective , both AFD and AFS learn to adjust the “ valves ” that control the flow of information for each channel neuron . The former adjusts the strength of regularization , thereby tuning the flow of knowledge being transferred from the source model . The latter allows salient information to pass on to the subsequent layer and stops the flow of unimportant information . A significant attribute that differentiates AFD and AFS from their existing counterparts is that we employ attention mechanisms to adaptively learn to “ turn the valves ” dynamically with small trainable auxiliary networks . Our main contributions are as follows : • We present attentive feature distillation and selection ( AFDS ) to effectively transfer learn CNNs , and demonstrate state-of-the-art performance on many publicly available datasets with ResNet-101 ( He et al. , 2016 ) models transfer learned from ImageNet ( Deng et al. , 2009 ) . • We paired a large range of existing transfer learning and network pruning methods , and examined their abilities to trade-off FLOPs with task accuracy . • By changing the fraction of channel neurons to skip for each convolution , AFDS can further accelerate the transfer learned models while minimizing the impact on task accuracy . We found that AFDS generally provides the best FLOPs and accuracy trade-off when compared to a broad range of paired methods . 2 Related Work . 2.1 Transfer Learning . Training a deep CNN to achieve high accuracy generally require a large amount of training data , which may be expensive to collect . Transfer learning ( Pan & Yang , 2009 ) addresses this challenge by transferring knowledge learned on a large dataset that has a similar domain to the training dataset . A typical approach for CNNs is to first train the model on a large source dataset , and make use of their feature extraction abilities ( Donahue et al. , 2014 ; Razavian et al. , 2014 ) . Moreover , it has been demonstrated that the task accuracy can be further improved by fine-tuning the resulting pre-trained model on a smaller target dataset with a similar domain but a different task ( Yosinski et al. , 2014 ; Azizpour et al. , 2015 ) . Li et al . ( 2018 ) proposed L2-SP regularization to minimize the L2-distance between each fine-tuned parameter and its initial pre-trained value , thus preserving knowledge learned in the pre-trained model . In addition , they presented L2-SP-Fisher , which further weighs each L2-distance using Fisher information matrix estimated from the source dataset . Instead of constraining the parameter search space , Li et al . ( 2019 ) showed that it is often more effective to regularize feature maps during fine-tuning , and further learns which features to pay attention to . Learning without Forgetting ( Li & Hoiem , 2018 ) learns to adapt the model to new tasks , while trying to match the output response on the original task of the original model using knowledge distillation ( KD ) ( Hinton et al. , 2014 ) . Methods proposed by Zagoruyko & Komodakis ( 2017 ) and Yim et al . ( 2017 ) transfer knowledge from a teacher model to a student by regularizing features . The former computes and regularizes spatial statistics across all feature maps channels , whereas the latter estimates the flow of information across layers for each pair of channels , and transfers this knowledge to the student . Instead of manually deciding the regularization penalties and what to regularize as in the previous approaches , Jang et al . ( 2019 ) used meta-learning to automatically learn what knowledge to transfer from the teacher and to where in the student model . Inspired by Li et al . ( 2019 ) and Jang et al . ( 2019 ) , this paper introduces attentive feature distillation ( AFD ) , which similarly transfers knowledge by learning from the teacher ’ s feature maps . It however differs from Jang et al . ( 2019 ) as the teacher and student models share the same network topology , and it instead learns which channel to transfer from the teacher to the student in the same convolutional output . 2.2 Structured Sparsity . Sparsity in neural networks has been a long-studied subject ( Reed , 1993 ; LeCun et al. , 1990 ; Chauvin , 1989 ; Mozer & Smolensky , 1989 ; Hassibi et al. , 1994 ) . Related techniques have been applied to modern deep CNNs with great success ( Guo et al. , 2016 ; Dong et al. , 2017a ) , significantly lowering their storage requirements . In general , as these methods zero out individual weights , producing irregular sparse connections , which can not be efficiently exploited by GPUs to speed up computation . For this , many recent work turned their attention to structured sparsity ( Alvarez & Salzmann , 2016 ; Wen et al. , 2016 ; Liu et al. , 2017 ; He et al. , 2017 ; 2018 ) . This approach aims to find coarse-grained sparsity and preserves dense structures , thus allowing conventional GPUs to compute them efficiently . Alvarez & Salzmann ( 2016 ) and Wen et al . ( 2016 ) both added group Lasso to penalize non-zero weights , and removed channels entirely that have been reduced to zero . Liu et al . ( 2017 ) proposed network slimming ( NS ) , which adds L1 regularization to the trainable channel-wise scaling parameters γ used in batch normalization , and gradually prunes channels with small γ values by threshold . He et al . ( 2018 ) introduced soft filter pruning ( SFP ) , which iteratively fine-tunes and sets channels with small L2-norms to zero . Pruning algorithms remove weights or neurons from the network . The network may therefore lose its ability to process some difficult inputs correctly , as the neurons responsible for them are permanently discarded . Gao et al . ( 2019 ) have found empirically that task accuracies degrades considerably when most of the computation are removed from the network , and introduced feature boosting and suppression ( FBS ) . Instead of removing neurons permanently from the network , FBS learns to dynamically prune unimportant channels , depending on the current input image . In this paper , attentive feature selection ( AFS ) builds on top of the advantages of both static and dynamic pruning algorithms . AFS not only preserves neurons that are important to some input images , but also removes unimportant ones for most inputs from the network , reducing both the memory and compute requirements for inference . There are methods that dynamically select which paths to evaluate in a network dependent on the input ( Figurnov et al. , 2017 ; Dong et al. , 2017b ; Bolukbasi et al. , 2017 ; Lin et al. , 2017 ; Shazeer et al. , 2017 ; Wu et al. , 2018 ; Ren et al. , 2018 ) . They however introduce architectural and/or training method changes , and thus can not be applied directly on existing popular models pre-trained on ImageNet ( Deng et al. , 2009 ) .

Summary:  This paper proposes a method called attentive feature distillation and selection (AFDS) to improve the performance of transfer learning for CNNs. The authors argue that the regularization should constrain the proximity of feature maps, instead of pre-trained model weights. Specifically, the authors proposes two modifications of loss functions: 1) Attentive feature distillation (AFD), which modifies the regularization term to learn different weights for each channel and 2) Attentive feature selection (AFS), which modifies the ConvBN layers by predicts unimportant channels and suppress them.