DELTA: DEEP LEARNING TRANSFER USING FEATURE MAP WITH ATTENTION FOR CONVOLUTIONAL NETWORKS

1 INTRODUCTION . In many real-world applications , deep learning practitioners often have limited number of training instances . Direct training a deep neural network with a small training data set usually results in the so-called over-fitting problem and the quality of the obtained model is low . A simple yet effective approach to obtain high-quality deep learning models is to perform weight fine-tuning . In such practices , a deep neural network is first trained using a large ( and possibly irrelevant ) source dataset ( e.g . ImageNet ) . The weights of such a network are then fine-tuned using the data from the target application domain . Fine-tuning is a specific approach to perform transfer learning in deep learning . The weights pretrained by the source dataset with a sufficiently large number of instances usually provide a better initialization for the target task than random initializations . In a typical fine-tuning approach , weights in lower convolution layers are fixed and weights in upper layers are re-trained using data from the target domain . In this approach parameters of the target model may be driven far away from initial values , which also causes over-fitting in transfer learning scenarios . Approaches called regularization using the starting point as the reference ( SPAR ) , were recently proposed to solve the over-fitting problem . For example , Li et al . ( Li et al. , 2018 ) proposed L2-SP that incorporates the Euclid distance between the target weights and the starting point ( i.e. , weights of source network ) as part of the loss . Minimizing this loss function , L2-SP aims to minimize the empirical loss of deep learning while reducing the distance of weights between source and target networks . They achieved significant improvement compared with standard practice of using the weight decay ( L2 normalization ) . However such regularization method may not deliver optimal solution for transfer learning . On one side , if the regularization is not strong , even with fine-turning , the weights may still be driven far away from the initial position , leading to the lose of useful knowledge , i.e . catastrophic memory loss . On the other side , if the regularization is too strong , newly obtained model is constrained to a local neighborhood of the original model , which may be suboptimal to the target data set . Although aforementioned methods demonstrated the power of regularization in deep transfer learning , we argue that we need to perform research on at least the following two aspects in order to further improve current regularization methods . Behavior vs. Mechanisms . The practice of weight regularization for CNN is motivated by a simple intuition — the network ( layers ) with similar weights should produce similar outputs . However , due to the complex structures of deep neural network with strong redundancies , regulating the model parameters directly seems an over-killing of the problem . We argue that we should regularize the “ Behavior ” , or in our case , the outer layer outputs ( e.g . the feature maps ) produced by each layer , rather than model parameters . With constrained feature maps , the generalization capacity could be improved through aligning the behaviors of the outer layers of the target network to the source one , which has been pre-trained using an extremely large dataset . In Convolutional Neural Networks , which we focus on exclusively in this paper , an outer layer is a convolution layer and the output of an outer layer is its feature map . Syntax vs Semantics . While regularizing the feature maps might improve the transfer of generalization capacity , it is still difficult to design such regularizers . It is challenging to measure the similarity/distance between the feature maps without understanding its semantics or representations . For example for image classification , some of the convolution kernels may be corresponding to features that are shared between the two learning tasks and hence should be preserved in transfer learning while others are specific to the source task and hence could be eliminated in transfer learning . In this paper , we propose a novel regularization approach DELTA to address the two issues . Specifically , DELTA selects the discriminative features from outer layer outputs through re-weighting the feature maps with a novel supervised attention mechanism . Through paying attention to discriminative parts of feature maps , DELTA characterizes the distance between source/target networks using their outer layer outputs , and incorporates such distance as the regularization term of the loss function . With the back-propagation , such regularization finally affects the optimization for weights of deep neural network and awards the target network generalization capacity inherited from the source network . In summary , our key insight is what we call ” unactivated channel re-usage ” . Specifically our approach identifies those transferable channels and preserves such filters through regularization and identify those untransferable channels and reuse them , using an attention mechanism with feature map regularization . We have conducted extensive experiments using a wide range of source/target datasets and compared DELTA to the existing deep transfer learning algorithms that are in pursuit of weight similarity . The experiment results show that DELTA significantly outperformed the state-of-the-art regularization algorithms including L2 and L2-SP with higher accuracy on a wide group of image classification data sets . The rest of the paper is organized as follows : in Section 2 related works are summarized , in Section 3 our feature map based regularization method is introduced , in Section 4 experimental results are presented and discussed , and finally in Section 5 the paper is concluded . 2 RELATED WORK AND BACKGROUNDS . In this section , we first review the related works to this paper , where we discuss the contributions made by this work beyond previous studies . Then , we present the backgrounds of our work . 2.1 RELATED WORK . Transfer learning is a type of machine learning paradigm aiming at transferring the knowledge obtained in a source task to a target task ( Caruana , 1997 ; Pan et al. , 2010 ) . Our work primarily focuses on inductive transfer learning for deep neural networks , where the label space of the target task dif- fers from that of the source task . For example , Donahue et al ( Donahue et al. , 2014 ) proposed to train a classifier based on feature extracted from a pre-trained CNN , where a large mount of parameters , such as filters , of the source network are reused directly in the target one . This method may overload the target network with tons of irrelevant features ( without discrimination power ) involved , while the key features of the target task might be ignored . To understand whether a feature can be transferred to the target network , Yosinki et al . ( Yosinski et al. , 2014 ) quantified the transferability of features from each layer considering the performance gain . Moreover , to understand the factors that may affect deep transfer learning performance , Huh et al . ( Huh et al. , 2016 ) empirically analyzed the features obtained by the ImageNet pre-trained source network to a wide range of computer vision tasks . Recently , more studies to improve the inductive transfer learning from a diverse set of angles have been proposed , such as filter subset selection ( Ge & Yu , 2017 ; Cui et al. , 2018 ) , sparse transfer ( Liu et al. , 2017 ) , filter distribution constraining ( Aygun et al. , 2017 ) , and parameter transfer ( Zhang et al. , 2018 ) . For deep transfer learning problems , the most relevant work to our study is ( Li et al. , 2018 ) , where authors investigated regularization schemes to accelerate deep transfer learning while preventing fine-tuning from over-fitting . Their work showed that a simple L2-norm regularization on top of the “ Starting Point as a Reference ” optimization can significantly outperform a wide range of regularization-based deep transfer learning mechanisms , such as the standard L2-norm regularization . Compared to above work , the key contributions made in this paper include 1 ) rather than regularizing the distance between the parameters of source network and target network , DELTA constrains theL2-norm of the difference between their behaviors ( i.e. , the feature maps of outer layer outputs in the source/target networks ) ; and 2 ) the regularization term used in DELTA incorporates a supervised attention mechanism , which re-weights regularizers according to their performance gain/loss . In terms of methodologies , our work is also related to the knowledge distillation for model compression ( Hinton et al. , 2015 ; Romero et al. , 2014 ) . Generally , knowledge distillation focuses on teacher-student network training , where the teacher and student networks are usually based on the same task ( Hinton et al. , 2015 ) . These work frequently intends to transfer the knowledge in the teacher network to the student one through aligning their outputs of some layers ( Romero et al. , 2014 ) . The most close works to this paper are ( Zagoruyko & Komodakis , 2016 ; Yim et al. , 2017 ) , where knowledge distillation technique has been studied to improve transfer learning . Compared to above work , our work , including other transfer learning studies , intends to transfer knowledge between different source/target tasks ( i.e. , source and target tasks ) , though the source/target networks can be viewed as teachers and students respectively . We follow the conceptual ideas of knowledge distillation to regularize the outer layer outputs of the network ( i.e. , feature maps ) , yet further extend such regularization to a supervised transfer learning mechanism by incorporating the labels of target task ( which is different from the source task/network ) . Moreover , a supervised attention mechanism has been adopted to regularize the feature maps according to the importance of filters . Other works relevant to our methodology include : continual learning ( Kirkpatrick et al. , 2017 ; Li & Hoiem , 2017 ) , attention mechanism for CNN models ( Mnih et al. , 2014 ; Xu et al. , 2015 ; Yang et al. , 2016 ; Zagoruyko & Komodakis , 2016 ) , among others . 2.2 BACKGROUNDS . Deep convolutional networks usually consist of a great number of parameters that need fit to the dataset . For example , ResNet-110 has more than one million free parameters . The size of free parameters causes the risk of over-fitting . Regularization is the technique to reduce this risk by constraining the parameters within a limited space . The general regularization problem is usually formulated as follow . 2.2.1 GENERAL REGULARIZATION . Let ’ s denote the dataset for the desired task as { ( x1 , y1 ) , ( x2 , y2 ) , ( x3 , y3 ) . . . , ( xn , yn ) } , where totally n tuples are offered and each tuple ( xi , yi ) refers to the input image and its label in the dataset . We further denote ω ∈ Rd be the d-dimensional parameter vector containing all d parameters of the target model . The optimization object with regularization is to obtain min w n∑ i=1 L ( z ( xi , ω ) , yi ) + λ · Ω ( ω ) , ( 1 ) where the first term ∑n i=1 L ( z ( xi , ω ) , yi ) refers to the empirical loss of data fitting while the second term is a general form of regularization . The tuning parameter λ > 0 balances the trade-off between the empirical loss and the regularization loss . Without any explicit information ( such as other datasets ) given , one can easily use the L0/L1/L2-norm of the parameter vector ω as the regularization to fix the consistency issue of the network .

Summary: This is a reasonable paper based on a simple intuition. The authors have noticed that some of the state of the art methods (they use Li et al - ICML18 as the main reference) are using only some simple normalization for improving the transfer learning and as such they propose preserving the outer layer output of the target network and aligning it with the one of the source network. On top of that they also propose modeling the difference of feature maps considering an attention mechanism obtain through supervised learning.