Pay Attention to Features, Transfer Learn Faster CNNs
Year: 2020

Abstract:
Deep convolutional neural networks are now widely deployed in vision applications, but a limited size of training data can restrict their task performance. Transfer learning offers the chance for CNNs to learn with limited data samples by transferring knowledge from models pretrained on large datasets. Blindly transferring all learned features from the source dataset, however, brings unnecessary computation to CNNs on the target task. In this paper, we propose attentive feature distillation and selection (AFDS), which not only adjusts the strength of transfer learning regularization but also dynamically determines the important features to transfer. By deploying AFDS on ResNet-101, we achieved a state-of-the-art computation reduction at the same accuracy budget, outperforming all existing transfer learning methods. With a 10× MACs reduction budget, a ResNet-101 equipped with AFDS transfer learned from ImageNet to Stanford Dogs 120, can achieve an accuracy 11.07% higher than its best competitor.

heading: 1 Introduction
text: Despite recent successes of CNNs achieving state-of-the-art performance in vision applications (Tan & Le, 2019; Cai & Vasconcelos, 2018; Zhao et al., 2018; Ren et al., 2015), there are two major shortcomings limiting their deployments in real life. First, training CNNs from random initializations to achieve high task accuracy generally requires a large amount of data that is expensive to collect. Second, CNNs are typically compute-intensive and memory-demanding, hindering their adoption to power-limited scenarios.
To address the former challenge, transfer learning (Pan & Yang, 2009) is thus designed to transfer knowledge learned from the source task to a target dataset that has limited data samples. In practice, we often choose a source dataset such that the input domain of the source comprises the domain of the target. A common paradigm for transfer learning is to train a model on a large source dataset, and then fine-tune the pre-trained weights with regularization methods on the target dataset (Zagoruyko & Komodakis, 2017; Yim et al., 2017; Li et al., 2018; Li & Hoiem, 2018; Li et al., 2019). For example, one regularization method, L2-SP (Li et al., 2018), penalizes the L2-distances of pretrained weights on the source dataset and the weights being trained on the target dataset. The pretrained source weights serves as a starting point when training on the target data. During fine-tuning on the target dataset, the regularization constrains the search space around this starting point, which in turn prevents overfitting the target dataset.
Intuitively, the responsibility of transfer learning is to preserve the source knowledge acquired by important neurons. The neurons thereby retain their abilities to extract features from the source domain, and contribute to the network’s performance on the target dataset.
∗Equal contribution, corresponding authors. †Work partially done during an internship at Baidu Research.
Moreover, by determining the importance of neurons, unimportant ones can further be removed from computation during inference with network pruning methods (Luo et al., 2017; He et al., 2017; Zhuang et al., 2018; Ye et al., 2018; Gao et al., 2019). The removal of unnecessary compute not only makes CNNs smaller in size but also reduces computational costs while minimizing possible accuracy degradations. As the source domain encompasses the target, many neurons responsible for extracting features from the source domain may become irrelevant to the target domain and can be removed. In Figure 1, a simple empirical study of the channel neurons’ activation magnitudes corroborates our intuition: as deeper layers extract higher-level features, more neurons become either specialized or irrelevant to dogs. The discussion above hence prompts two questions regarding the neurons: which neurons should we transfer source knowledge to, and which are actually important to the target model?
Yet traditional transfer learning methods fail to provide answers to both, as generally they transfer knowledge either equally for each neuron with the same regularized weights, or determine the strength of regularization using only the source dataset (Li et al., 2018). The source domain could be vastly larger than the target, giving importance to weights that are irrelevant to the target task.
Recent years have seen a surge of interest in network pruning techniques, many of which induce sparsity by pushing neuron weights or outputs to zeros, allowing them to be pruned without a detrimental impact on the task accuracies. Even though pruning methods present a solution to neuron/weight importance, unfortunately they do not provide an answer to the latter question, i.e. whether these neurons/weights are important to the target dataset. The reason for this is that pruning optimization objectives are often in conflict with traditional transfer learning, as both drive weight values in different directions: zero for pruning and the initial starting point for transfer learning. As we will see later, a näıve composition of the two methods could have a disastrous impact on the accuracy of a pruned CNN transferlearned on the target dataset.
In this paper, to tackle the challenge of jointly transferring source knowledge and pruning target CNNs, we propose a new method based on attention mechanism (Vaswani et al., 2017), attentive feature distillation and selection (AFDS). For the images in the target dataset, AFDS dynamically learns not only the features to transfer, but also the unimportant neurons to skip.
During transfer learning, instead of fine-tuning with L2-SP regularization which explores the proximity of the pre-trained weights, we argue that a better alternative is to mimic the feature maps, i.e. the output response of each convolutional layer in the source model when images from the target dataset are shown, with L2-distances. This way the fine-tuned model can still learn the behavior of the source model. Additionally, without the restriction of searching only the proximity of the initial position, the weights in the target model can be optimized freely and thus increasing their generalization capacity. Therefore, we present attentive feature distillation (AFD) to learn which relevant features to transfer.
To accelerate the transfer-learned model, we further propose attentive feature selection (AFS) to prune networks dynamically. AFS is designed to learn to predictively select important output channels in the convolution to evaluate and skip unimportant ones, depending
on the input to the convolution. Rarely activated channel neurons can further be removed from the network, reducing the model’s memory footprint.
From an informal perspective, both AFD and AFS learn to adjust the “valves” that control the flow of information for each channel neuron. The former adjusts the strength of regularization, thereby tuning the flow of knowledge being transferred from the source model. The latter allows salient information to pass on to the subsequent layer and stops the flow of unimportant information. A significant attribute that differentiates AFD and AFS from their existing counterparts is that we employ attention mechanisms to adaptively learn to “turn the valves” dynamically with small trainable auxiliary networks.
Our main contributions are as follows:
• We present attentive feature distillation and selection (AFDS) to effectively transfer learn CNNs, and demonstrate state-of-the-art performance on many publicly available datasets with ResNet-101 (He et al., 2016) models transfer learned from ImageNet (Deng et al., 2009).
• We paired a large range of existing transfer learning and network pruning methods, and examined their abilities to trade-off FLOPs with task accuracy.
• By changing the fraction of channel neurons to skip for each convolution, AFDS can further accelerate the transfer learned models while minimizing the impact on task accuracy. We found that AFDS generally provides the best FLOPs and accuracy trade-off when compared to a broad range of paired methods.

heading: 2 Related Work

heading: 2.1 Transfer Learning
text: Training a deep CNN to achieve high accuracy generally require a large amount of training data, which may be expensive to collect. Transfer learning (Pan & Yang, 2009) addresses this challenge by transferring knowledge learned on a large dataset that has a similar domain to the training dataset. A typical approach for CNNs is to first train the model on a large source dataset, and make use of their feature extraction abilities (Donahue et al., 2014; Razavian et al., 2014). Moreover, it has been demonstrated that the task accuracy can be further improved by fine-tuning the resulting pre-trained model on a smaller target dataset with a similar domain but a different task (Yosinski et al., 2014; Azizpour et al., 2015). Li et al. (2018) proposed L2-SP regularization to minimize the L2-distance between each fine-tuned parameter and its initial pre-trained value, thus preserving knowledge learned in the pre-trained model. In addition, they presented L2-SP-Fisher, which further weighs each L2-distance using Fisher information matrix estimated from the source dataset. Instead of constraining the parameter search space, Li et al. (2019) showed that it is often more effective to regularize feature maps during fine-tuning, and further learns which features to pay attention to. Learning without Forgetting (Li & Hoiem, 2018) learns to adapt the model to new tasks, while trying to match the output response on the original task of the original model using knowledge distillation (KD) (Hinton et al., 2014). Methods proposed by Zagoruyko & Komodakis (2017) and Yim et al. (2017) transfer knowledge from a teacher model to a student by regularizing features. The former computes and regularizes spatial statistics across all feature maps channels, whereas the latter estimates the flow of information across layers for each pair of channels, and transfers this knowledge to the student. Instead of manually deciding the regularization penalties and what to regularize as in the previous approaches, Jang et al. (2019) used meta-learning to automatically learn what knowledge to transfer from the teacher and to where in the student model.
Inspired by Li et al. (2019) and Jang et al. (2019), this paper introduces attentive feature distillation (AFD), which similarly transfers knowledge by learning from the teacher’s feature maps. It however differs from Jang et al. (2019) as the teacher and student models share the same network topology, and it instead learns which channel to transfer from the teacher to the student in the same convolutional output.

heading: 2.2 Structured Sparsity
text: Sparsity in neural networks has been a long-studied subject (Reed, 1993; LeCun et al., 1990; Chauvin, 1989; Mozer & Smolensky, 1989; Hassibi et al., 1994). Related techniques have been applied to modern deep CNNs with great success (Guo et al., 2016; Dong et al., 2017a), significantly lowering their storage requirements. In general, as these methods zero out individual weights, producing irregular sparse connections, which cannot be efficiently exploited by GPUs to speed up computation.
For this, many recent work turned their attention to structured sparsity (Alvarez & Salzmann, 2016; Wen et al., 2016; Liu et al., 2017; He et al., 2017; 2018). This approach aims to find coarse-grained sparsity and preserves dense structures, thus allowing conventional GPUs to compute them efficiently. Alvarez & Salzmann (2016) and Wen et al. (2016) both added group Lasso to penalize non-zero weights, and removed channels entirely that have been reduced to zero. Liu et al. (2017) proposed network slimming (NS), which adds L1 regularization to the trainable channel-wise scaling parameters γ used in batch normalization, and gradually prunes channels with small γ values by threshold. He et al. (2018) introduced soft filter pruning (SFP), which iteratively fine-tunes and sets channels with small L2-norms to zero.
Pruning algorithms remove weights or neurons from the network. The network may therefore lose its ability to process some difficult inputs correctly, as the neurons responsible for them are permanently discarded. Gao et al. (2019) have found empirically that task accuracies degrades considerably when most of the computation are removed from the network, and introduced feature boosting and suppression (FBS). Instead of removing neurons permanently from the network, FBS learns to dynamically prune unimportant channels, depending on the current input image. In this paper, attentive feature selection (AFS) builds on top of the advantages of both static and dynamic pruning algorithms. AFS not only preserves neurons that are important to some input images, but also removes unimportant ones for most inputs from the network, reducing both the memory and compute requirements for inference.
There are methods that dynamically select which paths to evaluate in a network dependent on the input (Figurnov et al., 2017; Dong et al., 2017b; Bolukbasi et al., 2017; Lin et al., 2017; Shazeer et al., 2017; Wu et al., 2018; Ren et al., 2018). They however introduce architectural and/or training method changes, and thus cannot be applied directly on existing popular models pre-trained on ImageNet (Deng et al., 2009).

heading: 3 Attentive Feature Distillation and Selection

heading: 3.1 High-Level Overview
text: We begin by providing a high-level overview of attentive feature distillation and selection (AFDS). AFDS introduces two new components to augment each conventional batchnormalized convolutional (ConvBN) layer (Ioffe & Szegedy, 2015), as illustrated in Figure 2. The AFS preemptively learns the importance of each channel, in the output of the ConvBN layer, and can suppress unimportant channels, thus allowing the expensive convolution op-
eration to skip evaluating these channels. The AFD learns the importance of each channel in the output activation, and use the importance as weights to regularize feature maps in the target model with L2-distance. Each component is a small neural network containing a small number of parameters that can be trained with conventional stochastic gradient descent (SGD).

heading: 3.2 Preliminaries
text: Consider a set of training data D where each sample (x, y) consists of an input image x ∈ RC×H×W , and a ground-truth label y ∈ N. Here C, H and W respectively denote the number of channels, and the height and width of the input image. Training a deep CNN classifier thus minimizes the following loss function with an optimization method based on SGD:
L(θ) = E(x,y)∼D[LCE(f(x,θ), y) +R(θ,x) + λ‖θ‖ 2 2], (1)
where θ comprises all parameters of the model, the loss LCE(f(x,θ), y) denotes the crossentropy loss between the CNN output f(x,θ) and the label y. The regularizer R(θ,x) is often used to reduce the risk of overfitting. In conventional training, R(θ,x) = 0. Finally, we impose a L2 penalty on θ, where ‖z‖2 represents the L2-norm of z across all its elements. We assume that f(x,θ) is a feed-forward CNN composed of N ConvBN layers for feature extraction, fl(xl−1,θl) with l ∈ L = {1, 2, . . . , N}, and a final fully-connected layer for classification, g(xN ,θg). Here, for the l
th layer, xl−1 is the input to the layer, with x0 indicating x, and θl is the layer’s parameters. Therefore, the l th layer is defined as:
xl = fl(xl−1,θl) = relu(γl · norm(conv(xl−1,θl)) + βl), (2)
where xl ∈ RCl×Hl×Wl contains Cl feature maps of the layer, each with a Hl height and Wl width. The function conv(xl−1,θl) is a convolution that takes xl−1 as input and uses trainable parameters θl, and norm(z) performs batch normalization. Finally, γl,βl ∈ RCl are trainable vectors, the multiplications (·) and additions (+) are channel-wise, and relu(z) = max(z, 0) stands for the ReLU activation. Although we use the feed-forward classifier above for simplicity, it can be easily modified to contain additional structures such as residual connections (He et al., 2016) and computations for object detection (Ren et al., 2015).
During transfer learning, as we fine-tune the network with a different task, the final layer g(xN ,θg) is generally replaced with a new randomly-initialized one h(xN ,θh). To prevent overfitting, additional terms are used during transfer learning, for instance, L2-SP (Li et al., 2018) further constrains the parameters θl to explore around their initial values θ ? l :
R(θ,x) = λSP ∑ l∈L ‖θl − θ?l ‖ 2 2 + λL2‖θ‖ 2 2. (3)
Instead of regularizing parameters, methods based on knowledge distillation (Hinton et al., 2014) encourages the model to mimic the behavior of the original while learning the target task. Learning without Forgetting (LwF) (Li & Hoiem, 2018) uses the following regularizer to mimic the response from the original classifiers:
R(θ,x) = λLwF LCE(g?(fL(x,θL),θ?g)), (4)
where fL(x,θL) indicates the first N layers, and g ? and θ?g respectively denote the original fully-connected (FC) layer and its associated parameters, and generally λLwF = 1. Zagoruyko & Komodakis (2017), Yim et al. (2017) and Li et al. (2019) chose to regularize feature maps in some intermediate layers L′ ⊆ L. We assume that x?l is the lth layer output of the original model with weights θ? when the input x is shown to the model, and r is a method-dependent function that constrains the relationship between x?l and xl. The regularizer can then be defined as follows:
R(θ,x) = λKD ∑ l∈L′ r(x?l ,xl). (5)

heading: 3.3 Attentive Feature Distillation
text: A simple way to extend Equation (5) is to constrain the L2-norm-distance between x?l and xl, and thus pushing the target model to learn the feature map responses of the source:
R(θ,x) = λFD ∑ l∈L′ ‖x?l − xl‖22. (6)
The above formulation, however, places equal weight to each channel neurons of the feature maps. As we discussed earlier, the importance of channel neurons varies drastically when different input images are shown. it is thus desirable to enforce a different penalty for each channel depending on the input x. For this purpose, we design the regularizer:
R(θ,x) = λAFD ∑ l∈L′ ∑ c∈Cl ρ [c] l (x ? l )‖(x?l − xl)[c]‖22. (7)
Note that in Equation (7), for any tensor z, the term z[c] denotes the cth slice of the tensor. The transfer importance predictor ρl : RCl×Hl×Wl → RCl computes for each channel the importance of the source activation maps, which governs the strength of the L2 regularization for each channel. The predictor function is trainable and is defined as a small network with two FC layers:
ρ [c] l (x ? l ) = softmax(relu([(x ? l )ϕl + νl)ϕ ′ l + ν ′ l). (8)
The function [ : RC×H×W → RC×HW flattens the spatial dimensions in a channel-wise fashion; The parameters ϕl ∈ RHW×H , νl ∈ R1×H , ϕ′l ∈ RH and ν′l ∈ RC can thus be trained to adjust the importance of each channel dynamically; finally, the softmax activation is borrowed from attention mechanism (Vaswani et al., 2017) to normalize the importance values. In our experiments, ϕl and ϕ ′ l use He et al. (2015)’s initialization, νl and ν ′ l are both initialized to 0.

heading: 3.4 Attentive Feature Selection
text: In a fashion similar to feature boosting and suppression (FBS) (Gao et al., 2019), AFS modifies the ConvBN layers from Equation (2):
f̂l(xl−1,θl) = relu(πl(xl−1) · norm(conv(xl−1,θl)) + βl), (9)
where the predictor function takes as input the activation maps of the previous layer, i.e. πl : RCl−1×Hl−1×Wl−1 → RC , is used to replace the vector γl. This function dynamically predicts the importance of each channel, and suppresses certain unimportant channels by setting them to zero. The expensive conv function can hence be accelerated by skipping the disabled output channels. The predictor function is defined as below:
πl(xl−1) = ml · ql(xl−1), where ql(xl−1) = wtaddCle(sl · hl(xl−1) + (1− sl) · γl), (10)
where ml, sl ∈ {0, 1}Cl are both constant masks that take binary values: ml prunes output channels by permanently setting them to zeros, and sl decides for each channel whether the output of hl(xl−1) or γl should be used. It is clear that when ml = 1, no channel neurons are removed from the network. In Section 3.5, we explain how ml and γl can be determined during the fine-tuning process. The winner-take-all function wtaddCle(z) preserves the ddCle most salient values in z, and suppresses the remaining ones by setting them to zeros. The density value 0 < d ≤ 1 is a constant that controls the number of channels to preserve during inference, with 1 preserving all Cl channels. The smaller d gets, the more channels can be skipped, which in turn accelerates the model. Finally, the function hl : RCl−1×H×W → RCl is a small network that is used to predict the importance of each channel. It is composed of a global average pool followed by a FC layer, where pool : RCl−1×H×W → RCl−1 computes the average across the spatial dimensions for each channel:
h(xl−1) = relu(pool(xl−1)ϕ ′′ l + ν ′′ l ). (11)
For the initialization of the FC parameters, we apply He et al. (2015)’s method on the trainable weights ϕ′′l ∈ RCl−1×Cl and ν′′l ∈ RCl is initialized to zeros.

heading: 3.5 Training Procedure
text: In this section, we describe the pipeline of AFDS for transferring knowledge from a source model to a new model by fine-tuning on target dataset. The detailed algorithm can be found in Appendix A.
Initially, we have a pre-trained model f with parameters θ? for the source dataset (e.g. ImageNet). To ensure better accuracies on compressed target models, All ConvBN layers fl in f are extended with AFS as discussed in Section 3.4, with d initially set to 1, which means that all output channels in a convolutional layer are evaluated during inference, i.e. no acceleration. The pre-trained model is then fine-tuned on the target training dataset D with the AFD regularization proposed in Section 3.3.
Empirically we found that in residual networks with greater depths, AFS could become notably challenging to train to high accuracies. To mitigate this, for each output channel of a layer l we update sl according to the variance of hl(xl−1) observed on the target dataset. For each channel if the variance is smaller than a threshold δs, then we set the entry in sl to zero for that particular channel. This action replaces the output of hl(xl−1) with γl, which is a trainable parameter initialized to the mean of hl(xl−1). We compute the mean and variance statistics using Welford (1962)’s online algorithm which can efficiently compute the statistics in a single-pass with O(1) storage. In our experiments, δs is set to a value such that 50% of the channel neurons use the predictor function hl.
Moreover, we discovered that many of the channel neurons are rarely activated in a AFSbased network. We further propose to remove the channel neurons that are activated with a low frequency. In each layer l, the mask ml is used to disable certain channels from the network by setting their output to a constant 0, if the probability of a channel neuron being active is lower than δm. Zeroed-out channels can thus be permanently removed when the model is used in inference.

heading: 4 Experiments
text: In this section we provide an extensive empirical study of the joint methods of transfer learning and channel pruning. We evaluate the methods with 6 different benchmark datasets: Caltech-256 (Griffin et al., 2007) of 256 general object categories; Stanford Dogs 120 (Khosla et al., 2011) specializes to images containing dogs; MIT Indoors 67 (Quattoni & Torralba, 2009) for indoor scene classification; Caltech-UCSD Birds-200-2011 (CUB-200-2011) (Wah et al., 2011) for classifying birds; and Food-101 (Bossard et al., 2014) for food categories. We refer to Li et al. (2018) and Li et al. (2019), for a detailed description of the benchmark datasets. For Caltech-256, we randomly sample either 30 or 60 images from the training set for each category to produce Caltech-256-30 and -60 training datasets.
We use the ResNet-101 from torchvision1 pre-trained on ImageNet as the network for experiments. For ResNet-101 equipped with AFS, we start by extending the pre-trained model and replacing each batch normalization with a randomly initialized AFS, and fine-tune the resulting model on ImageNet for 90 epochs with a learning rate of 0.01 decaying by a factor of 10 every 30 epochs. The resulting model matches its original baseline accuracy.
For each benchmark dataset, the final FC layer of the network is replaced with a new FC randomly initialized with He et al. (2015)’s method to match the number of output categories accordingly. We then perform transfer learning with 4 different methods: L2 (fine-tuning without additional regularization), L2-SP (Li et al., 2018), learning without forgetting (LwF) (Li & Hoiem, 2018), and finally AFD for models using AFS.
To accelerate the resulting fine-tuned models, we continue fine-tuning the model while gradually pruning away channels used during inference. For this, we separately examine 3 pruning strategies: network slimming (NS) (Liu et al., 2017), soft filter pruning (SFP) (He et al., 2018) and finally AFS for models transfer learned with AFD. Note that NS prunes channels by sorting them globally, while SFP does so in a layer-wise manner with identical prune
1https://pytorch.org/docs/stable/torchvision/index.html
ratios. During this procedure, we start with an unpruned model and incrementally remove 10% of the channels used in inference, i.e. preserving 90%, 80%, and etc., down to 10% of all channels for the accelerated models. At each step, we fine-tune each model using 4500 steps of SGD with a batch size of 48, at a learning rate of 0.01, before fine-tuning for a further 4500 steps at a learning rate of 0.001. AFS additionally updates the m and s masks between the two fine-tuning runs.
For each pruned model, we can compute the number of multiply-accumulate operations (MACs) required to perform inference on an image. For each accelerated convolution, the required number of MACs is k2HWCinCout, where Cin and Cout are the number of input and output channels that are not pruned, respectively. We compute the total number of MACs by summing up the MACs in all convolutions, residual connections, and the final pooling and FC layers. For AFS as we dynamically select which channels to evaluate during inference, we additionally add the overhead of the importance predictor layers to the number of total MACs.
In Figure 3, we present the trade-off relationship between the number of vs. the target dataset accuracies for Stanford Dogs and Caltech-256-60. It is clear that AFDS (ours) exceeds various combinations of pruning methods (NS, SFP) and transfer learning methods (L2, L2-SP, LwF). The results for the remaining datasets can be found in Appendix B. The trade-off curves show that AFDS minimizes accuracy degradation even if 47% of the total MACs are removed from the original model, AFDS resulted in only 1.83% drop in accuracy for the model trained on Stanford Dogs. In extreme cases where we permit only 1 10 of the original computations, our method can still manage a 70.70% accuracy, which is substantially better when compared to other pruning algorithms: NS drops to 1.33% and SFP only has 59.63%.
Table 1 provide numerical comparisons of different pruning methods against AFS under various speed-up constraints. Table 2 similarly compares transfer learning strategies against AFD. Under most acceleration requirements, the combined method, AFDS, achieves the best accuracies on the target datasets. Finally, Table 3 compares AFDS against other literatures that performs transfer learning. AFDS can achieve state-of-the-art accuracies when compared to methods that produce models with similar number of MACs.

heading: 5 Conclusion
text: In this paper, we introduced attentive feature distillation and selection (AFDS), a dualattention method that aims to reap the advantages of transfer learning and channel pruning methods. By applying AFDS during fine-tuning, we can not only learn a new model with a higher target task accuracy, but also further accelerates it by computing a subset of channel neurons in each convolutional layers. Under a wide range of datasets, we demonstrated the smallest drop in validation accuracies under the same speed-up constraints when compared to traditional compression methods such as network slimming (Liu et al., 2017) and soft filter pruning (He et al., 2018).

heading: Acknowledgements
text: This work is supported in part by National Key R&D Program of China (No. 2019YFB2102100), Science and Technology Development Fund of Macao S.A.R (FDCT) under number 0015/2019/AKP, Shenzhen Discipline Construction Project for Urban Computing and Data Intelligence, the National Natural Science Foundation of China (Nos. 61806192, 61802387), Shenzhen Science and Technology Innovation Commission (No. JCYJ2017081853518789, JCYJ20190812160003719), the Guangdong Science and Technology Plan Guangdong-Hong Kong Cooperation Innovation Platform (No. 2018B050502009), and China’s Post-doctoral Science Fund (No. 2019M663183).

heading: A The Overall Training Algorithm
text: In Algorithm 1 we illustrate the complete training procedure described above. Here, the function takes as input the target training dataset D, the source model f and its parameters θ?, the total number of steps to fine-tune S, the initial learning rate α, and the threshold hyperparameters δs and δm respectively for sl and ml. The function returns the optimized parameters θ for the target dataset, and both constant masks for all layers s = (s1, s2, . . . , sL) and m = (m1,m2, . . . ,mL). The function SGD then fine-tunes the model parameters. For each layer l, we compute the mean µl and variance σl statistics of ql(xl−1), and use it to compute sl.
Algorithm 1 Training Procedure
1: function AFDS(D, f,θ?, S, α, δs, δm) 2: for l ∈ L : sl ← 1 3: for l ∈ L : ml ← 1 4: θ ← SGD(D, f,θ?, s,m, dS2 e, α,R) 5: for l ∈ L do 6: µl ← E(x,y)∼D[ql(xl−1)] 7: σ2l ← E(x,y)∼D[(ql(xl−1)− µl)2] 8: pl ← E(x,y)∼D[πl(xl−1) > 0] 9: sl ← σ2l > δs
10: γl ← µl 11: ml ← pl > δm 12: end for 13: θ ← SGD(D, f,θ, s,m, dS2 e, α 10 ,R) 14: return θ, s,m 15: end function

heading: B Additional Results

Review summaries:
 This paper proposes a method called attentive feature distillation and selection (AFDS) to improve the performance of transfer learning for CNNs. The authors argue that the regularization should constrain the proximity of feature maps, instead of pre-trained model weights. Specifically, the authors proposes two modifications of loss functions: 1) Attentive feature distillation (AFD), which modifies the regularization term to learn different weights for each channel and 2) Attentive feature selection (AFS), which modifies the ConvBN layers by predicts unimportant channels and suppress them. 

The paper presents an improvement to the task of transfer learning by being deliberate about which channels from the base model are most relevant to the new task at hand. It does this by apply attentive feature selection (AFS) to select channels or features that align well with the down stream task and attentive feature distillation (AFD) to pass on these features to the student network. In the process they do channel pruning there by decreasing the size of the network and enabling faster inference speeds. Their major argument is that plain transfer learning is redundant and wasteful and careful attention applied to selection of the features and channels to be transfered can lead to smaller faster models which in several cases presented in the paper provide superior performance.