Pay Attention to Features, Transfer Learn Faster CNNs
Authors: Kafeng Wang
Xitong Gao
Yiren Zhao
Xingjian Li
Dejing Dou
Cheng-Zhong Xu
Venue:  (2020)
Keywords: transfer learning
pruning
faster CNNs

Abstract: Deep convolutional neural networks are now widely deployed in vision applications, but a limited size of training data can restrict their task performance. Transfer learning offers the chance for CNNs to learn with limited data samples by transferring knowledge from models pretrained on large datasets. Blindly transferring all learned features from the source dataset, however, brings unnecessary computation to CNNs on the target task. In this paper, we propose attentive feature distillation and selection (AFDS), which not only adjusts the strength of transfer learning regularization but also dynamically determines the important features to transfer. By deploying AFDS on ResNet-101, we achieved a state-of-the-art computation reduction at the same accuracy budget, outperforming all existing transfer learning methods. With a 10x MACs reduction budget, a ResNet-101 equipped with AFDS transfer learned from ImageNet to Stanford Dogs 120, can achieve an accuracy 11.07% higher than its best competitor.

Published as a conference paper at ICLR 2020
PAY ATTENTION TO FEATURES, TRANSFER LEARN FASTER CNNS
Kafeng Wang Xitong Gao²; Yiren Zhao? Xingjian Li; Dejing Dou Cheng-Zhong X11
1.2 Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences. University of Chinese Academy of Sciences. University of Cambridge. 4,5 Big Data Lab, Baidu Research. "niieessiy of Macau. kf. wangdsiat. ac. cn, 3xt. eet.gao@siat.ae ac.
ABSTRACT
Deep convolutional neural networks are now widely deployed in vision ap- plications, but a limited size of training data can restrict their task perfor- mance. Transfer learning offers the chance for CNNs to learn with limited data samples by transferring knowledge from models pretrained on large datasets. Blindly transferring all learned features from the source dataset, however, brings unnecessary computation to CNNs on the target task. In this paper, we propose attentive feature distillation and selection (AFDS), which not only adjusts the strength of transfer learning regularization but also dynamically determines the important features to transfer. By de- ploying AFDS on ResNet-101, we achieved a statc-of-the-art computation reduction at the same accuracy budget, outperforming all existing trans- fer learning methods. With a 10x MACs reduction budget, a ResNet-101 equipped with AFDS transfer learned from ImageNet to Stanford Dogs 120, can achieve an accuracy 11.07% higher than its best competitor.
1 INTRODUCTION
Despite recent successes of CNNs achieving state-of-the-art performance in vision applica- tions (Tan & Le, 2019; Cai & Vasconeelos, 2018; Zhao al., 2018; Ren al., 2015), there are two major shorteomings limiting their deployments in real life. First, training CNNs from random initializations to achieve high lask accuracy generally requires a large amount of data that is expensive to collect. Second, CNNs are typically compute-intensive and memory-demanding, hindering their adoption to power-limited scenarios.
To address the former challenge, transfer learning (Pan & Yang, 2009) is thus designed to transfer knowledge learned from the source task to a target dataset that has limited data samples. In practice often choose a source dataset such that the input domain of the source comprises the domain the target. common paradigm for transfer learning is to train model on large source dataset, and then fine-tune the pre-trained weights with regularization methods on the target dataset (Zagoruyko & Komodakis, 2017; Yim et al., 2017: Li et al., 2018; Li & Hoiem, 2018; Ti et al., 2019). For example, one regularization method, S (Li et al., 2018), penalizes the L2-distances of pretrained weights on the source dataset and the weights being trained on the target dataset. The pretrained source weights serves as starting point when training on the target data. During fine-tuning on the target dataset, the rogularization constrains the scarch space around this starting point, which in turn prevents overfitting the target dataset.
Intuitively, the responsibility of transfer learning is to preserve the source knowledge acquired by important neurons. The neurons thereby retain their abilities to extract features from the source domain, and contribute to the network's performance on the target dataset.
"Equal contribution, corresponding authors.
Work partially done during an internship Baidu Research.

---

Published as a conference paper at ICLR 2020
Moreover, by determining the importance of neurons, unimportant ones can further be removed from computation during inference with network pruning methods (Luo et al., 2017; He et al. 2017; Zhuang et al., 2018; Ye et al., 2018; Gao et al., 2019). The removal of unnecessary compute not only makes CNNs smaller in size but also reduces computational costs while minimizing possible accuracy degradations. As the source domain encompasses the target, many neurons responsible for extracting features from the source domain may become irrelevant the target domain and can be removed. In Figure 1, a simple empirical study of the channel neurons' activation magnitudes corroborates our intuition: as deeper layers extract higher-level features, more neurons become either specialized or irrelevant to dogs. The discussion above hence prompls two questions regarding the neurons: which neurons should we transfer source. knowledge to, and which are actually imporlant to the larget model?
Imagahist
3b3
4b22
5c
Drgs Imageltel
Degs Imagehel
Dogs
Imagake:
0
10
20
100
20
10
20
(a) Example images.
(b) Maximum channel activations.
Figure 1: (a) shows sample images from two datasets, ImageNet contains images with greater diversity. (b) shows the average maximum activations of 20 channel neurons layers of ResNet-101 that are most excited by images from Dogs.
Yet traditional transfer learning methods fail to provide answers to both, as generally they transfer knowledge either equally for each neuron with the same regularized weights, or determine the strength of regularization using only the source dataset (Li et al., 2018). The source domain could be vastly larger than the target, giving importance to weights that are irrelevant to the target task.
Recent years have seen surge of interest in network pruning techniques, many of which induce sparsity by pushing neuron weights or outputs to zeros, allowing them to be pruned without a detrimental impact on the lask accuracies. Even though pruning methods present a solution to neuron neuron/weight weight importance, unfortunately they do not provide an answer to the latter question, i.e. whether these neurons/ seurons/weeghhts weights are important to the target dataset. The reason for this is that pruning optimization objectives are often in conflict with traditional transfer learning, both drive weight values in different directions: zero for pruning and the initial starting point for transfer learning. As we will see later, a naive composition of the two methods could have a disastrous impact on the accuracy of a pruned CNN transfer- learned on the target dataset.
In this paper, to tackle the challenge of jointly transferring source knowledge and pruning target CNNs, we propose a new method based on attention mechanism (Vaswani et al., 2017), attentive feature distillation and selection (AFDS). For the images in the target dataset, AFDS dynamically learns not only the features to transfer, but also the unimportant neurons to skip.
During transfer learning, instead of fine-tuning with - regularization which explores the proximity of the pre-trained weights, we argue that better alternative is to mimic the feature maps, i.e. the output response of each convolutional layer in the source model when images from the target dataset are shown, with 12-distances. This way the fine-tuned model can still learn the behavior of the source model. Additionally, without the restriction of searching only the proximity of the initial position, the weights in the target model can be optimized freely and thus incrcasing their generalization capacity. Therefore, present attentive feature distillation (AFD) to learn which relovant foatures to transfer.
To accelerate the transeer-learned model, we further propose attentive feature selection (AFS) to prune networks dynamically. AFS is designed to learn to predictively select impor- lant output channels in the convolution to evaluate and skip unimportant ones, depending
2

---

Published as a conference paper at ICLR 2020
on the imput to the convolution. Rarcly activated channel neurons can further be removed from the network, reducing the model's memory footprint.
From informal perspective, both AFD and AFS learn to adjust the "valves" that control the flow of information for each channel neuron. The former adjusts the strength of regu- larization, thereby tuning the flow of knowledge being transferred from the source model. The latter allows salient information to pass on to the subsequent layer and stops the flow of unimportant information. A significant attribute that differentiates AFD and AFS from their existing counterparts is that we employ attention mechanisms to adaptively learn to "turn the valves" dynamically with small trainable auxiliary networks.
Our main contributions are as follows:
We present attentive feature distillation and selection (AFDS) to effectively trans- for learn CNNs, and demonstrate srato-of-thh-ars performance on many publicly available datasets with ResNet-101 (He et al., 2016) models transfer learned from ImageNet (Deng et al., 2009).
We paired a large range existing transfer learning and network pruning methods, and examined their abilities to trade-off FLOPs with task accuracy.
By changing the fraction of channel neurons to skip for each convolution, AFDS can further accelerate the transfer learned models while minimizing the impact on task accuracy. We found that AFDS generally provides the best FLOPs and accuracy trade-off when compared to a broad range of paired methods.
2 RELATED WORK
2.1 TRANSFER LEARNING
Training deep CNN to achieve high accuracy generally require large amount of training data, which may be expensive to collect. Transfer learning (Pan & Yang, 2009) addresses this challenge by transferring knowledge learned a large dataset that has similar domain to the training dataset. A typical approach for CNNs is to first train the model on a large source dataset, and make use of their feature extraction abilities (Donahue Razavian et al., 2014). Moreover, it has been demonstrated that the task accuracy can be further improved by fine-tuning the resulting pre-trained model on a smaller target datasct with a similar domain but a different task (Yosinski et al., 2014; Azizpour et al., 2015). Li et al. (2018) proposed IPPSPP regularization to minimize the I²-distance between each fine-tuned parameter and its initial pre-trained value, thus preserving knowledge learned in the pre-trained model. In addition, they presented L'=SP-Fishee, which further weighs each 12-distance using Fisher information matrix estimated from the source dataset. Instead of constraining the parameter search space, Li et al. (2019) showed that it is often more effective to regularize feature maps during fine-tuning, and further learns which features to pay attention to. Learning without Forgetting (Li & Hoiem, 2018) learns to adapt the model to new tasks, while trying to match the output response on the original task of the original model using knowledge distillation (KD) (Hinton et al., 2014). Methods proposed by Zagoruyko & Komodakis (2017) and Yim et al. (2017) transfer knowledge from teacher model to a student by regularizing features. The former computes and regularizes spatial statistics across all feature maps channels, whereas the latter estimates the flow of information across layers for each pair of channels, and transfers this knowledge to the student. Instead of manually deciding the regularization penalties and what to regularize as in the previous approaches, Jang et al. (2019) used meta-learning to automatically learn what knowledge to transfer from the teacher and to where in the student model.
al., 2014;
Inspired by Li et al. (2019) and Jang et al. (2019), this paper introduces attentive feature distillation (AFD), which similarly transfers knowledge by learning from the teacher's fea- ture maps It however differs from Jang et al. (2019) the teacher and student models share the same network topology, and it instead learns which channel to transfer from the teacher to the student in the same convolutional output.
3

---

Published as a conference paper at ICLR 2020
2.2 STRUCTURED SPARSITY
Sparsity in neural networks has been a long-studied subject (Reed, 1993; LeCun et al., 1990; Chauvin, 1989; Mozer & Smolensky, 1989; Hassibi et al., 1994). Related techniques have been applied to modern deep CNNs with great success (Guo et al., 2016; Dong et al,, 2017a), significantly lowering their storage requirements. In general, as these methods zero out individual weights, producing irregular sparse connections, which cannot be efficiently exploited by GPUs to speed up computation.
For this, many recent work turned their attention to structured sparsity (Alvarez & Salz- mann, 2016; Wen al., 2016; Liu et al., 2017; He et al., 2017; 2018). This approach aims to find coarse-grained sparsity and preserves dense structures, thus allowing conventional GPUs compule them efficiently. Alvarez Salzmann (2016) and Wen et al. (2016) both added group Lasso to penalize non-zero weights, and removed channels entirely that have been reduced to zero. Liu et al. (2017) proposed network slimming (NS); which adds L' reg- ularization to the trainable channel-wizg wise scaling parameters used in batch normalization, and gradually prunes channels with small y values by threshold. He et al. (2018) introduced soft filter pruning (SFP), which itcratively fino-tunes and sets channels with small L L²2norms to zero.
Pruning algorithms remove weights neurons from the network. The network may there- fore lose its ability to process some difficult inputs correctly, as the neurons responsible for them are permanently discarded, Gao al. (2019) have found empirically that task accu- racies degrades considerably when most of the computation are removed from the network, and introduced feature boosting and suppression (FBS). Instead of removing neurons perma- nently from the network, FBS learns to dynamically prune unimportant channels, depending on the current input image. In this paper, attentive feature selection (AFS) builds on top of the advantages of both static and dynamic pruning algorithmss. AFS not only preserves neurons that are important to some input images, but also removes unimportant ones for most inputs from the network, reducing both the memory and compute requirements for inference.
There are methods that dynamically select which paths to evaluate in network dependent on the input (Figurnov et al., 2017: Dong et al., 2017b; Bolukbasi et al., 2017: Lin et al., 2017; Shazeer et al., 2017; Wu et al., 2018; Ren et al., 2018). They however introduce architectural and/or training mothod changes, and thus cannot be applied directly on existing popular models pre-trained ImageNet (Deng et al., 2009).
ATTENTIVE FEATURE DISTILLATION AND SELECTION
3.1 HIGE-LEVEL OVERVIEW
Source model
AFS
Conv+BN layer
AFD
Target model
L2-dist
AFD loss
AFS
Task loss
Conv+BN BN layer
Figure 2: High-level overview of AFDS.
We begin by providing high-level overview of attentive feature distillation and selec- tion (AFDS). AFDS introduces two new components to augment each conventional batch- normalized convolutional (ConvBN) layer (Ioffe & Szegedy, 2015), as illustrated in Figure 2. The AFS preemptively learns the importance of each channel, in the output of the ConvBN layer, and can suppress unimportant channels, thus allowing the expensive convolution op-
4

---

Published as a conference paper at ICLR 2020
eration to skip evaluating these channels. The AFD learns the importance of each channel in the output activation, and use the importance as weights to regularize feature maps in the target model with d'-distance. Each component is small neural network containing a small number of parameters that can be trained with conventional stochastic gradient descent (SGD).
3.2 PRELIMINARIES
Consider a set of training data D where each sample (æ, y) consists of an imput image a E (CxxHWW and a ground-truti truth label y E N. Here C, and W respectively denote the number of channels, and the height and width of the imput image. Training deep CNN classifier thus minimizes the following loss function with an optimization method based on SGD:
(()) = ( (/(e,, 0). R(0, x) + (11011)
(1)
where O comprises all parameters of the model, the loss L (f( 0). v) denotes the cross- entropy loss between the CNN output f f(x 0) and the label y. The regularizer R(O, æ) is often used to reduce the risk of overfitting. In conventional training, R(0, x) O. Finally, we impose L penalty on O, where ||z||z represents the - of z across all its elements.
We assume that f(x, 0) is feed-forward CNN composed of N ConvBN layers for feature extraction, f((2)11,,1) 0:) with E L = f1.2....,NN NY. and a final fully-connected layer for classification, g(æx, 0g). Here, for the 8th layer, Ø;-1 is the input to the layer, with To indicating x, and 0: is the layer's parameters. Therefore, the 7th layer is defined as:
(2) where 21 E RCxcH.WW contains Ct feature maps of the layer, each with a Hi height and Wi width. The function conv(x)-1. B1) is a convolution that takes x;-1 as input and uses trainable parameters 0,, and norm(z) performs batch normalization. Finally, nt, Br E R are trainable vectors, the multiplications (-) and additions (+) are channel-wist and relu(z)= max(z, 0) stands for the ReL.U activation. Although we the feed-forward classifier above for simplicity, it can be easily modified to contain additional structures such as residual connections (He et al., 2016) and computations for object detection (Ren al., 2015).
x fi( 0.) relu(> norm(conv(x11110) 01)) + Bi),
During transfer learning, as we fine-tune the network with a different Lask, the final layer g(øN 0g) is generally replaced with a new randomly-inttialized one h(zN, 0%). ). To prevent overfitting, additional terms are used during transfer learning, for instance, I--SPP (Li et al., 2018) further constrains the parameters 0, to explore around their initial values 07:
R(0,x)) = AsP Σ10, 0, 212l101/
Instead of regularizing paramcters, methods based on knowledge distillation (Hinton et al., 2014) encourages the model to mimic the behavior of the original while learning the target task. Learning without Forgetting (LwF) (Li & Hoiem, 2018) uses the following regularizer to mimic the response from the original classifiers:
R(0, x) = ALWF C (g (fL(
(4)
where fz 0z) indicates the first N layers, and 9" and 0f respectively denote the orig- inal fully-connected (FC) layer and its associated parameters, and generally ÅI.wF = 1. Zagoruyko Komodakis (2017), Yim et al. (2017) and Li et al. (2019) chose to regularize feature maps in some intermediate layers L' C L. We assume that क is the tth layer out- put of the original model with weights 0* when the input æ is shown to the model, and r is method-dependent function that constrains the relationship between af and m. The regularizer can then be defined as follows:
R(0, x)) = AKD Σ r(xj,
(5)
leb
5

---

Published as a conference paper at ICLR 2020
3.3 ATTENTIVE FEATURE DISTILLATION
simple way to extend Equation (5) is to constrain the L d--norm-distance between oं and st, and thus pushing the larget model to learn the feature map responses of the source:
R(0, = AFD Σ
c -2 22
(6)
LET
The above formulation, however, places equal weight to each channel neurons of the feature maps. As we discussed earlier, the importance channel neurons varies drastically when different input images shown. it is thus desirable to enforce different penalty for each channel depending on the imput æ. For this purpose, we design the regularizer:
R(0,x) NAED ΣΣ TEL'
p (
Note that in Equation (7), for any tensor 2, the term z/0 denotes the tt slice of the tensor. The transfer importance predictor Pr RCixHixWi + R computes for each channel the importance of the source activation maps, which governs the strength of the L regularization for each channel. The predictor function is trainable and is defined as a small network with two FC layers:
Pi |ol (at) = softmaxx(eel(((a)))))) + v1) v) +1).
(8)
The function b C + RC flattens the spatial dimensions in channel-wise fashion; The parameters yp: E "MW" i E RIxl s E R" and vi E RC can thus be trained to adjust the importance of each channel dynamically; finally, the softmax activation borrowed from attention mechanism (Vaswani et al., 2017) to normalize the importance values. In experiments, 60: and s He et al. (2015)'s initialization, and ví are both initialized to 0.
3.4 ATTENTIVE FEATURE SELECTION
In fashion similar to feature boosting and suppression (FBS) (Gao et al., 2019), AFS modifies the ConvBN layers from Equation (2):
fi( 0:) relu(m (x1-1) norm(conv(21111,00 0:)) +,))),
(9)
where the predictor function takes as input the activation maps of the previous layer, i.e. m -C--IxH--IXW>-- + RC, is used to replace the vector 7. This function dynamically predicts the importance of each channel, and suppresses certain unimportant channels by setting them to zero. The expensive conv function can hence be accelerated by skipping the disabled output channels. The predictor function is defined as below:
(x:-1) m: ((()-1)), where (xx-11) wtapao (sl h=(2111) + (1 (1)s:) 7) (10)
where m 10 are both constant masks that take binary values: my prunes output channels by permanently setting them to zeros, and s decides for each channel whether the output of h=(x)-1) or 7) should be used. It is clear that when m, = 1, no channel neurons are removed from the network. In Section 3.5, we explain how m, and Y can be determined during the fine-tuning process. The winner-taak-all. function wtarac;1 preserves the [dC]] most salient values in and suppresses the remaining ones by setting them to zeros. The density value 0 < d < 1 is constant that controls the number of channels preserve during inference, with 1 preserving all Ct channels. The smaller d gets, the more channels can be skipped, which in turn accelerates the model. Finally, the function h; 1 Hxx RC is small network that is used to predict the importance of each channel. It is composed of a global average pool followed by a FC layer, where pool: : : I 4 RC--1 computes the average across the spatial dimensions for each channel:
h( = relu(poo((z=1199" +2²).
(11)
For the initialization of the FC parameters, we apply He et al. (2015)'s method on the trainable weights sp" RC-IX and vi RC is initialized to zeros.

---

Published as a conference paper at ICLR 2020
3.5 TRAINING PROCEDURE
In this section, we describe the pipeline of AFDS for transferring knowledge from a source model to a new model by fine-tuning on larget dataset. The detailed algorithm can be found in Appendix A.
Initially, we have a pre-trained model f with parameters 0 for the souree dataset (e.g. Im- ageNet). To ensure better accuracies compressed target models, All ConvBN layers fi in I are extended with AFS as discussed in Section 3.4, with d initially set to 1, which means that all output channels in a convolutional layer are evaluated during inference, i.e. ac- celeration. The pre-trained model is then fine-tuned on the target training dataset D with the AFD regularization proposed in Section 3.3.
Empirically we found that in residual networks with greater depths, AFS could become notably challenging to train to high accuracies. To mitigate this, for each output channel of a layer we update 8t according to the variance of (x1-)) observed on the target datasct. For each channel if the variance smaller than a threshold 8,, then we set the entry in St to zero for that particular channel. This action replaces the output of hx with n, which a trainable parameter initialized to the mean of h1(2x:1). We compute the mean and variance statistics using Welford (1962)'s 'ss online algorithm which can efficiently compute the statistics single-pass with O(1) storage. our experiments, 6g is set to a value such that 50% of the channel neurons use the predictor function hy
Morcover, we discovered that many of the channel neurons are rarely activated in a AFS- based network. We further proposc to remove the channel neurons that are activated with a low frequency. In each layer I, the mask m; is used to disable certain channels from the network by setting their output to constant O, if the probability of a channel neuron being active is lower than 5m- Zeroed-out channels can thus be permanently removed when the model is used in inference.
4 EXPERIMENTS
In this section we provide an extensive empirical study of the joint methods of transfer learning and channel pruning. We evaluate the methods with 6 different benchmark datasets: Callech-256 (Griffin el al., 2007) of 256 general object categories; Stanford Dogs 120 (Khosla el. al., 2011) specializes to images containing dogs; MIT Indoors 67 (Qualtoni & Torralba, 2009) for indoor scene classification; Callech- UCSD Birds-200-2011 (CUB-200-22 et al., 2011) for classifying birds; and Food-101 (Bossard et al., 2014) for food categories. We refer to Li et al. (2018) and Li et al. (2019), for detailed description of the benchmark datasets. For Caltech-256, we randomly sample either 30 or 60 images from the training set for each category to produce Caltech-256- and -60 training datascts.
(Wah
We use the ResNet- 101 from torchvision pre-trained on ImageNet as the network for exper- iments. For ResNet-101 equipped with AFS, we start by exlending the pre-trained model and replacing each batch normalization with a randomly initialized AFS, and fine-tune the resulting model on ImageNet for 90 epochs with learning rate of 0.01 decaying by factor of 10 every 30 epochs. The resulting model matches its original baseline accuracy.
For each benchmark dataset, the final FC layer of the network is replaced with new FC randomly initialized with He et al. (2015)'s method to match the number of output categories accordingly. We then perform transfer learning with different methods: La (fine-tuning without additional regularization), L (Li et al., 2018), learning without forgetting (IwF) (T. & Hoiem, 2018), and finally AFD for models using AFS.
To accelcrate the resulting fine-tuned models, we continue fine-tuning the model while gradu- ally pruning away channels used during inference. For this, we scparately cxamine 3 pruning strategies: network slimming (NS) (Liu et al., 2017), soft filter pruning (SFP) (He et al., 2018) and finally AFS for models transfer learned with AFD. Note that NS prunes channels by sorting them globally, while SEP does so in layer-wise manner with identical prune
http::///yyoocccooo////////////s htmm

---

Published as a conference paper at ICLR 2020
ratios. During this procedure, start with an unpruned model and incrementally remove 10% of the channels used in inference, i. preserving 90%, 80%, and etc., down to 10% of all channels for the accelerated models. At each step, fine-tune each model using 4500 steps of SGD with a batch size of 48, at a learning rate of 0.01, before fine-tuning for a further 4500 steps at learning rate of 0.001. AFS additionally updates the m and s masks between the two fine-tuming runs.
Top-1 accuracy (%) comparisons of NS, SFP and AFDS on 6 datasets fine-tuned with their respective best transfer learning methods under various speed-up up constraints.
Table 1:
MAC's reduction
NS
SFP
AFDS
2x
81.83
0.35
79.43
1 0.50
82.05 1 0.43
MIT Indoors 67
5x
69.38
0.27
60.43
+ 0.31
69.93 +\2 0.52
10x
0.30 1.50
58.49
= 0.34
66.72 + 0.53
2x
87.21
0.58
81.74 + 0.26
87.41 + 0.56
Stanford Dogs 120
5x
73.44 = 0.27
61.20 + 0.31
75.14 + 0.52
10x
1.33 + 0.50
59.63 + 0.23
70.70 = 0.33
2x
85.87
0.38
77.26 + 0.28
85,15 + 0.75
Saltech-25--00
5x
66.57 + 0.23
64.27 + 0.31
66.64 + 0.32
10x
0.39 1 0.04
57.11 1 0.54
61.45 1 0.43
2x
88.02 = 0.45
84.59 + 0.28
87.15 = 0.75
Caltech-256-60
5x
73.95 0.27
08.38 + 0.59
74.46 = 0.52
10x
5.05 0.11
61.27 1 0.49
70.16 1 0.53
2x
78.88 = 0.65
75.65 + 0.26
78.03 = 0.45
CUB-200-2011
5x
73.44 - 0.27
61.50 1 0.31
73.35 1 0.52
10x
0.52 0.50
57.88 + 0.23
69.07 1 0.43
2x
83.78
0.61
75.65 + 0.26
84.21 + 0.65
Food-101
5x
73.36
0.45
17.10 1 0.17
79.12 1 0.52
10x
0.99 + 0.04
3.85 1 0.09
76.95 +\2 0.49
Table 2: Top-1 accuracy (%) comparisons of I22, L²-SP, LWF, AFDS on datasets fine-tuned with their respective best pruning methods under various speed-up constraints.
MACs reduction
I2-SP
LwF
AFDS
79.13
1 0.16
78.09
0.56
81.83
0.35
82.05
0.43
MIT Indoors 67
5x
64.22
1 0.21
62.00
0.31
69.38
0.27
69.93
0.52
10x
58.04
+ 0.38
58.49 + 0.34
48.09 0.52
66.72
0.53
2x
85.38 1 0.67
87.21
0.58
87.07
0.35
87.41
0.56
Stanford Dogs 120
5x
70.20 + 0.37
67.10 + 0.31
73.44 0.27
75.14
0.52
10x
59.63 + 0.23
42.89 + 0.48
0.50 17.79
70.70
0.33
2x
83.83 1: 0.62
83.67 1: 0.53
85.87 1 0.38
85.15 1 0.75
Caltech-256-80 250-300
5x
61.45 + 0.17
60.03 + 0.21
66.57
0.23
66.64 - 0.32
10x
57.11 1: 0.34
56.12 1: 0.31
40.32 0.34
61.45 1 0.43
2x
86.27 + 0.47
85.84 + 0.51
88.02 + 0.45
87.15 + 0.75
Caltech-256-00
5x
71.02 + 0.37
69.9 4: 0.31
73.95 0.27
74.46 = 0.52
10x
61.27 1 0.49
0.71 39.41
26.75 0.50
70.16 0.53
2x
76.27 + 0.37
75.58
0.46
78.88 + 0.65
78.03
0.45
CUB-200-2011
66.48 1 0.37
0.31 64.49
73.44
0.27
73.35
0.52
10x
57.88 1 0.23
57.13 1 0.38
29.57
0.31
69.07
0.43
83.78 + 0.61
82.27
0.23
82.38
0.85
84.21
0.65
Food-101
5x
73.36 1 0.33
70.12 0.71
73.05
0.64
79.12
0.52
10x
1.6 + 0.04
3.56 + 0.08
3.85 0.09
76.95
0.49
For each pruned model, we can compute the number of multiphy-accumuula operations (MACs) required to perform inference on an image. For each accelcrated convolution, the required number MACs is H2HWCin and output channels that are not pruned, respectively. We compute the total number of MACs by summing up the MACs in all convolutions, residual connections, and the final pooling and FC layers. For AFS we dynamically select which channels to evaluate during inference, we additionally add the overhead of the importance predictor layers to the number of total MACs.
where Cnn and Cout are the number input
8

---

Published as a conference paper at ICLR 2020
Table 3: Comparison related transfer learning methods.
Dataset
Method
Model
Accuracy
MACs
ResNet-34
73.5
3.6G
Zagoruyko & Komodakis (2017)
ResNet-18
73.0
1.8G
CUB-200-2011
Jang et al. (2019)
ResNet-18
65.05
1.8G
ResNet-101
76.34
2.4G
AFDS
ResNet-101
73.35
1.9
Zagoruyko & Komodakis (2017)
ResNet-34
74.0
3.6G
ResNet-18
72.9
1.8G
MIT Trdnnor 67
Jang et al. (2019)
ResNet-18
64.85
1.8G
ResNet-101
78.00
2.4C
AFDS
ResNet-101
74.57
1.9
80
80
60
Accuracy
NS-
Accuracy
NS+
SFP+LWF
SFP+LWW
NS+L2
NS+L2
SFP4L2
SFP+L2
20
NS+L2-SP
NS+L2-SP
20
SFP+L2-SP
SFP+L2-SP
2
7
3
6
8
MACS
103
MACs
103
(a) Stanford Dogs 120.
(b) Caltech-256-60.
Figure MACs and accuracy (%) trade-off comparisons among different joint methods.
In Figure 3, we present the trade-off relationship between the number of es. the target dataset accuracies for Stanford Dogs and Caltech-256-60. It clear that AFDS (ours) exceeds various combinations of pruning methods (NS, SFP) and transfer learning methods (L? LP-SP, LwF). The results for the remaining datasets can be found in Appendix B. The trade-off curves show that AFDS minimizes accuracy degradation even if 47% of the lotal MACs are removed from the original model, AFDS resulted in only 1.83% drop in accuracy for the model trained on Stanford Dogs. In extreme cases where we permit only 1 of the original computations, our method can still manage 70.70% accuracy, which is substantially better when compared to other pruning algorithms: NS drops to 1.33% and SFP only has 59.63%.
TO
Table provide nunerical comparisons of different pruning methods against AFS under varions speed-up constraints. Table similarly compares transfer learning strategies against AFD. Under most acceleration requirements, the combined method, AFDS, achieves the best accuracies on the target datasets. Finally, Table 3 compares AFDS against other literatures that performs transfer learning. AFDS achieve state-of-the-art. accuracies when compared to methods that produce models with similar number of MACs.
CONCLUSION
In this paper, we introduced attentive feature distillation and selection (AFDS), a dual- attention method that aims to reap the advantages of transfer learning and channel pruning methods. By applying AFDS during fine-tuning, we can not only learn a new model with a higher target task accuracy, but also further accelerates it by computing subset of channel neurons in each convolutional layers. Under a wide range of datasets, we demonstrated the smallest drop in validation accuracies under the same speed-up constraints when compared to traditional compression methods such as network slimming (Liu et al., 2017) and soft filter pruning (He et al., 2018).

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGEMENTS
This work is supported in part by National Key R&D Program of China (No. 2019YFB2102100), Science and Technology Development Fund of Macao S.A.R (FDCT) under number 0015 (15/2019/AN/ AKP,/AKP, Shenzhen Discipline Construction Project for Urban Com- puting and Data Intelligence, the National Natural Science Foundation of China (Nos. 61806192, 61 61802387), Shenzhen Science and Technology Innovation Commission (No. JCYJ2017081 85351877 8789, (T1J009001211000377 ogy Plan Guangdong- Hong Kong Cooperation Innovation Platform (No. 20188050502009) and China's Post-doctoral Science Fund (No. 2019M663183).
the Guangdong Science and Technol-
CEFERENCES
Jose M Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. In D. D. Loe, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (cds.), Advances in Neural Information Processing Systems (NIPS), pp. 2270-2278. 2016.
Hossein Azizpour, Ali Sharif Razavian, Josephine Sullivan, Atsuto Maki, and Stefan Carls- son. From generic to specific deep reprosentations for visual recognition. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW '15, pp. 36 45, 2015.
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 527-536, 2017.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative components with random forests. In European Conference Computer Vision, 2014.
Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: Delving into high quality object de- tection. In Proceedings of the IEFE conference computer vision and pattern recognition, pp. 6154 6162, 2018.
Yves Chauvin. A back- broo-propog algorithm with optimal use of hidden units. In D. S. Touretzky (ed.), Advances in Neural Information Processing Systems 1, pp. 519-526. Morgan-Kaufmann 1989.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hicrarchical image database. In 2009 IEEE Conference on Computer Vision and Pallern Recognilion, pp. 248 255, June 2009.
Jeff Donahue, Yangging Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. DeCAF: A deep convolutional activation feature for generic visual recog- nition. In Eric P. Xing and Tony Jebara (eds.), Proceedings of the 31st International Con- ference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 647-655, Bejing, China, 22-24 Jun 2014. PML.R
Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 4857 4867. Curran Associates, Inc., 2017a.
Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan. More is less: A more complicated network with less inference complexity. In The IEEE Conference on Computer Vision and Pallern Recogrilion (CVPR), July 2017b.
Michael Figurnov, Maxwell D. Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual net- works. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
10

---

Published as a conference paper at ICLR 2020
Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng-zhong Xu. Dy- namic channel pruning: Feature boosting and suppression. In International Conference Learning Representations (ICLR), 2019.
Gregory Griffin, Alex Holub, and Pictro Perona. Caltech-255 object category dataset. Tech- nical report, 2007.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient DNNs. In Advances in Neural Information Processing Systems (NIPS), 2016.
Babak Hassibi, David G. Stork, and Gregory Wolff. Optimal brain surgeon: Extensions and performance comparisons. In J. D. Cowan, G. Tesauro, and J. Alspector (eds.), Advances in Neural Information Processing Systems (NIPS), pp. 263-270. 1994.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proceedings of the 2015 IEEE International Conference Computer Vision (ICCV), ICCV '15, pp. 1026- 1034, Washington, DC, USA, 2015. IEEE Computer Society. ISBN 978-1-4673-8391-2 doi: 10. 10.1109/ICCV 20155.123.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual lcarning for image rocognition. In Proceedings of the IEEE conference on computer vision and pottern recognitiin,, pp. 770 778, 2016.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In International Joint Conference on Artificial Intellignnce (IJCAI), pp. 2234 2240, 2018.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. Advances in neural information, processing systems 2014, Deep Learning Workshop, 2014.
Sergey Tofle and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning Volume 37, ICMT/15, pp. 448 448-456. JMLR.org, 2015. URL attp://dl. acm. org/ pong/iitaiion..m cfm?id=3045118 3045167.
Yunhun Jang, Hankook Lee, Sung Ju Hwang, and Jinwoo Shin. Learning what and where to transfer. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3030 3039, Long Beach, California, USA, 09 15 Jun 2019, PMLR.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- -reegra image categorization. In First Workshop on Fine-Grained Grained Visual Cat- egorization, IEFE Conference Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems (NIPS), pp. 198-605, 1990.
Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, and Jun Huan. DELTA: Deep learning transfer using feature map with attention for convolutional networks. In International Conference on Learning Representations (ICLR), 2019.
Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. Thirty-fifth International Conference on Machine Learning, 2018.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions Pottern Analysis and Machine Intelligence, 29(12):2935-22) -947, Dec 2018. ISSN 0162-8828. doi: 10.1 11110//TPA...01.222277
11

---

Published as a conference paper at ICLR 2020
Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In Advances in Neural Information Processing Systems (NIPS), pp. 2181-2191, 2017.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In International Conference on Computer Vision (ICCV), 2017.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. ThiNet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference computer vision, pp. 5058 5066, 2017.
Michael C Mozer and Paul Smolensky, Skeletonization: A technique for trimming the fat from a network via relevance assessment. In D. S. Touretzky (ed.), Advances Neural Information Processing Systems I, pp. 107 115. Morgan-Kaufmann, 1989.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345 1359, 2009.
Ariadna Quattoni and Antonio Torralba. Rocognizing indoor scenes. In 2009 IEEE Con- ference on Compuler Vision and Pallern Recognition, pp. 413 420, June 2009. doi: 10. LL109//PP.2000
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. CNN features off-the-shelf: An astounding baseline for recognition. In Proceedings of the 2014 TEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW '14, pp. 512 519, Washington, DC, USA, 2014. IEEE Computer Society, ISBN 978-1-4799- 4308-1. doi: 10.1109/CVPRN 22144.11...
R. Reed. Pruning algorithms a survey, IEEE Transactions on Neural Networks, 4(5):740 747, Sep. 1993. doi: 10. 10.1109/72.248422
Mengye Ren, Andrei Pokrovsky, Bin Yang. and Raquel Urtasun. SBNet: Sparse blocks network for fast inference. In The IEEE Conference Computer Vision and Pottern Recogrition (CVPR), June 2018
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real- time object detection with region proposal networks. In Advances in neural informalion processing systems, pp. 91 99, 2015.
Noam Shazeer, Azalia Mirhoseini, Kraysztol Maziarz, Andy Davis, Quoc Le, Geolfrey Hin- lon, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of- experts layer, 2017.
Mingxing Tan and Quoe Le. EfficcenttNt:: Rethinking model scaling for convolutional neural networks. In International Conference Machine Learning, pp. 6105 6114, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds- 200-2011 Datasct. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
B. Welford. Note on mothod for calculating corrected sums of squares and prod- ucts. Technometrics, 4(3):419-420, 1962. ISSN 00401706. URL http://www.jstee org/ stable/1266577.
Wei Wen, Chumpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems (NTPS), pp. 2074-2082. 2016.
12

---

Published as a conference paper at ICLR 2020
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S. Davis, Kristen Grauman, and Rogerio Feris. BlockDrop: Dynamic inference paths residual networks. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the smaller-norm-less- informative assumption in channel pruning of convolution layers. 2018.
Junho Yim, Donggyu Joo, Jihoon Bac, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7130 7138, July 2017. doi:
N..1109/CVP..201...17
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (ods.), Aduances in Newral Information Processing Systems 27, pp. 0820-3328. Curran Associates, Inc., 2014.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In International Conference on Learning Representations (ICLR), 2017.
Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia. ICNet for real- time semantic segmentation on high-resolution images. In Proceedings of the Europeon Conference on Computer Vision (ECCV), pp. 405-420, 2018.
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Jun- xhou Huang, and Jinhui Zhu. Discrinination-wware channel pruning for deep neural networks. In Advances in Neural Information Processing Systems, pp. 875-886, 2018.
A THE OVERALL TRAINING ALGORITHM
In Algorithm 1 we illustrate the complete training procedure described above. Here, the function takes as imput the target training dataset D, the source model f and its param- eters 0-, the total number of steps to fine-tune S, the initial learning rate a, and the threshold hyperparameters & and 6m respectively for S/ and m, The function returns the optimized parameters 0 for the target dataset, and both constant masks for all layers .,,
($1, ,,,, St) and m (m) m2.... my). The function SGD then fine-tunes the model paramcters. For each layer I, we compute the mean bi and variance statistics of q 11)-1), and use to compute Sj.
Algorithm 1 Training Procedure
1: function AFDS(D, O" 0"
S,a, 6,,
2: for E L: <
3: for le L my +-
4:
+ H-SGDDDD,f. f. m, [ a
5: for
I E T. do
6:
14 + E( (2)-1)]
7:
of + Ejo (m (0) )
8:
pt -- Ey T (,((1-1) 1)= > 0]
9:
8: + o² > dg
10:
Y + μi
11:
+ p: 5m
end for
13:
0 - SGD(D. f. m.
[ 1. R)
return 0,s, m
15: end function
13

---

Published as a conference paper at ICLR 2020
B ADDITIONAL RESULTS
80
80
70
60
50
NS+
Accuracy
NS+LWF
40
SFP+LWF
40
SFP+LWF
NS+L2
NS+L2
30
SFP+L2
SFP+L2
20
NS+L2-SP
20
NS+L2-SP
10
SFP+L2-SP
SFP+L2-SP
ours
ours
0
MACS
MACS
(a) MTT Indoors 67.
(b) Food-101.
80
70
60
60
Accuracy
50
NS
Accuracy
NS+LWF
40
SFP+LWF
SFP+LWF
NS+L2
NS+L2
SFP+L2
SFP+L2
NS+L2-SP
20
NS+L2-SP
SFP+L2-SP
SFP+
ours
ours
MACS
MACS
les
(c) Caltech-UCSD Birds-200-200
(d) Caltech-256-30.
Figure 4: MACs and accuracy (%) trade-off comparisons among different joint methods.
14


Reviews:
[{"review_id": "Ske1UQAX9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": " This paper proposes a method called attentive feature distillation and selection (AFDS) to improve the performance of transfer learning for CNNs. The authors argue that the regularization should constrain the proximity of feature maps, instead of pre-trained model weights. Specifically, the authors proposes two modifications of loss functions: 1) Attentive feature distillation (AFD), which modifies the regularization term to learn different weights for each channel and 2) Attentive feature selection (AFS), which modifies the ConvBN layers by predicts unimportant channels and suppress them. \n\nOverall, this is a good work in terms of theory and experimentation, thus I would recommend to accept it. The approach is well motivated, and the literature is complete and relevant. The author's argument is validated by experiments comparing the proposed AFDS method and other existing transfer learning methods. \n\nTo improve this paper, the authors are suggested to address the following issues:\n1.  Section 3.5 is not well organized. Besides, it is not mentioned what value the threshold hyper-parameter delta_m is set. \n2. In page 9, \"MACs\" is missing in the sentence \"In Figure 3, ...the number of vs. the target...\"\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BklJwjhd5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an improvement to the task of transfer learning by being deliberate about which channels from the base model are most relevant to the new task at hand. It does this by apply attentive feature selection (AFS) to select channels or features that align well with the down stream task and attentive feature distillation (AFD) to pass on these features to the student network. In the process they do channel pruning there by decreasing the size of the network and enabling faster inference speeds. Their major argument is that plain transfer learning is redundant and wasteful and careful attention applied to selection of the features and channels to be transfered can lead to smaller faster models which in several cases presented in the paper provide superior performance.\n\nPaper is clear and concise and experimentally sound showing a real contribution to the body of knowledge in transfer learning and pruning.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BkeROrlU5B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In general, I think it is a good paper and I like the contribution of the author. I think they explain in detail the methodology. The results compare the new methodologies with different databases which increase the credibility of the results. However, there is a couple of additional question that is important to manage:  \n\n1) The paper presents three different contributions. However, it is so clear how this work helps for \"By changing the fraction of channel neurons to skip for each convolution, AFDS can further accelerate the transfer learned models while minimizing the impact on task accuracy\" I think a better explanation of this part it would be necessary. \n\n2) The comparison of the results are very focused on AFDS, Did you compare the results with different transfer learning approach? \n\n3) During the training procedure. We need a better explanation of why \"we found that in residual networks with greater depths, AFS could become notably challenging to train to high accuracies\". Also, the results of the empirical test it would be useful to understand the challenges to train the network. \n\n4) I think it would be useful to have the code available for the final version. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an attention-based approach to transfer faster CNNs, which tackles the problem of jointly transferring source knowledge and pruning target CNNs.

Reviewers are unanimously positive on the paper, in terms of a well-written paper with a reasonable approach that yields strong empirical performance under the resource constraint.

AC feels that the paper studies an important problem of making transfer learning faster for CNNs, however, the proposed model is a relatively straightforward combination of fine-tuning and filter-pruning, each having very extensive prior works. Also, AC has very critical comments for improving this paper:

- The Attentive Feature Distillation (AFD) module is very similar to DELTA (Li et al. ICLR 2019) and L2T (Jang et al. ICML 2019), significantly weakening the novelty. The empirical evaluation should consider DELTA as baselines, e.g. AFS+DELTA.

I accept this paper, assuming that all comments will be well addressed in the revision.
Decision: Accept (Poster)