Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper presents an improvement to the task of transfer learning by being deliberate about which channels from the base model are most relevant to the new task at hand. It does this by apply attentive feature selection (AFS) to select channels or features that align well with the down stream task and attentive feature distillation (AFD) to pass on these features to the student network. In the process they do channel pruning there by decreasing the size of the network and enabling faster inference speeds. Their major argument is that plain transfer learning is redundant and wasteful and careful attention applied to selection of the features and channels to be transfered can lead to smaller faster models which in several cases presented in the paper provide superior performance.

Paper is clear and concise and experimentally sound showing a real contribution to the body of knowledge in transfer learning and pruning.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: In general, I think it is a good paper and I like the contribution of the author. I think they explain in detail the methodology. The results compare the new methodologies with different databases which increase the credibility of the results. However, there is a couple of additional question that is important to manage:  

1) The paper presents three different contributions. However, it is so clear how this work helps for "By changing the fraction of channel neurons to skip for each convolution, AFDS can further accelerate the transfer learned models while minimizing the impact on task accuracy" I think a better explanation of this part it would be necessary. 

2) The comparison of the results are very focused on AFDS, Did you compare the results with different transfer learning approach? 

3) During the training procedure. We need a better explanation of why "we found that in residual networks with greater depths, AFS could become notably challenging to train to high accuracies". Also, the results of the empirical test it would be useful to understand the challenges to train the network. 

4) I think it would be useful to have the code available for the final version. 
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content:  This paper proposes a method called attentive feature distillation and selection (AFDS) to improve the performance of transfer learning for CNNs. The authors argue that the regularization should constrain the proximity of feature maps, instead of pre-trained model weights. Specifically, the authors proposes two modifications of loss functions: 1) Attentive feature distillation (AFD), which modifies the regularization term to learn different weights for each channel and 2) Attentive feature selection (AFS), which modifies the ConvBN layers by predicts unimportant channels and suppress them. 

Overall, this is a good work in terms of theory and experimentation, thus I would recommend to accept it. The approach is well motivated, and the literature is complete and relevant. The author's argument is validated by experiments comparing the proposed AFDS method and other existing transfer learning methods. 

To improve this paper, the authors are suggested to address the following issues:
1.  Section 3.5 is not well organized. Besides, it is not mentioned what value the threshold hyper-parameter delta_m is set. 
2. In page 9, "MACs" is missing in the sentence "In Figure 3, ...the number of vs. the target..."

review_title: Official Blind Review #3

Meta-review: This paper presents an attention-based approach to transfer faster CNNs, which tackles the problem of jointly transferring source knowledge and pruning target CNNs.

Reviewers are unanimously positive on the paper, in terms of a well-written paper with a reasonable approach that yields strong empirical performance under the resource constraint.

AC feels that the paper studies an important problem of making transfer learning faster for CNNs, however, the proposed model is a relatively straightforward combination of fine-tuning and filter-pruning, each having very extensive prior works. Also, AC has very critical comments for improving this paper:

- The Attentive Feature Distillation (AFD) module is very similar to DELTA (Li et al. ICLR 2019) and L2T (Jang et al. ICML 2019), significantly weakening the novelty. The empirical evaluation should consider DELTA as baselines, e.g. AFS+DELTA.

I accept this paper, assuming that all comments will be well addressed in the revision.
Decision: Accept (Poster)