Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper addresses the problem of transfer learning in deep networks. A pretrained network on a large dataset exists, what is the best way to retrain the model on a new small dataset? 
It argues that the standard regularization done in conventional fine-tuning procedures is not optimal, since it tries to get the parameters close to zero, thereby forgetting the information learnt on the larger dataset.
It proposes to have a regularization term that penalizes divergence from initialization (pretrained network) as opposed to from zero-vector. It tries different norms (L2, L1, group Lasso) as well as Fisher information matrix to avoid interfering with important nodes, and shows the effectiveness of these alternatives over the standard practice of “weight decay”.

Although the novelty of the paper is limited and have been shown for transfer learning with SVM classifiers prior to resurgence of deep learning, the reviewer is unable to find a prior work doing same regularization in deep networks. Number of datasets and experiments are moderately high, results are consistently better than standard fine-tuning and fine-tuning is a very common tool for ML practitioners in various application fields, so, I think there is benefit for transfer learning audience to be exposed to the experiments of this paper.

---- Post rebuttal
I think the paper has merits to be published. As I note above, it's taking a similar idea with transfer learning of SVM models from a decade ago to deep learning and fine-tuning. It's simple with no technical novelty but shows consistent improvement and has wide relevance. 

review_title: limited novelty, but consistently improving fine-tuning 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work addresses the scenario of fine-tuning a pre-trained network for new data/tasks and empirically studies various regularization techniques. Overall, the evaluation concludes with recommending that all layers of a network whose weights are directly transferred during fine-tuning should be regularized against the initial net with an L2 penalty during further training. 

Relationship to prior work:
Regularizing a target model against a source model is not a new idea. The authors miss key connections to A-SVM [1] and PMT-SVM [2] -- two proposed transfer learning models applied to SVM weights, but otherwise very much the same as the proposed solution in this paper. Though the study here may offer new insights for deep nets, it is critical to mention prior work which also does analysis of these regularization techniques. 

Significance:
As the majority of visual recognition problems are currently solved using variants of fine-tuning, if the findings reported in this paper generalize, then it could present a simple new regularization which improves the training of new models. The change is both conceptually simple and easy to implement so could be quickly integrated by many people.

Clarity and Questions:
The purpose of the paper is clear, however, some questions remain unanswered. 
1) How is the regularization weight of 0.01 chosen? This is likely a critical parameter. In an experimental paper, I would expect to see a plot of performance for at least one experiment as this regularization weighting parameter is varied. 
2) How does the use of L2 regularization on the last layer effect the regularization choice of other layers? What happens if you use no regularization on the last layer? L1 regularization?
3) Figure 1 is difficult to read. Please at least label the test sets on each sub-graph.
4) There seems to be some issue with the freezing experiment in Figure 2. Why does performance of L2 regularization improve as you freeze more and more layers, but is outperformed by un-freezing all. 
5) Figure 3 and the discussion of linear dependence with the original model in general seems does not add much to the paper. It is clear that regularizing against the source model weights instead of 0 should result in final weights that are more similar to the initial source weights. I would rather the authors use this space to provide a deeper analysis of why this property should help performance. 
6) Initializing with a source model offers a strong starting point so full from scratch learning isn’t necessary -- meaning fewer examples are needed for the continued learning (fine-tuning) phase. In a similar line of reasoning, does regularizing against the source further reduce the number of labeled points needed for fine-tuning? Can you recover L2 fine-tuning performance with fewer examples when you use L2-SP?

[1] J. Yang, R. Yan, and A. Hauptmann. Adapting svm classifiers to data with shifted distributions. In ICDM Workshops, 2007.
[2]  Y. Aytar and A. Zisserman. Tabula rasa: Model transfer for object category detection. In Proc. ICCV, 2011.

------------------
Post rebuttal
------------------
The changes made to the paper draft as well as the answers to the questions posed above have convinced me to upgrade my recommendation to a weak accept. The experiments are now clear and thorough enough to provide a convincing argument for using this regularization in deep nets. Since it is simple and well validated it should be easily adopted. 

review_title: A reasonably thorough study of regularization techniques for transfer learning through fine-tuning

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes an analysis on different adaptive regularization techniques for deep transfer learning. 
Specifically it focuses on the use of an L2-SP condition that constraints the new parameters to be close to the
ones previously learned when solving a source task. 

+ The paper is easy to read and well organized
+ The advantage of the proposed regularization against the more standard L2 regularization is clearly visible 
from the experiments

- The idea per se is not new: there is a list of shallow learning methods for transfer learning based 
on the same L2 regularization choice
[Cross-Domain Video Concept Detection using Adaptive SVMs, ACM Multimedia 2007]
[Learning categories from few examples with multi model knowledge transfer, PAMI 2014]
[From n to n+ 1: Multiclass transfer incremental learning, CVPR 2013]
I believe this literature should be discussed in the related work section

- It is true that the L2-SP-Fisher regularization was designed for life-long learning cases with a 
fixed task, however, this solution seems to work quite well in the proposed experimental settings. 
From my understanding L2-SP-Fisher can be considered the best competitor of L2-SP so I think
the paper should dedicate more space to the analysis of their difference and similarities both
from the theoretical and experimental point of view. For instance:
--  adding the L2-SP-Fisher results in table 2
--  repeating the experiments of figure 2 and figure 3 with L2-SP-Fisher





review_title: well written, needs more comparisons/analysis

Meta-review: This paper addresses the question of how to regularize when starting from a pre-trained convolutional network in the context of transfer learning.  The authors propose to regularize toward the parameters of the pre-trained model and study multiple regularizers of this type.  The experiments are thorough and convincing enough.  This regularizer has been used quite a bit for shallow models (e.g. SVMs as the authors mention, but also e.g. more general MaxEnt models).  There is at least some work on regularization toward a pre-trained model also in the context of domain adaptation with deep neural networks (e.g. for speaker adaptation in speech recognition).  The only remaining novelty is the transfer learning context.  This is not a sufficiently different setting to merit a new paper on the topic.
Decision: Reject