Adaptive Feature Ranking for Unsupervised Transfer Learning
Authors: Son N. Tran, Artur d'Avila Garcez
ICLR 2014

Abstract: Transfer Learning is concerned with the application of knowledge gained from solving a problem to a different but related problem domain. In this paper, we propose a method and efficient algorithm for ranking and selecting representations from a Restricted Boltzmann Machine trained on a source domain to be transferred onto a target domain. Experiments carried out using the MNIST, ICDAR and TiCC image datasets show that the proposed adaptive feature ranking and transfer learning method offers statistically significant improvements on the training of RBMs. Our method is general in that the knowledge chosen by the ranking function does not depend on its relation to any specific target domain, and it works with unsupervised learning and knowledge-based transfer.


Peer review:
[Review]:
The submission proposes a scoring method for RBMs to rank hidden units according to their information content, and to use that ranking to prune networks and to do transfer learning. The transfer learning mechanism uses the ranking in the original domain to choose which hidden units to use when training in the target domain, and then uses the output of those transferred units to affect the training of the added units.   

The submission is clearly written, has relevance and interest to the ICLR community, and a number of experiments are described that validate the method. The results are promising, however there are a number of missing experiments that would have helped to validate the method. An important baseline experiment would be to compare the pruning of low-scoring units with the pruning of randomly selected, or even high-scoring units. Similarly, with the transfer learning experiments, it would be important to verify that the author's metric for replacement of sub-networks is optimal, or at least better than random choice. The submission also suffers from not making comparisons to other approaches to transfer learning in neural networks. 
 
pros: well-written, relevant and interesting
cons: simple approach, not adequately validated, not adequately compared against other transfer learning approaches.