Adaptive Feature Ranking for Unsupervised Transfer Learning
Authors: Son N. Tran, Artur d'Avila Garcez
ICLR 2014

Abstract: Transfer Learning is concerned with the application of knowledge gained from solving a problem to a different but related problem domain. In this paper, we propose a method and efficient algorithm for ranking and selecting representations from a Restricted Boltzmann Machine trained on a source domain to be transferred onto a target domain. Experiments carried out using the MNIST, ICDAR and TiCC image datasets show that the proposed adaptive feature ranking and transfer learning method offers statistically significant improvements on the training of RBMs. Our method is general in that the knowledge chosen by the ranking function does not depend on its relation to any specific target domain, and it works with unsupervised learning and knowledge-based transfer.


Peer review:
[Review]:
We thank the reviewer for the comments, and we have carried out further experiments as suggested, as follows: we’ve updated Figures 4 and 5 to include a comparison between pruning of low-scoring units and pruning of high-scoring units.
 
The results seem to confirm that high-scored units capture more relevant information. Furthermore, Figure 7 indicates that high-scored units are more significant for transfer learning. Therefore, the role of high-scored units seems relevant for transfer learning. 

For transfer low-scored features, we performed experiments follows what we have done with transferring high-scored features (we use the validation sets to select the features for transfer, but in this case the features are ranked from low-scores to high-scores). The results show that the highest accuracies were achieved only when a large number of high-scored features are among those which have been transferred. 
MNIST_30k: ICDAR_d  : 38.86%  ± 0.32 (when transfer 2000/2000 units)
MNIST_30k: TiCC_w_A : 76.99%  ± 0.58 (when transfer 2000/2000 units)
MNIST_5k : TiCC_a   : 62.50%  ± 0.16 (when transfer  500/500 units)
MNIST_5k : TiCC_d   : 65.19%  ± 0.15 (when transfer 500/500 units)

To the best of our knowledge, most connectionist transfer learning use labels in the source domain or self-taught learning [1 2 3 4]. For the task of unsupervised representation transfer learning, as proposed in this paper, we have therefore chosen to focus on comparisons with our closest competitor: self-taught learning [1]. In addition, we propose to study how much knowledge should be transferred to the target domain, which has not been studied yet in self-taught mode.
 
[1] Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, and Andrew Y. Ng. Self-taught learning: transfer learning from unlabeled data. In Proceedings of the 24th international conference on Machine learning, ICML ’07, page 759766, New York, NY, USA, 2007. ACM
[2] Grégoire Mesnil, Yann Dauphin, Xavier Glorot, Salah Rifai, Yoshua Bengio, Ian J. Goodfellow, Erick Lavoie, Xavier Muller, Guillaume Desjardins, David Warde-Farley, Pascal Vincent, Aaron C. Courville, James Bergstra: Unsupervised and Transfer Learning Challenge: a Deep Learning Approach. ICML Unsupervised and Transfer Learning 2012: 97-110
[3] WEI, B.; PAL, C.. Heterogeneous Transfer Learning with RBMs. AAAI Conference on Artificial Intelligence, North America, aug. 2011
[4] Honglak Lee, Chaitanya Ekanadham, and Andrew Y. Ng. Sparse deep belief net model for visual area v2. In Advances in Neural Information Processing Systems. MIT Press, 2008.