Adaptive Feature Ranking for Unsupervised Transfer Learning
Authors: Son N. Tran, Artur d'Avila Garcez
ICLR 2014

Abstract: Transfer Learning is concerned with the application of knowledge gained from solving a problem to a different but related problem domain. In this paper, we propose a method and efficient algorithm for ranking and selecting representations from a Restricted Boltzmann Machine trained on a source domain to be transferred onto a target domain. Experiments carried out using the MNIST, ICDAR and TiCC image datasets show that the proposed adaptive feature ranking and transfer learning method offers statistically significant improvements on the training of RBMs. Our method is general in that the knowledge chosen by the ranking function does not depend on its relation to any specific target domain, and it works with unsupervised learning and knowledge-based transfer.


Peer review:
[Review]:
This paper presents a feature ranking method for RBMs and a method to transfer RBM representations from a source domain to a target domain in an adaptive manner. While the method achieves good performance when transfering between very similar domains, the paper suffers from the following problems:

- The proposed ranking function is equivalent to the L1 norm of the weight vector associated with each hidden unit, which makes the method qualitatively similar to PCA in that it prioritizes the components that explain the most variance in the data (Bengio, JMLR 7 (2011) 1--20). However, it is not obvious from the experiments that the proposed ranking has an advantage for transfer learning. A connection with the existing literature also seems lacking.

- To the best of the reviewer's knowledge, the adaptive learning method for RBMs (with theta=1) is novel, but many very similar approaches to transfer learning using RBMs already exist and, once again, a review and comparison with competing approaches would be crucial.

- Despite the simplicity of the concepts presented, I have found the paper confusing at times. For example, what exact setup does each row correspond to in Tables 3 and 4? What is the difference between the last and second to last rows?

- Finally, the experimental section considers only the task of character recognition.

Other questions:

- In Figure 2(b), would we get a different interpretation if the fitler bases were normalized? Using the L1 norm of the bases for scoring, it is hardly surprising that the low-score bases have a lower magnitude when plotted on the same scale.
- In Figure 3 (sparse RBM), are the bases normalized by the L2 norm?