Learning Non-Linear Feature Maps, With An Application To Representation Learning
Authors: Dimitrios Athanasakis, John Shawe-Taylor, Delmiro Fernandez-Reyes
ICLR 2014

Abstract: Recent non-linear feature selection approaches employing greedy optimisation of Centred Kernel Target Alignment(KTA) exhibit strong results in terms of generalisation accuracy and sparsity. However, they are computationally prohibitive for large datasets. We propose randSel, a randomised feature selection algorithm, with attractive scaling properties. Our theoretical analysis of randSel provides strong probabilistic guarantees for correct identification of relevant features. RandSel's characteristics make it an ideal candidate for identifying informative learned representations. We've conducted experimentation to establish the performance of this approach, and present encouraging results, including a 3rd position result in the recent ICML black box learning challenge as well as competitive results for signal peptide prediction, an important problem in bioinformatics.


Peer review:
[Review]:
I fully confess in advance that I am not a very good reviewer for this paper. I have no real research experience with kernel learning or feature selection. As such, I’ll be reporting my scores with low confidence. I haven’t made any attempt to evaluate the sensibility or novelty of the proposed method itself or the related proofs. If none of the other reviewers is able to provide a more confident review I can take some time to study the literature and improve my review.

The main thing I do feel able to review somewhat confidently is the empirical results. It’s nice that the authors were able to get good accuracy on the Black Box Learning challenge. It looks like that was a fairly competitive challenge, with over 200 competitors, and they took 3rd place. It’s also nice that they were able to improve over SignalP’s state of the art result on the cleavage site prediction task.

One thing I’m a bit concerned about is whether the proposed method improves over the state of the art for feature selection methods. As the authors state, it is somewhat difficult to compare performance on the cleavage site prediction task. Beyond these issues, it’s also not clear to me that randSel improves over state of the art feature selection methods. It looks like it does improve over RFE, but it would be nice if there was a baseline run by other authors. It’s not clear to me that RFE is a state of the art method to beat though. It would be nice to have more explanation of the significance of beating SignalP.

Detailed comments:

Title: I consider 'Representation Learning' to be a superset of 'Learning Non-Linear Feature Maps'. So it doesn't make much sense for the former to be an application of the other. Also both are pretty generic. Pretty much any feature learning method except PCA learns non-linear feature maps. Seems like you really could come up with a more informative title.
I'd argue your title should have the phrase 'Feature Selection' in it. And probably not 'Feature Learning.'

Abstract: 
missing a space before the paren (this actually happens throughout the paper, not just in the abstract)

Fig 1: You might want to put all the different rows on the same scale so that it’s possible to visually compare the height of the bars across rows. 

Page 4: “So far have”: there is a word missing here

Page 6: “The original data where projected” -> “were projected”
	“The organizers did not reveal the source of the dataset”: this sounds like the organizers did not reveal the source of the data was SVHN. You might want to specify that you didn’t know it was created by multiplication by a random projection matrix either.
	“where provided” -> “were provided”
	“ranking third in both cases”: 
		Here’s the public leaderboard: http://www.kaggle.com/c/challenges-in-representation-learning-the-black-box-learning-challenge/leaderboard/public
		According to this, Bing Xu is 3rd. But according to the report put out by the organizers ( http://arxiv.org/pdf/1307.0414.pdf ) you guys actually were 3rd and it sounds like Bing Xu was ranked worse than 3rd originally. Did you delete your kaggle account and get taken off the leaderboard or something?


Supplementary Material T:
The box around 'OVER-COMPLETE LEARNED REPRESENTATION' is too tight