Continuous Learning: Engineering Super Features With Feature Algebras
Authors: Michael Tetelman
ICLR 2014

Abstract: In this paper we consider a problem of searching a space of predictive models for a given training data set. We propose an iterative procedure for deriving a sequence of improving models and a corresponding sequence of sets of non-linear features on original input space. After finite number of iterations $N$ the non-linear features become $2^N$-degree polynomials on original space. We show that in a limit of infinite number of iterations derived non-linear features must form an algebra, so for any given input point a product of two features is a linear combination of features from same feature space. Due to convexity of each iteration and its ability to fall back to solutions found in previous iteration the models in the sequence have always increasing likelihood with each iteration while dimensionality of each model parameter space is set to a limited controlled value.


Peer review:
[Review]:
> - A brief summary of the paper's contributions, in the context of prior work.

The way we represent data -- the features we use -- has proven essential to a wide variety of tasks in machine learning. Often, the trivial features we naturally get from the data perform poorly. One approach is to hand-engineer features, having humans carefully construct features based on their understanding of the data and the task. Another approach, taken in deep learning, is to learn features as part of the optimization process of a multi-layer model. This paper proposes an alternative approach in which 'super-features' are generated by an iterative exploration process.

As the reviewer understands it, this iterative process begins with random sigmoidial features. Features are evaluated on several subsets of the dataset and good features are selected. Principal component analysis is performed on the parameters of the good features in order to define a lower-dimensional space of good features. The next iteration proceeds on products of the principal components discovered in the previous step. In the limit, the discovered features form a space with nice algebraic properties.

> - An assessment of novelty and quality.

The reviewer is not an expert but to the best of their knowledge this approach is novel.

The reviewer is concerned that this paper may be a bit rushed: it suffers from many grammatical errors, and some parts of the paper are hard to follow. The reviewer thinks that a bit of further revision would greatly benefit this paper.

> - A list of pros and cons (reasons to accept/reject)

Pros:
* The paper introduces some interesting, novel ideas. In particular, attempting to do dimensionality reduction in parameter space seems like a really interesting strategy. (I'm concerned that PCA may not be a very good way to do this, but the idea is very interesting.)

Cons:
* The paper does not do any experiments to test the efficacy of the proposed approach. Given how radically different the proposed approach is from things that have been previously tried, it seems really difficult to have any significant confidence in it without experimental results. (Not having experimental results for this would seem more appropriate if this were a workshop track submission.)
* The paper suffers from many grammatical issues. These really need to be fixed in a final version of this paper.
* The paper is quite hard to follow at some points.

Testing this approach on a standard dataset would also provide an opportunity for the author to walk the reader through a concrete application of this approach, in addition to providing experimental validation of it.