Continuous Learning: Engineering Super Features With Feature Algebras
Authors: Michael Tetelman
ICLR 2014

Abstract: In this paper we consider a problem of searching a space of predictive models for a given training data set. We propose an iterative procedure for deriving a sequence of improving models and a corresponding sequence of sets of non-linear features on original input space. After finite number of iterations $N$ the non-linear features become $2^N$-degree polynomials on original space. We show that in a limit of infinite number of iterations derived non-linear features must form an algebra, so for any given input point a product of two features is a linear combination of features from same feature space. Due to convexity of each iteration and its ability to fall back to solutions found in previous iteration the models in the sequence have always increasing likelihood with each iteration while dimensionality of each model parameter space is set to a limited controlled value.


Peer review:
[Review]:
The idea presented in this paper consists in iteratively building new features
from products of so-called 'super features', obtained from a PCA in parameter
space of multiple models trained on a subset of the data. The theoretical
analysis is done with logistic regression, and no experiments are performed.

In its current form, this paper is definitely not ready for publication. The
fact that there is no experiment is already a pretty big downside, and the
theory does not seem complete enough to me to justify it. In particular:
- There is no proof or even intuition for the convergence of the procedure. To
  be fair, there is no claim that it will converge either, but what is the
  point of Section 5 if this is not the case?
- The abstract mentions a guarantee of increasing likelihood which is not shown 
  in the paper.
- There is no analysis of the complexity of the algorithm (it seems costly, 
  especially since each iteration involves sampling multiple datasets).
- There is no discussion of extensions beyond logistic regression.
- There is no justification for using PCA in parameter space (which works best 
  with Gaussian-like unimodal distributions, but may fail in other situations:
  how can we tell it makes sense for a specific application / model?)

That being said, this is an intriguing idea, and I am hoping the author will be able to investigate it more thoroughly so as to be able to present it in a more
fleshed out paper in the future.

There are several points that were unclear to me and may be worth mentioning in 
addition to the above high level comments:
- The equation below eq. 1 does not seem to be used anywhere.
- 'Values of regularization parameters must be found by maximizing Bayesian 
  integral in Equation 1.' => this integral is the likelihood -- optimizing
  regularization parameters to maximize the likelihood seems wrong to me.
- '[eq. 1] is estimated by maximum likelihood approximation': I do not see how 
  MLA is a good approximation here.
- eq. 5 seems arbitrary to me, especially since the datasets are of different
  sizes,  so each L_w_s involves a different number of terms.
- It is not mentioned exactly how the 'sample data' step works (for instance in
  bootstrap one typically samples t_max points with replacement, but it does
  not seem to be the case here).
- I do not understand how G_alpha,beta can be non-zero in eq. 16, and still
  lead to a linear combination.
- Not clear what are func and F in eq. 19.