Correlation-based construction of neighborhood and edge features
Authors: Balázs Kégl
ICLR 2014

Abstract: Motivated by an abstract notion of low-level edge detector filters, we propose a simple method of unsupervised feature construction based on pairwise statistics of features. In the first step, we construct neighborhoods of features by regrouping features that correlate. Then we use these subsets as filters to produce new neighborhood features. Next, we connect neighborhood features that correlate, and construct edge features by subtracting the correlated neighborhood features of each other. To validate the usefulness of the constructed features, we ran AdaBoost.MH on four multi-class classification problems. Our most significant result is a test error of 0.94% on MNIST with an algorithm which is essentially free of any image-specific priors. On CIFAR-10 our method is suboptimal compared to today's best deep learning techniques, nevertheless, we show that the proposed method outperforms not only boosting on the raw pixels, but also boosting on Haar filters.


Peer review:
[Review]:
In its present form, the paper proposes a very neat task-agnostic feature transformation trick that brings Adaboost nearly up to the start of the art on image classification tasks, which is quite a feat. The trick seems so far to bring top performance on tasks where there are known feature correlations (like pixels) but not on others.

While it would be a nice workshop contribution  in its present state, I think it requires some work to be a significant contribution to representation learning: some important aspects of the work are not properly explained, and experiments only use one algorithm (Adaboost.MH with hamming trees) and would need to be try  other algorithms (SVMs, first layer of deep learning).

The authors take the traditional local smoothing and edge detection used for image processing, but remove all knowledge that pixels are adjacent. Thus the smoothing can be done on any set of pixels provided they are similar (measured through correlation). The correlation is measured again between smoothed pixels to determine edges. I assume this is original, as in its exhaustive form, one would probably discard the method as computationally not practical (I assume it scales as O(N_example*N_features^2)).
To my opinion, an important contribution is the use of boosting in an auto-associative setting to select a subset of pixels that is used to select the neighborhoods, however, this appears in the paper only as a speedup trick. From an image processing viewpoint, what are these most predictive pixels picked up by Adaboost? This actually a very neat representation learning trick I have not seen elsewhere.