Correlation-based construction of neighborhood and edge features
Authors: Balázs Kégl
ICLR 2014

Abstract: Motivated by an abstract notion of low-level edge detector filters, we propose a simple method of unsupervised feature construction based on pairwise statistics of features. In the first step, we construct neighborhoods of features by regrouping features that correlate. Then we use these subsets as filters to produce new neighborhood features. Next, we connect neighborhood features that correlate, and construct edge features by subtracting the correlated neighborhood features of each other. To validate the usefulness of the constructed features, we ran AdaBoost.MH on four multi-class classification problems. Our most significant result is a test error of 0.94% on MNIST with an algorithm which is essentially free of any image-specific priors. On CIFAR-10 our method is suboptimal compared to today's best deep learning techniques, nevertheless, we show that the proposed method outperforms not only boosting on the raw pixels, but also boosting on Haar filters.


Peer review:
[Review]:
'To my opinion, an important contribution is the use of boosting in an auto-associative setting to select a subset of pixels that is used to select the neighborhoods, however, this appears in the paper only as a speedup trick. From an image processing viewpoint, what are these most predictive pixels picked up by Adaboost? This actually a very neat representation learning trick I have not seen elsewhere.'

The selected pixels (and channels) are in Figure 2. 

We have experimented for a while with autoassociative boosting about six years ago to replace autoassociative neural nets, it worked nicely for the first layer but stacking never improved the error, so we dropped the line of research.