Correlation-based construction of neighborhood and edge features
Authors: Balázs Kégl
ICLR 2014

Abstract: Motivated by an abstract notion of low-level edge detector filters, we propose a simple method of unsupervised feature construction based on pairwise statistics of features. In the first step, we construct neighborhoods of features by regrouping features that correlate. Then we use these subsets as filters to produce new neighborhood features. Next, we connect neighborhood features that correlate, and construct edge features by subtracting the correlated neighborhood features of each other. To validate the usefulness of the constructed features, we ran AdaBoost.MH on four multi-class classification problems. Our most significant result is a test error of 0.94% on MNIST with an algorithm which is essentially free of any image-specific priors. On CIFAR-10 our method is suboptimal compared to today's best deep learning techniques, nevertheless, we show that the proposed method outperforms not only boosting on the raw pixels, but also boosting on Haar filters.


Peer review:
[Review]:
The paper proposes a method for feature construction/augmentation based on grouping features that correlate, in an iterative/recursive fashion. These feature sets are validated using AdaBoost.MH and a few interesting results on MNIST and CIFAR are presented, as well as a few interesting negative results. The authors claim a relatively state of the art results on MNIST (without using image priors). While the CIFAR results are not as competitive with state of the art, they do improve on the boosting state of the art.

The method is relatively simple: group features that correlate, then connect neighborhood features that correlate and construct edge features by subtracting the correlated neighborhood features. The authors suggest this was inspired by biology/Haar/Gabor filters, but I am not sure this connection is particularly informative.  A hybrid AdaBoost.MH with decision stumps (for picking the features to augment) with AdaBoost.MH + Hamming trees was run.

On MNIST this gives a 0.94% test error, which sounds like state-of-the-art(-ish). Interestingly, on CIFAR-10 this kind of approach beats boosting approaches on top of Harr filter features (though is not very close to state of the art!). For smaller and lower-dimensional datasets, the gains are relatively small -- the authors hypothesize that this is because the dimensionality is too low basically.

Why not run something like a linear SVM on top of the constructed features? I think it would be an interesting result too. Or even just a second-degree polynomial expansion of the feature-set found by the decision stump phase? Would make the results in this paper stronger, in my opinion: right now, it’s unclear if the extra gain is because of the clever selection of which features to combine or simply because features were combined at all.