Correlation-based construction of neighborhood and edge features
Authors: Balázs Kégl
ICLR 2014

Abstract: Motivated by an abstract notion of low-level edge detector filters, we propose a simple method of unsupervised feature construction based on pairwise statistics of features. In the first step, we construct neighborhoods of features by regrouping features that correlate. Then we use these subsets as filters to produce new neighborhood features. Next, we connect neighborhood features that correlate, and construct edge features by subtracting the correlated neighborhood features of each other. To validate the usefulness of the constructed features, we ran AdaBoost.MH on four multi-class classification problems. Our most significant result is a test error of 0.94% on MNIST with an algorithm which is essentially free of any image-specific priors. On CIFAR-10 our method is suboptimal compared to today's best deep learning techniques, nevertheless, we show that the proposed method outperforms not only boosting on the raw pixels, but also boosting on Haar filters.


Peer review:
[Review]:
The paper proposes a new method for constructing features by hand. Features are constructed as follows: First, one averages the input dimensions that are strongly correlated (for images these will be nearby pixels). Then one attaches to the result the differences between subsets of pairs of these features which are still highly correlated.

The beginning of the introduction is just a copy of the abstract. 

I find the section Constructing the representation hard to follow. It would help to use a much more detailed description. Also, under 1. the term 'features' is used to refer to the learned representations as well as to pixels (second page, after 1.), under 2. the same indexes are used on the two sides of the equal sign, and under 3. defining the symbol 'equal with triangle on top' would help.

The construction of the features reminded me of locally binary patterns LBP (unless I am missing something), so it feels like it would be good to compare to these.

While 94 errors on MNIST seems decent, the method contains a huge number of hyperparameters (first level correlation threshold, second level correlation threshold, several boosting parameters, size and choice of random matrix to estimate correlations etc. etc. etc.).

The result of CIFAR-10 seems encouraging. Is this without image priors (that is, permutation invariant)?

As is, I find the paper quite hard to read, and it will be important to improve the clarity of the presentation.