Learning Transformations for Classification Forests
Authors: Qiang Qiu, Guillermo Sapiro
ICLR 2014

Abstract: This work introduces a transformation-based learner model for classification forests. The weak learner at each split node plays a crucial role in a classification tree. We propose to optimize the splitting objective by learning a linear transformation on subspaces using nuclear norm as the optimization criteria. The learned linear transformation restores a low-rank structure for data from the same class, and, at the same time, maximizes the separation between different classes, thereby improving the performance of the split function. Theoretical and experimental results support the proposed framework.


Peer review:
[Review]:
This paper studies a new split rule for classification trees, based on the distances to two linear subspaces constructed at each node.  Subspaces are constructed by partitioning a data subset by class, and finding a linear transformation that attempts to concentrate each same-class space, while maximizing angle distance between the two (different-class) spaces.  At test time, a left/right decision is made based on which subspace the test point is closer to (based on projected reconstruction).  For multiclass problems, classes are randomly assigned at each node to one of the two subspaces.  The method is supported by theoretical analysis, toy examples, simple image classification datasets, and a pixel labeling task.

I'm not very knowledgeable in random forests, but this seems like a decent proposal to me, effectively integrating a linear subspace method into the decision rule.

The image classification tasks are on fairly simple datasets, but nonetheless may be good fits for the approach.  One question I have is how dependent this method is on image alignment.  The three datasets all have fairly well aligned images, potentially lending themselves to gains by integrating linear subspace methods (in pixel-space).  Notably, the proposed method performs quite well on 15-Scenes -- I wonder if there may be any additional take-aways concerning alignment of the images in this dataset and how else this might be used in classification of such scene data.

The results from the kinect pixel labeling look pretty good, and strengthens the paper with an application different from classifying aligned images, but it would be nice to include a comparison to Denil et al (the dataset providers).


Pros:

- Method is well motivated and theoretically analyzed
- Demonstrated to be effective in several tasks
- Very well written

Cons:

- Applications are somewhat simple


More questions:


- Fig. 5:  It would be interesting to put the 'Identity learner' (subspace assignment without the transformation T) in all 3 of these plots.  I'd be particularly interested to know the effect of this step on MNIST, which according to Fig 6 seems reasonably separated in the original space as well.

- Pixel descriptors for kinect:  I think they are depth differences from the center pixel obtained at 32 equally spaced points along circles at 3 different radii (8, 32, and 64), forming a 96-dim descriptor -- is this correct?  I wasn't entirely sure based on the description.

- 'All the images were resized to 16 x 16':  It's ambiguous whether this was for just YaleB, or all the datasets?