Sequentially Generated Instance-Dependent Image Representations for Classification
Authors: Matthieu Cord, patrick gallinari, Nicolas Thome, Ludovic Denoyer, Gabriel Dulac-Arnold
ICLR 2014

Abstract: In this paper, we investigate a new framework for image classification that adaptively generates spatial representations. Our strategy is based on a sequential process that learns to explore the different regions of any image in order to infer its category. In particular, the choice of regions is specific to each image, directed by the actual content of previously selected regions.The capacity of the system to handle incomplete image information as well as its adaptive region selection allow the system to perform well in budgeted classification tasks by exploiting a dynamicly generated representation of each image. We demonstrate the system's abilities in a series of image-based exploration and classification tasks that highlight its learned exploration and inference abilities.


Peer review:
[Review]:
This paper describes a method to select the most relevant grid regions from  an input image to be used for classification.  This is accomplished by training a chain of region selection predictors, each one of which outputs the k'th region to take, given the image features from k-1 already-selected regions.  Each selector is trained to choose a region that leads to an eventual correct classification, given already-trained downstream selectors.  Since downstream predictions are required for training, the chain is trained last-to-first, and random region selection is used to generate training inputs at each stage.  Both the conditional region selection chain and its training method are interesting contributions.

The method is evaluated on two tasks, playing vs. holding a musical instrument (PPMI) and outdoor scene classification (15-Scenes).  Here, I wish the authors were more detailed in their descriptions of these tasks.  In particular, I'm a bit unsure whether the PPMI task is a 12-way classification on musical instruments, or an average of 12 binary classification tasks (playing vs holding), one for each instrument.  I think it's the latter -- if so, I'm also unclear on whether a different selection/classification chain was trained and tested for each of the 12 subsets, or if a single classifier was trained over the entire dataset.

Still, the proposed method beats a random selection baseline for both tasks (though not by much for 15-Scenes), and for PPMI it also beats a baseline of including all regions.  The latter is a particularly nice result, since intuitively region selection should stand to help performance, yet such gains can be hard to find.  Evaluating the 12- or 24-way instrument classification task for PPMI would have been good here as well, though, as there is clearly a compatibility between region selection and this data and/or task, and this may help provide insight into why that is.

Pros:

- Interesting and new method of region selection trained for classification
- Shows a nice gain in one task and reasonable results in another
- Interesting discussion sheds light on how the method operates (figs 7, 8)

Cons:

- Tasks and datasets could be better explained


Questions:

- Why does selecting 8 regions beat using all 16 for PPMI but is only about the same for 15-Scenes?  Some discussion on the difference between the two datasets and their fit with region selection would be nice here.

- Fig. 8:  I might have expected the B=8 (right) histogram to have its highest values mostly where the B=4 (left) histogram does, since one would think the best regions would be required in both cases.  However, region 3 (x=1,y=3) seems to be used with more frequency for B=4 than B=8, for example.  Why does this occur?

- Is it possible to continue retraining the classifier and selectors?  Currently the chain is trained once, starting with the classifier and proceeding upstream.  Yet by doing this, each stage must be trained on a random sample of input regions, which can include many more configurations than would be seen at test time.  Could each stage (particularly the final classifier f) be iteratively retrained given *all* selectors?  Would this help by adjusting the training distribution closer to the test distribution and allowing better use of resources, or might it lead to worse generalization by narrowing the training set too much?

- Alg. 4 adds a sample to the training set only if it leads to a correct prediction.  But what if no region has this property -- is it better to ignore these cases, or should some be included (perhaps by trying to predict the choice closest to correct)?  Surely such cases will arise at test time, and the consequences of ignoring them isn't entirely clear.  I suppose there's an argument to be made that the classifier will eventually fail anyway, so it's better to bail on these cases and concentrate the predictor's resources only on those where it stands a chance.  But in cases where the classifier is wrong, might this also lead to more arbitrary region selection and more drastic types of mistakes (e.g. mistaking a clarinet for a harp vs a recorder)?