Counting Grids : Reconstructing the feature layout from bags by capturing spatial interdependence in image features

Motivation A popular way to deal with diversity of imaging conditions and geometric variation in entire scenes is to represent images as disordered “bags” of features. It is tempting to use here the existing discrete models, such as histograms, or latent Dirichlet allocation (LDA, [2]). However, the bags of features extracted from natural images have an imprint of the images’ spatial structure, which is evident when the bags from related images are considered together, and ignoring the constraints imposed on the feature counts may have negative consequences in classification tasks. Fig. 1 provides a synthetic example of several images of a train station, taken as windows into the larger scene i). Just for illustrative purposes, we hand-labeled the scene with feature labels as shown in ii). Assuming that a few images are taken at random from the scene, we wonder if the feature counts in these images are sufficient to predict the possible feature counts in other images of the scene. In particular, we consider images taken from the regions close to A,B, and C in ii) and ask the question if the image D would fit the so defined train station class. The literature uses two sets of approaches to this problem. In one set of approaches, kernel or nearest-neighbor techniques [6] start with the comparisons of the feature counts in the test image and each of the previously studied exemplars. Although this comparison can be done in many different ways, we note here that these approaches would be complicated by the fact that none of images A,B, C have the combination of all five features that are present in D. The other approach is to consider all bags of features together and generalize [2]; in this case, there is a danger of overgeneralization. For this particular example, there is a need for interpolating between the feature count vectors for A,B,C and other images and the interpolation is best performed by spatial reasoning. Given that in some training images we see, from the top to bottom, roof, train, tracks, and in others mountain, grass, roof, train, we can infer that the existence of grass, roof, train, tracks combination is likelier than the existence of the mountain, roof, train, tracks combination of features. Furthermore, the proportions of different features in the images carry the information about the thickness of the layers of these features, which should be useful for inferring which previously unseen feature count combinations can be found elsewhere in the scene. Surprisingly, not much of the spatial organization of the features in the training images1 needs to be retained in order to perform the spatial reasoning about which feature combinations are likely. The recent Counting Grid (CG) model [1] represents each input image as a point in a large latent grid of feature counts. This latent point is a corner of a window of grid points which are all uniformly combined to match the (normalized) feature counts in the image. Being a bag of word model with spatial layout in the latent space, they has superior handling of field of view changes in comparison to other bag of word models. In Fig.1-iii) we show the Counting Grid inferred on the label counts from 50 windows into the scene taken at random, but avoiding all windows that contain all five of the features in D in any proportion. Although none of the training examples was taken from the area close to D where all five of D’s features can be seen in a single image, that part of the scene is reconstructed as well, and D’s histogram can be matched well. In this simple example, the training images are different views of a single scene. However, at the feature level, images of other train stations are likely have a similar layout, and so they could be used to learn a counting grid. For example in Fig.1iv) we used the high level annotation of LabelMe [3] while in Fig.2 hundreds of real features and inferred the counting grids from related images of different scenes of the SenseCam dataset [4]. The CG model is able to reconstruct the feature layout only exploiting the spatial interdependence in image features. Panoramas of office, and corridors are well visible in Fig.2iii) and, in general, every window opened in a Counting Grid, represent a possible bag of feature combinations2 present in the dataset.