Learning Image Representations for Efficient Recognition of Novel Classes

Introduction In this work we consider the problem of efficient object-class recognition in large image collections. We are specifically interested in scenarios where the classes to be recognized are not known in advance. The motivating application is “object-class search by example” where a user provides at query time a small set of training images defining an arbitrary novel category and the system must retrieve from a large database images belonging to this class. This application scenario poses challenging requirements on the system design: the object classifier must be learned efficiently at query time from few examples; recognition must have low computational cost with respect to the database size; finally, compact image descriptors must be used to allow storage of large collections in memory rather than on disk for additional efficiency. Traditional object categorization methods do not meet these requirements as they employ high-dimensional descriptors and they typically use non-linear kernels which render them computationally expensive to train and test. For example, the LP-β multiple kernel combiner described in [2] achieves state-of-the-art accuracy on several categorization benchmarks but it requires over 23 Kbytes to represent each image and it uses 39 feature-specific nonlinear kernels. Note that the use of this recognition model in our application would require costly query-time kernel evaluations for each image in the database since the training set varies with every new query and thus pre-calculation of kernel distances is not possible. We propose to address these storage and efficiency requirements by learning a compact binary image representation optimized to yield good categorization accuracy with linear (i.e., efficient) classifiers. The binary entries in our image descriptor are thresholded nonlinear projections of low-level visual features extracted from the image, such as descriptors encoding texture or the appearance of local image patches. Each non-linear projection can be viewed as implementing a nonlinear classifier using multiple kernels. The intuition is that we can then use these pre-learned multiple kernel combiners as a classification basis to define recognition models for arbitrary novel categories: the final classifier for a novel class is obtained by linearly combining the binary outputs of the basis classifiers, which we can pre-compute for every image in the database, thus enabling efficient novel object-class recognition even in large datasets. The idea of describing images in terms of basis classes is evocative of the use of attributes [1, 3] which are fully-supervised classifiers trained to recognize certain properties in the image such as “has beak”, “near water”. The recognition model for each class is then defined by hand-specifying its associated attributes. The approach proposed here is most closely related to our prior work [5], where we have introduced the “classeme” image descriptor for general class recognition: the entries of this descriptor are the outputs of a large set of weakly-trained basis classifiers evaluated on the image. We have shown that linear classifiers trained on classeme vectors can recognize novel classes with near state-of-the-art accuracy even when the descriptor is compressed down to less than 200 bytes per image. In subsequent work [4], Li et al. have proposed using the localized outputs of object detectors as image representation. The advantage of this representation is that it encodes spatial information; furthermore, object detector are more robust to clutter and uninformative background than classifiers evaluated on the entire image. These prior methods work under the assumption that an “overcomplete” representation for classification can be obtained by pre-learning classifiers for a large number of basis classes, some of which will be related to those encountered at testtime. Such high-dimensional representations are then compressed down using quantization, dimensionality reduction or feature selection methods in a stage subsequent the learning of the basis classifiers [5, 4]. Unlike prior work, where the basis classifiers are learned disjointly and each is optimized to recognize a predefined basis class, we propose instead to jointly seek the basis classifiers such that linear combinations of these classifiers yield optimal classification accuracy on a given training set. In other words, our basis classifiers are recognizers of “abstract” (as opposed to predefined) object categories and they are collectively trained to produce good classification when used as a basis in a linear combination. This allows us to decouple the number of basis classifiers from the number of categories in the training set and thus to learn