Zero-Shot Learning by Convex Combination of Semantic Embeddings
Authors: Tomas Mikolov, Andrea Frome, Samy Bengio, Jonathon Shlens, Yoram Singer, Greg S. Corrado, Jeffrey Dean, Mohammad Norouzi
ICLR 2014

Abstract: Several recent publications have proposed methods for mapping images into continuous semantic embedding spaces. In some cases the semantic embedding space is trained jointly with the image transformation, while in other cases the semantic embedding space is established independently by a separate natural language processing task, and then the image transformation into that space is learned in a second stage. Proponents of these image embedding systems have stressed their advantages over the traditional n-way classification framing of image understanding, particularly in terms of the promise of zero-shot learning -- the ability to correctly annotate images of previously unseen object categories. Here we propose a simple method for constructing an image embedding system from any existing n-way image classifier and any semantic word embedding model, which contains the n class labels in its vocabulary. Our method maps images into the semantic embedding space via convex combination of the class label embedding vectors, and requires no additional learning. We show that this simple and direct method confers many of the advantages associated with more complex image embedding schemes, and indeed outperforms state of the art methods on the ImageNet zero-shot learning task.


Peer review:
[Review]:
This paper addresses the problem of zero shot learning for image classification. Like in their recent NIPS2013 work, the authors relies on an embedding representation of the classes inferred from a language model. Their prediction scheme first predicts an embedding vector which linearly combines the vectors representing the n-best prediction among the training classes and  then looks for the nearest neighbors of the predicted vector among the test class embedding.

The paper reads well and appropriate reference to related work is given. The proposed approach is very simple and yet improve over the DEVISE classifier. I have however a concern regarding the results which either indicates (i) the implementation differs from the paper description, (ii) I missed something. It seems to me that hit@1 for ConSE(1) predicts the embedding of the best prediction of the 'Softmax baseline' over the 1,000 training classes, i.e. f(x) = s(y_0(x,t)) and outputs its nearest neighbor in the search space. When the search space include the training classes, it should output y_0(x,t). This implies that in table 4, the first column should contain hit@1 for ConSE(1) = 55.6. Similarly, the result of hit@1 for ConSE(1) in the (+1K) results should be 0. This is not the case, could you explain/correct?

Apart from this technicality, this is a good paper. The approach is simple and improves the state of the art. It might be further improved by a deeper analysis of the errors, possibly grouping them according by type of classes, looking at the accuracy of the convnet on the classes related to the labels or the quality of the corresponding text embeddings.