Zero-Shot Learning by Convex Combination of Semantic Embeddings
Authors: Tomas Mikolov, Andrea Frome, Samy Bengio, Jonathon Shlens, Yoram Singer, Greg S. Corrado, Jeffrey Dean, Mohammad Norouzi
ICLR 2014

Abstract: Several recent publications have proposed methods for mapping images into continuous semantic embedding spaces. In some cases the semantic embedding space is trained jointly with the image transformation, while in other cases the semantic embedding space is established independently by a separate natural language processing task, and then the image transformation into that space is learned in a second stage. Proponents of these image embedding systems have stressed their advantages over the traditional n-way classification framing of image understanding, particularly in terms of the promise of zero-shot learning -- the ability to correctly annotate images of previously unseen object categories. Here we propose a simple method for constructing an image embedding system from any existing n-way image classifier and any semantic word embedding model, which contains the n class labels in its vocabulary. Our method maps images into the semantic embedding space via convex combination of the class label embedding vectors, and requires no additional learning. We show that this simple and direct method confers many of the advantages associated with more complex image embedding schemes, and indeed outperforms state of the art methods on the ImageNet zero-shot learning task.


Peer review:
[Review]:
A very interesting paper that proposes an extremely simple technique of mapping words and images to the same latent space for facilitating zero-shot learning. The paper shows how such a simple technique beats the recently proposed and more complicated DeVISE model. A few minor points though: 

1. As discussed by the above reader, ConSE(1) is exactly the same as Softmax baseline, especially when we include the 1K classes from the training set. The  results table seem to suggest something else. What's going on? 

2. The evaluation based on excluding and including the 1K classes from the training set seems to be a bit artificial. In the real deployment scenario one does not have access to this information and hence by default one should have the classes from training set as part of your evaluation classes. And that is the true performance of ConSE(*). 

3. Lastly, I don't really buy the author's argument that the proposed model is general and independent of how the word embeddings or the image features are generated. While I agree with the image part (that the model is independent of how the image features are generated), the same is not true for the word embeddings. My sense is that one is able to meaningfully take simple weighted averages of the embeddings of a collection of words is a by-product of the linear model used to train the word embeddings. If one was to use a non-linear model (NN for instance) so that the embeddings lie on a non-linear manifold, things might not work as well. Any thoughts? 

Otherwise, a very nice paper. Well written too.