Zero-Shot Learning by Convex Combination of Semantic Embeddings
Authors: Tomas Mikolov, Andrea Frome, Samy Bengio, Jonathon Shlens, Yoram Singer, Greg S. Corrado, Jeffrey Dean, Mohammad Norouzi
ICLR 2014

Abstract: Several recent publications have proposed methods for mapping images into continuous semantic embedding spaces. In some cases the semantic embedding space is trained jointly with the image transformation, while in other cases the semantic embedding space is established independently by a separate natural language processing task, and then the image transformation into that space is learned in a second stage. Proponents of these image embedding systems have stressed their advantages over the traditional n-way classification framing of image understanding, particularly in terms of the promise of zero-shot learning -- the ability to correctly annotate images of previously unseen object categories. Here we propose a simple method for constructing an image embedding system from any existing n-way image classifier and any semantic word embedding model, which contains the n class labels in its vocabulary. Our method maps images into the semantic embedding space via convex combination of the class label embedding vectors, and requires no additional learning. We show that this simple and direct method confers many of the advantages associated with more complex image embedding schemes, and indeed outperforms state of the art methods on the ImageNet zero-shot learning task.


Peer review:
[Review]:
This paper presents a simple but really neat idea of combining semantic word vectors trained on text with a softmax classifier's output.
Instead of taking the softmax output as is, it uses its probabilities to weigh the semantic vectors of all the classes which allows the model to assign labels that were not present in the training data.

The results are not always better than previous work from the group but in many settings they are.

Simple but overall very cool idea.