Zero-Shot Learning by Convex Combination of Semantic Embeddings
Authors: Tomas Mikolov, Andrea Frome, Samy Bengio, Jonathon Shlens, Yoram Singer, Greg S. Corrado, Jeffrey Dean, Mohammad Norouzi
ICLR 2014

Abstract: Several recent publications have proposed methods for mapping images into continuous semantic embedding spaces. In some cases the semantic embedding space is trained jointly with the image transformation, while in other cases the semantic embedding space is established independently by a separate natural language processing task, and then the image transformation into that space is learned in a second stage. Proponents of these image embedding systems have stressed their advantages over the traditional n-way classification framing of image understanding, particularly in terms of the promise of zero-shot learning -- the ability to correctly annotate images of previously unseen object categories. Here we propose a simple method for constructing an image embedding system from any existing n-way image classifier and any semantic word embedding model, which contains the n class labels in its vocabulary. Our method maps images into the semantic embedding space via convex combination of the class label embedding vectors, and requires no additional learning. We show that this simple and direct method confers many of the advantages associated with more complex image embedding schemes, and indeed outperforms state of the art methods on the ImageNet zero-shot learning task.


Peer review:
[Review]:
We thank reviewers for their valuable feedback. We will prepare a new version of the paper shortly to address your comments.

R1: ... concern regarding the results which either indicates (i) the implementation differs from the paper description, (ii) I missed something ...
R2: ... why the performance of ConSE(1) in hits@1 is not 0 in the +1K setting ...
Sumit Chopra: ... ConSE(1) is exactly the same as Softmax baseline ... What's going on?

This is a good point related to a technical detail of our implementation. In the Imagenet experiments, following the experimental setup of the DeViSE model, there is not a one-to-one correspondence between the class labels and the word embedding vectors. Rather, because of the way the Imagenet synsets are defined, each class label is associated with several synonym terms, and hence several word vectors. When we perform the mapping from the softmax scores to the continuous embedding space, we average the word vectors associated with each class label, and then linearly combine the average vectors according to the softmax scores. However, when we rank the word vectors to find the k most likely class labels, we search over individual word vectors, without any averaging of the synonym words. Thus, the ConSE(1) might produce an average embedding, which is not the closest vector to any of the word vectors corresponding to the original class label, and this results in a slight difference in the hit@1 scores for ConSE(1) and the softmax baseline in the +1K setting. While other alternatives exist for this part of the algorithm, we intentionally kept the ranking procedure exactly the same as the DeViSE model to perform a direct comparison with DeViSE. We will include this description in the new version of the paper.

Sumit Chopra: The evaluation based on excluding and including the 1K classes ... a bit artificial. ... one should have the classes from training set as part of your evaluation classes. And that is the true performance of ConSE(*).

The separation of the training and unseen classes enables for a more detailed analysis of the results to obtain better insights into the behavior of the algorithm. Based on these results we concluded that our model overfits to the training class labels, and we need some better regularization mechanism to overcome this behavior. We believe that one can exploit better methods for distinguishing between the 1K training labels and the zero-shot labels, one of which was mentioned in the conclusion. Alternatively, one can collect training labels for a category called 'none-of-training', which includes instances from all categories except the 1K training classes (not reflecting their specific labels). Using examples from this additional category, one can re-train an additional node for the softmax layer to detect examples of the none-of-training category. This helps deciding whether we should perform a zero-shot prediction or we should stick to the original softmax predictions. That said, the experimental results that exclude the 1K training classes report the accuracy that one can hope for in the best case scenario, i.e., when the zero-shot labels can be perfectly distinguished from the 1K training labels. Moreover, these results suggest that our naive way of combining the 1K training labels with the zero-shot labels, i.e., treating them as if they are the same hurts the performance, and it is important to implement a better method for detecting the zero-shot labels.

R2: ... it is claimed several times that ConSE can be used with any semantic embeddings. Is this really true? ...
Sumit Chopra: ... simple weighted averages of the embeddings of a collection of words is a by-product of the linear model used to train the word embeddings ...

We do not argue that any word embedding model used within our framework is equally effective. However, our observation is that we did not fine-tune our algorithm for any specific word embedding representation. This suggests that our algorithm is relatively robust to the specific choice of the word embedding representation, and in fact, we obtained some promising results on the use of a very different word embedding model within exactly the same model. That said, we agree that the topology of the manifold of the word embedding vectors matters, and some word embedding representations might be more suitable for our framework than others. We will reword the paper to clarify that we claim some degree of robustness, and not invariance against the choice of the language model.

R2: ... how important is it to normalize the T top probabilities of the combination? ...

It is not important to normalize the top T probabilities. In our model, all of the word embedding vectors are unit-norm, and we use cosine similarity between the ConSE's prediction and the unit-norm word vectors for k-nearest neighbor ranking. Thus, even if the ConSE's prediction is not normalized, the ranking of the word vectors does not change. We agree that using the norm of the ConSE's prediction, one can come up with better ways to address the separation of training and unseen classes, but we keep this as a future work.

R2: ... how significant are the results. ... perhaps ConSE outperforms DeVISE, but performance are very low ... Can we consider that such poor performance is still actually meaningful and useful in some way?

Although the performances do not look great (5% on hits@10), they are somewhat much better in reality. Humans are robust to minor mistakes between very similar concepts (e.g., multiple types of sea lion, or types of hamster), but our flat metric is not good at rating minor mistakes higher than major errors. If the model predicts 'Australian sea lion' but the correct label is 'Steller sea lion', then we get a score of zero according to the flat metric. Figure 1 shows some actual predictions of the model, which demonstrates that even when we fail to return the right answer (most of the time), we often return very reasonable labels. So, if the question is whether this is useful for applications, the answer is clearly we are moving in that direction.