Zero-Shot Learning by Convex Combination of Semantic Embeddings
Authors: Tomas Mikolov, Andrea Frome, Samy Bengio, Jonathon Shlens, Yoram Singer, Greg S. Corrado, Jeffrey Dean, Mohammad Norouzi
ICLR 2014

Abstract: Several recent publications have proposed methods for mapping images into continuous semantic embedding spaces. In some cases the semantic embedding space is trained jointly with the image transformation, while in other cases the semantic embedding space is established independently by a separate natural language processing task, and then the image transformation into that space is learned in a second stage. Proponents of these image embedding systems have stressed their advantages over the traditional n-way classification framing of image understanding, particularly in terms of the promise of zero-shot learning -- the ability to correctly annotate images of previously unseen object categories. Here we propose a simple method for constructing an image embedding system from any existing n-way image classifier and any semantic word embedding model, which contains the n class labels in its vocabulary. Our method maps images into the semantic embedding space via convex combination of the class label embedding vectors, and requires no additional learning. We show that this simple and direct method confers many of the advantages associated with more complex image embedding schemes, and indeed outperforms state of the art methods on the ImageNet zero-shot learning task.


Peer review:
[Review]:
This paper proposes a method for performing zero-shot learning of an image labeling system. The proposed method is very simple and yet general and efficient: it consistently outperforms the DeVISE system presented recently on the ImageNet benchmark.

The paper is nicely written and ConSE is actually so simple that the paper is not too complicated too understand anyway. But I have no problem accepting a paper even if the method is simple, if it proves to be efficient. ConSE appears to be but some questions/comments remain.

The method is simple so I would have expected more studies/experiments/discussions to explain its good performance. A simple intuition is given is Section 4.1 with the (funny) 'liger' example. But this could perhaps detailed more. For instance:

- it is claimed several times that ConSE can be used with any semantic embeddings. Is this really true? According to the 'liger' example, ConSE works because s(liger) ~ 0.5*s(tiger) + 0.5*s(lion). It is true for the skip-grams embeddings, since it has been shown that such linear relationships (and translations) were existing among those embeddings. I'm not sure that one can claim that all word embeddings work the same way and have such linear relationships. Without such property within the embedding space, would ConSE still perform well?

- how important is it to normalize the T top probabilities of the combination? Doing so, they are implicitly calibrated on the train labels, whereas one would like better to calibrate them on train + test labels. In the conclusion, there is an interesting comment regarding the norm of the convex embedding combination, especially when probabilities are not normalized, indicating that it gives a measure of the confidence of the prediction. I feel like the main point of the paper might be there but the paper does not exploit it well. Basically, one of the most difficult problem in zero-shot learning is to detect whether to choose a label among training labels or test labels (before even trying to choose the right one). That's why for me the most interesting (and realistic) experiments of the paper are when train labels are also added to the candidate label sets (+1K setting). These show that the bias towards training labels is big, especially at top-1 (this is not surprising). The intuition about the norm of the convex combination and its connection to confidence seems to be promising to soften this bias, but this is just sketched unfortunately.

I wonder why the performance of ConSE(1) in hits@1 is not 0 in the +1K setting. If I understand correctly, the output of ConsE(1) is simply the embedding of the top-predicted train label and hence, the closest according to the cosine distance should be this very train label. Since, no test example is labeled with a train label, it should always be a mistake.

On a more general point, it could also be discussed how significant are the results. I mean, perhaps ConSE outperforms DeVISE, but performance are very low (5% of hits@10 in the most general setting). Can we consider that such poor performance is still actually meaningful and useful in some way?

Minor:
- Tables 1 & 4 are in %, whereas tables 2 & 3. It should be consistent.