Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper tackles zero-shot and generalised zero-shot learning by using the per-image semantic information. An instance-based loss is introduced to align images and their corresponding text in the same embedding space. To solve the extreme imbalanced issue of generalized zero-shot learning, the authors propose to scale the prediction scores of seen classes by a constant factor. They demonstrate technic contributions on CUB and Flowers datasets and the results achieve the state-of-the-art.
This paper should be rejected because (1) motivation is not well justified. This paper fails to convince me that it is practical to use per-image semantic information. Zero-shot learning aims to reduce the cost of annotation, but the per-image semantic information used in this paper conversely increases the annotation efforts. More specifically, this paper directly applies the per-image text descriptions proposed by Reed et al. '16, in which they annotated each image of CUB and Flowers by 10 different sentences. Due to the lack of such expensive annotations, this paper can be only evaluated on CUB and Flowers datasets,  results on other popular zero-shot learning datasets e.g., AWA, SUN, ImageNet, are essentially missing.
(2) novelty is limited. Using per-image semantic information is not new in zero-shot learning at all. Reed et al. '16 studied this problem and showed that per-image text-description can surpass per-image attributes in zero-shot learning. The loss function of this paper is also similar to  Reed et al. '16, which used the max-margin loss to align image and text pairs, v.s. the cross-entropy loss of this paper. Claiming that they are the first to use per-image semantic information in GZSL is not convincing because GZSL was not introduced at the time of Reed et al. '16. Metric scaling is not novel either. It is, in fact, equivalent to the calibration technic proposed by Chao et al. '16. The theory of metric scaling is redundant in my point of view because it is obvious that rescaling decreases the seen class scores. 
(3) experiments are insufficient to support the contribution. Their experiments are not comparing apple to apple. Specifically, the competing methods are using per-class semantic information while their approach uses per-image semantic information which is unfair because per-image semantic information includes much more supervision. The authors should have compared the results of those methods trained with per-image semantic information. I would expect all the approach will benefit from per-image side information. 



Post-rebuttal comments:  In the author responses, the authors have written long stories to fight against my reviews. But unfortunately,  a large part of the responses is not addressing my concerns. The authors repeatedly argue that their main contribution is to show that image-level supervision is an effective way to tackle (generalized) zero-shot learning. However, this contribution is not significant because Reed et al. '16 has demonstrated this point in the scenario of zero-shot learning. Extending this idea to generalized zero-shot learning is not a sufficient contribution for ICLR.  

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper proposes to to use four different losses to train a joint text-image embedding space for zero-shot learning. The four losses consist of a classification loss given text descriptions, a classification loss given images, two contrastive losses given pairs of text and images. The paper also discusses how to balance seen and unseen classes, and it seems that, empirically, embeddings for seen classes are closer together while the embeddings for the unseen classes are further apart. A scaling factor that makes sure these distances are comparable for seen and unseen classes is introduced, and the paper gives an explanation for such a scaling. The final performance on the CUB and FLOWERS data set is impressive.

I am giving a score of 3. The engineering effort in the paper is appreciated, but the novelty is lacking. The losses introduced in the paper, broadly classified as contrastive loss, has been around since (Frome et al., 2013). See also (Karpathy et al., 2014; Wang et al., 2016) and in particular (Xian et al., 2016). The subtle differences between all these losses are whether there is a margin, whether the loss is smooth (i.e., a softmax as opposed a max), and how the negative samples are selected. I am surprised that the loss function being used by many are considered a contribution in the paper. The property of metric scaling is certainly interesting, but it does not answer why metric scaling is needed in the first place. Overall, the novelty is limited.

Below are some minor points and questions for the paper.

... anchor embeddings learned in one modality as prototypes ...
--> the word "prototype" is used extensively in the rest of the paper. i know this is a common term in the zero-shot learning community, but it might be good to give a formal definition early or at least give an informal definition.

Algorithm 1
--> what is the choice of d in the experiments?

... the probability of image v_i and text t_j to belong to the same object distance
--> isn't this the constrastive loss? what does it mean for the two to belong to the same object distance?

equations (2)
--> it seems that v_i and t_i are paired, but it is unclear from the text.

equation (1) and (2)
--> have you considered other (distributions of) negative samples?

... P{\hat{y} \in Y^{tr} | y_v \in Y^{ts}} is significantly greater than P{\hat{y} \in Y^{tr} | y_v \in Y^{tr}}
--> what's the definition of P{\hat{y} | y_v}? the notation is confusing and hinders the understanding of the rest of the discussion. it's probably easier to just talk about distances and use the nearest neighbor argument.

Figure 2
--> what's different between the 10 different runs? random seeds?

Figure 3
--> why is the performance significantly lower when lambda=0? can this be related to how negative samples are selected?

References:

DeViSE: a deep visual-semantic embedding model
A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, T. Mikolov
Neurips, 2013

Deep fragment embeddings for bidirectional image sentence mapping
A. Karpathy, A. Joulin, F. Li
Neurips, 2014

Learning deep structure-preserving image-text embeddings
L. Wang, Y. Li, S. Lazebnik
CVPR, 2016

Latent embeddings for zero-shot classification
Y. Xian, Z. Akata, G. Sharma, Q. Nguyen, M. Hein, B. Schiele
CVPR, 2016

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper presents two main contributions: first, a simple retrieval-based objective for learning joint text and image representations, and second, a metric-scaling term that improves performance on unseen classes in the generalized zero-shot (GZSL) setting. They evaluate on two datasets, CUB and FLOWERS, consisting of images paired with text descriptions, and show that their relatively simple technique outperforms complex GAN and VAE-based models.

Weak accept, see considerations below. The results seem solid, and the architecture is quite simple compared to other models on this task. However, there are some gaps in the analysis, and the paper would benefit from a more careful comparison to the literature.

The results are strong, and it’s encouraging to see a simple approach like this outperform more complex models. However, it’s not clear how novel the retrieval model is; as implemented, it’s similar to a sampled softmax or noise contrastive estimation - the paper would benefit from a more careful comparison of their approach to established alternatives, and a discussion of why this approach should learn better representations than a VAE or GAN. It’s also not clear how novel the metric scaling technique is. It undeniably works well, but the use seems remarkably similar to (Das & Lee 2019, Section II.D).

The observation (Section 4.5) that the retrieval-only model performs well without any class labels is interesting. How does this compare to other unsupervised approaches? It also appears that the version with the metric scaling here still performs better - is this because the retrieval model is trained only on examples from the “seen” classes? Does this gap go away if (unlabeled) examples from the unseen classes are available at training time?

A few other notes:
* Section 4.5: “Our interpretation is that the addition of the image/text classifier loss helps to reduce the intraclass variability in embeddings and provides for tighter clustering.” - could you test this directly, or provide some visualization, such as t-SNE plots?
* The image representations are initialized from an ImageNet model, but text is trained from scratch. Why not use a pretrained text encoder like BERT?
* Table 3: it would help to have text labels for these rows, or a reminder in the caption of what \lambda = 0 and \kappa = 0 mean (otherwise, one needs to flip/scroll back to page 4)
* Figure 2: what are the actual values of u and s that contribute to this plot?

review_title: Official Blind Review #2

Meta-review: This paper demonstrates that per-image semantic supervision, as opposed to class-only supervision, can benefit zero-shot learning performance in certain contexts.  Evaluations are conducted using CUB and FLOWERS fine-grained zero-shot data sets.  In terms of evaluation, the paper received mixed final scores (two reject, one accept).

During the rebuttal period, both reject reviewers considered the author responses, but in the end did not find the counterarguments sufficiently convincing.  For example, one reviewer maintained that in its present form, the paper appeared too shallow without additional experiments and analyses to justify the suitability of the contrastive loss used for obtaining embeddings applied to zero-shot learning.  Another continued to believe post-rebuttal that reference Reed et al., (2016) undercut the novelty of the proposed approach.

And consistent with these sentiments, even the reviewer who voted for acceptance alluded to the limited novelty of the proposed approach; however, the author response merely states that a future revision will clarify the novelty.  But this then requires another round of reviewing to determine whether the contribution is sufficiently new, especially given that all reviewers raised this criticism in one way or another.  Furthermore, the rebuttal also mentions the inclusion of some additional experiments, but again, we don't know how these will turn out.

Based on these considerations then, the AC did not see sufficient justification for accepting a paper with aggregate scores that are otherwise well below the norm for successful ICLR submissions.
Decision: Reject