Title: CLAREL: CLASSIFICATION VIA RETRIEVAL LOSS FOR ZERO-SHOT LEARNING

Abstract: We address the problem of learning fine-grained cross-modal representations. We propose an instance-based deep metric learning approach in joint visual and textual space. The key novelty of this paper is that it shows that using per-image semantic supervision leads to substantial improvement in zero-shot performance over using class-only supervision. On top of that, we provide a probabilistic justification for a metric rescaling approach that solves a very common problem in the generalized zero-shot learning setting, i.e., classifying test images from unseen classes as one of the classes seen during training. We evaluate our approach on two finegrained zero-shot learning datasets: CUB and FLOWERS. We find that on the generalized zero-shot classification task CLAREL consistently outperforms the existing approaches on both datasets.

1. INTRODUCTION:

Deep learning-based approaches have demonstrated superior flexibility and generalization capabilities in information processing on a wide variety of tasks, such as vision, speech and language (LeCun et al., 2015) . However, it has been widely realized that the transfer of deep representations to real-world applications is challenging due to the typical reliance on massive hand-labeled datasets. Learning in the low-labeled data regime, especially in the zero-shot (Wang et al., 2019) and the few-shot (Wang & Yao, 2019) setups, have recently received significant attention in the literature. In the problem of zero-shot learning (ZSL), the objective is to recognize categories that have not been seen during the training (Larochelle et al., 2008) . This is typically done by relying on anchor embeddings learned in one modality as prototypes and by associating a query embedding from the other modality with the closest prototype. In the generalized ZSL (GZSL) case (Xian et al., 2018c) , the objective is more challenging as recognition is performed in the joint space of seen and unseen categories. ZSL, as well as its generalized counterpart, provide a viable framework to learn cross-modal representations that are flexible and adaptive. For example, in this paradigm, the adaptation to a new classification task based on text/image representation space alignment could be as easy as defining/appending/modifying a set of text sentences to define classes of new classifiers. This is an especially relevant problem as machine learning is challenged with the long tail of classes, and the idea of learning from pairs of images and sentences, abundant on the web, looks like a natural solution. Therefore, in this paper we specifically target the fine-grained scenario of paired images and their respective text descriptions. The uniqueness of this scenario is in the fact that the co-occurance of image and text provides a rich source of information. The ways of leveraging this source have not been sufficiently explored in the context of GZSL. Although we focus exclusively on the GZSL recognition setup in this paper, we believe that the research in this direction has potential to enable zero-shot flexibility in a wider array of high-level tasks such as segmentation or conditional image generation (Zhang et al., 2018) . The contributions of this work can be characterized under the following two themes.

Instance-based training loss. Most prominent zero-shot learning approaches rely heavily on classlevel modality alignment (Xian et al., 2018c) . We propose a new composite loss function that balances instance-based pairwise image/text retrieval loss and the usual classifier loss. The retrieval loss term does not use class labels. We demonstrate that the class-level information is important, but in the fine-grained text/image pairing scenarios, most of the GZSL accuracy can be extracted from the instance-based retrieval loss. To the best of our knowledge, this type of training has not been used in the GZSL literature. Its impressive performance opens up new promising research directions. Metric space rescaling. Metric-based ZSL approaches rely on distances between prototypes and query embeddings during inference. They are known to suffer from imbalanced performance on seen and unseen classes (Liu et al., 2018) . Previous work proposed to use a heuristic trick, calibrated stacking (Chao et al., 2016) or calibration (Das & Lee, 2019) , to solve the problem. We refer to this technique as metric rescaling in our work, and provide a sound probabilistic justification for it.

2. PROPOSED METHOD:

In this paper, we specifically target the fine-grained visual description scenario, as defined by Reed et al. (2016) . In this setting, the dataset consists of a number of images from a given set of classes and each image is accompanied by a number of textual descriptions. The task is to learn a joint representation space for images and texts that can be used for zero-shot recognition. An instance of the zero-shot multimodal representation learning problem can then be defined as follows. Given a training set To build g, most approaches to joint representation learning rely on class labeling to train a representation. For example, all the methods reviewed by Xian et al. (2018c) require the access to class labels at train time. We hypothesise that in the fine-grained learning scenario, such as the one described by Reed et al. (2016) , a lot of information can be extracted simply from pairwise image/text cooccurrences. The class labels really only become critically necessary when we define class prototypes, i.e. at zero-shot test time. Following this intuition, we define a composite loss function that relies both on the pairwise relationships and on the class labels. The high-level description of the proposed framework is depicted in Figure 1 . The framework enables us, among other things, to experiment with the effects of train-time availability of class labels on the quality of zero-shot representations.

S = {(v n , t n , y n ) | v n ∈ V , t n ∈ T , y n ∈ Y , n = 1 . . . N }

The framework is based on projecting texts and images into a common space and then learning a representation based on a mixture of four loss functions: a pairwise text retrieval loss, a pairwise image retrieval loss, a text classifier loss and an image classifier loss (see Algorithm 1).

Algorithm 1 Loss calculation for a single optimization iteration of the proposed method. N is the number of instances in the training set S tr , B is the number of instances per batch, C is the number of classes in the train set. RANDOMSAMPLE(S, B) denotes a set of B elements chosen uniformly at random from a set S, without replacement.

Input: Training set S tr = {(v 1 , t 1 , y 1 ), . . . , (v N , t N , y N )}, λ ∈ [0, 1], κ ∈ [0, 1].

Output: The loss J(φ, θ) for a randomly sampled training batch.

I ← RANDOMSAMPLE({1, . . . , N }, B) Select B instance indices for batch J T C (θ), J IC (φ) ← 0, 0 Initialize classification losses for i in I do z vi , z ti ← f φ (v i ), f θ (t i ) Embed images and texts p I ← softmax(W I z vi + b I ) Image classifier probabilities p T ← softmax(W T z ti + b T ) Text classifier probabilities J T C (θ) ← J T C (θ) + 1 B crossentropy(p T , y i ) Text classification loss J IC (φ) ← J IC (φ) + 1 B crossentropy(p I , y i ) Image classification loss end for J T R (φ, θ), J IR (φ, θ) ← 0, 0

Initialize retrieval losses for i in I do

J T R (φ, θ) ← J T R (φ, θ) + 1 B   d(z vi , z ti ) + log j∈I exp(-d(z vi , z tj ))   Text retrieval loss J IR (φ, θ) ← J IR (φ, θ) + 1 B   d(z vi , z ti ) + log j∈I exp(-d(z ti , z vj ))   Image retrieval loss end for J(φ, θ) ← λJ T R (φ, θ) + (1 -λ)J IR (φ, θ) Add retrieval loss to the total loss J(φ, θ) ← (1 -κ)J(φ, θ) + κ 2 (J T C (θ) + J IC (φ))

Add classification loss to the total loss

2.1. RETRIEVAL LOSS FUNCTION:

Pairwise cross-modal loss function is based solely on the pairwise relationships between texts and images. We choose to use the metric learning approach to capture the relationship between images and texts. Now, suppose d is a metric d : Z × Z → R + , v i is an image and τ = {t j } is a collection of arbitrary texts sampled uniformly at random, of which text t j belongs to v i . We propose the following model for the probability of image v i and text t j to belong to the same object instance:

p φ,θ (i = j|v i , t j , τ ) = exp(-d(f φ (v i ), f θ (t j ))) t j ∈τ exp(-d(f φ (v i ), f θ (t j )))

.

(1)

The learning is then based on the following cross-entropy loss defined on the batch of size B:

EQUATION

where i,j is a binary indicator of the true match ( i,j = 1, if i = j and 0 otherwise). Note that the expression above has the interpretation of the text retrieval loss. It attains its smallest value when for each image in the batch we manage to assign probability 1 to its respective text and 0 to all other texts. This can be further expanded as:

EQUATION

Exchanging the order of image and text in the probability model (1) leads to the image retrieval loss, J IR (φ, θ). The two losses are mixed using parameter λ ∈ [0, 1] as shown in Algorithm 1.

The pairwise retrieval loss functions are responsible for the modality alignment. In addition to those, we propose to include, as mentioned above, the usual image and text classifier losses. These losses are responsible for reducing the intraclass variability of representations. The classifier losses are added to the retrieval losses using a mixing parameter κ ∈ [0, 1] as shown in Algorithm 1.

2.2. BALANCING ACCURACY ON THE SEEN AND UNSEEN CLASSES:

Let us define class prototypes p(y), each based on the set of texts T y belonging to class y, {p(y) =

1 |T y | ti∈T y f θ (t i ) | y ∈ Y }.

In the context of GZSL, the nearest neighbor decision rule for a given image v and its features z v = f φ (v) has the following form:

y = arg min y∈Y d(z v , p(y)) .

(4)

The most acute problem faced in this setup is the accuracy imbalance between seen and unseen classes.

A very representative case clearly outlining the imbalance problem is presented in Table 6 of (Xian et al., 2018c) , where accuracy on the seen classes is always significantly greater than the accuracy on unseen ones. In order to measure and control the imbalance, three metrics are commonly used to assess the classification performance in the GZSL scenario: the Top-1 accuracy on the seen categories (s), the Top-1 accuracy on the unseen categories (u) and their harmonic mean, H = u • s/(u + s). The main metric to assess GZSL performance is then H, which quantifies both u and s.

To formalize the problem, we first introduce y v , the true class label of image v. Mathematically, the main GZSL pain point is that P{ y ∈ Y tr |y v ∈ Y ts } is significantly greater than P{ y ∈ Y ts |y v ∈ Y tr }. In other words, the problem is that a given image is more likely to be confused with one of the seen classes if it belongs to an unseen class than vice versa. Our approach to solving the problem is based on the following probabilistic representation of the event space for the decision rule in Equation ( 4):

EQUATION

Rephrasing, the most acute GZSL error happens when the prototype of one of the seen classes is closer to an image embedding from an unseen class than any of the prototypes of the unseen classes.

To rectify the situation we propose the following very direct solution to balance P{ y ∈ Y tr |y v ∈ Y ts } and P{ y ∈ Y ts |y v ∈ Y tr }. We introduce a positive scalar α ∈ R + and scale all the distances corresponding to the seen prototypes by 1 + α. This gives rise to the following scaled distance d α :

EQUATION

The misclassification between unseen as seen classes for the classifier y α , based on ( 6) is then:

EQUATION

and it has the following property: for any CUB FLOWERS CADA-VAE (Schönfeld et al., 2019) n/a n/a f-CLSWGAN (Xian et al., 2018d) 57.3 67.2 f-VAEGAN-D2 (Xian et al., 2019) 61.0 67.7 cycle-(U)WGAN (Felix et al., 2018) 

0 ≤ α 1 ≤ α 2 , P{ y α1 ∈ Y tr |y v ∈ Y ts } ≥ P{ y α2 ∈ Y tr |y v ∈ Y ts }, i.e. P{ y α ∈ Y tr |y v ∈ Y ts } is

3. RELATED WORK:

ZSL approaches aim at recognizing objects belonging to classes unseen during training (Larochelle et al., 2008; Palatucci et al., 2009) . This has been extended to the GZSL framework in which the decision space consists of both seen and unseen classes (Socher et al., 2013; Xian et al., 2018c) . The classical zero-shot approaches build a joint visual-semantic space, relying on a linear cross-modal compatibility function (e.g. dot-product between query embedding and semantic prototypes or a variation of a hinge loss) (Frome et al., 2013; Akata et al., 2015; 2016; Reed et al., 2016) . Non-linear variants of the compatibility has also been explored (Xian et al., 2016; Socher et al., 2013) . Extending previously proposed cross-modal transfer approaches based on auto-encoders (Hubert Tsai et al., 2017) and cross-domain learning (Gretton et al., 2007) , more recent line of work (Schönfeld et al., 2019; Xian et al., 2018d; 2019; Felix et al., 2018; Verma et al., 2018) relies on combining these approaches and their variations with dataset augmentation tools such as GAN (Goodfellow et al., 2014) and VAE (Kingma & Welling, 2014) . It is argued that the use of those tools helps to resolve one of the prominent problems in GZSL scenario: classifying images from unseen classes as one of the seen classes. There exist approaches that try to tackle this same problem via temperature calibration (Liu et al., 2018) originally proposed by Hinton et al. (2015) . Chao et al. (2016) ; Das & Lee (2019) proposed an approach to seen/unseen accuracy balancing that is very similar to ours, based on heuristic arguments. We extend this line of work here by providing a probabilistic justification for the balancing effect observed when applying metric rescaling. Atzmon & Chechik (2019) propose a more sophisticated way to deal with seen/unseen imbalance via adaptive confidence smoothing and gating, yet as authors note it is much simpler to train than the existing GAN-based zero-shot approaches. In this work, we introduce arguably the simplest zero-shot representation training approach of all, and we demonstrate that when the image level text information is available, it achieves the state-of-the-art results on GZSL task on two well-known datasets.

4. EXPERIMENTAL RESULTS:



4.1. DATASETS:

We focus on learning embeddings for fine-grained visual descriptions and test them in ZSL/GZSL scenario. To test the quality of trained embeddings we focus on datasets that provide paired images and text descriptions, such as Caltech-UCSD-Birds (CUB) (Welinder et al., 2010) and Oxford Flowers (FLOWERS) (Nilsback & Zisserman, 2008) , that were augmented with textual descriptions by Reed et al. (2016) . We use the GZSL splits proposed by Xian et al. (2018c) . The attribute-based datasets, such as SUN (Patterson et al., 2014) and AWA (Lampert et al., 2014) do not contain this information and do not have a notion of entity of a class in them. They are out of the scope of the current paper.

4.2. ARCHITECTURE AND TRAINING DETAILS:

Below, we provide more detailed description of parameters used to build and train the architecture depicted in Figure 1 . We use exactly the same hyperparameter settings for CUB and FLOWERS. The text feature extractor is built by cascading two ResNet blocks, followed by a BiLSTM. Each ResNet block has 3 convolutional/batch norm layers. The number of filters in the ResNet blocks is 128 and 256, BiLSTM has 512 filters for forward and backward branches (1024 total). All variables in the convolutional stack (including the batch normalization parameters γ and β) are L2-penalized with weight 0.001. The image feature extractor is a ResNet-101 with fixed weights pretrained on the split of ImageNet proposed by Xian et al. (2018c) . In this work we use precomputed image features, available in (Xian et al., 2018a) for CUB and in (Xian et al., 2018b) for FLOWERS. Image and text features are projected in the common embedding space of size 1024 with FC layers with no non-linearity. They are preceded with a dropout of 0.25. The trainable components of the model are trained for 150k batches of size 32 using SGD with initial learning rate of 0.1 that is annealed by a factor of 10 every 50k batches. For each batch, we sample 32 instances, each instance includes a vector of precomputed ResNet-101 features and 10 text descriptions corresponding to it, according to the original dataset definition Reed et al. (2016) . All 10 text descriptions are processed via the CNN/LSTM stack and the resulting embeddings are average pooled to create a vector representation of length 1024.

4.3. KEY RESULTS:

Our key empirical results are compared in Table 1 and in Table 2 against the latest state of the art. Our results are based on the settings of λ = 0.5, κ = 0.5 and α selected on the validation sets of CUB and FLOWERS datasets. Please refer to Section 4.5 for the analysis of stability with respect to the choices of λ and κ and Sections 2.2 and 4.4 for more details on the selection of α. The combination of the proposed training method and the rebalancing of the metric space results in the state-of-the-art performance. Most of the current methods rely on the dataset augmentation techniques based on GANs, VAEs or combinations thereof. Those are clearly complementary w.r.t. our method and their addition to the training procedure is likely to further boost the performance of our proposed approach. However, this is outside of the scope of the current work. Moreover, the proposed method is state-of-the-art on FLOWERS even when compared against (Atzmon & Chechik, 2019 ) that uses both more sophisticated GAN based embedding learning approach and a more sophisticated seen/unseen error rate balancing based on COSMO. It is important to note that Atzmon & Chechik (2019) did not report the sentence level results on CUB. Yet, when applied on attributes together with f-CLSWGAN (Xian et al., 2018d) COSMO resulted in 0.8% performance drop and when applied with LAGO (Atzmon & Chechik, 2018 ) it achieved 0.5 % improvement over the attribute based state of the art. Once the value of α is determined we train the representation on the full train+val subset and report results on the test split (the usual practice in GZSL). We confirm on the validation set that H exhibits an inverted U-shape behavior as a function of α, which was theoretically predicted in Section 2.2. Therefore, α can be selected on the validation set and then applied to re-scale the metric space to balance the accuracy on seen and unseen classes during test time as described in Section 2.2.

4.5. ABLATION STUDY:

Figure 3 presents the results of the ablation study on the importance of image and text retrieval losses. We see that all of the Top-1 accuracies (H, s, u) are stable in the range of λ ∈ [0.2, 0.9], when both losses have tangible weight. Removing either text or image retrieval losses (setting λ to 0 or 1 respectively) leads to performance drop in both cases. Removing the text retrieval loss (case λ = 0) results in the most significant drop. This is due to the fact that the text retrieval task is more tightly related to the GZSL task. At the batch level, retrieving the right text given an image is equivalent to identifying the correct class encoded by a text prototype during ZSL inference step. The image retrieval task is not directly related to solving the ZSL problem and yet it does yield a positive regularizing effect on both CUB and FLOWERS.

Figure 4 shows the results of the ablation study of the interplay between the retrieval loss and the classification loss. We observe, just as in the case with λ, that there exists a reasonably flat and stable range of κ ∈ [0.2, 0.6]. The range for κ is a bit smaller. κ = 1 results in the catastrophic performance drop: the classification losses by themselves do not enforce any modality alignment (please refer to Fig. 1 and Algorithm 1 clearly demonstrating this).

Table 3 studies the effects of different loss terms on the harmonic mean Top-1 accuracy H. The best result is achieved when all loss terms are active and when the metric space rescaling is on (the case of λ = 0.5, κ = 0.5 and α is checked, the last line in the table). Comparing this with the case when there is no metric space rescaling (first line with α = 0), we see that the rescaling helps to decrease the gap between seen and unseen classification accuracy. For CUB, the discrepancy reduction is from around 30% to around 6%, for FLOWERS it is from around 30% to around 1%. We would like to stress that we only use images and texts from the training set to achieve that. Going to the second line in the table (the image/text classification loss is inactive, κ = 0) and comparing it to the first one, we assess the effect of the image/text classification loss. It barely affects the performance on unseen set, but it significantly boosts the classification accuracy on the seen set (around 8% on both datasets). This is logical: adding a classifier loss results in a better classifier of the test images from the seen classes. This alone does not make it a better GZSL classifier, however. Only when applied together with metric space rescaling, this results in the performance boost (please refer to lines 1 and 6 in Table 3 ). Our interpretation is that the addition of the image/text classifier loss helps to reduce the intraclass variability in embeddings and provides for tighter clustering. However, this also leads to overfit on the classification task. This is accounted for by metric rescaling that enables the learnings from the image/text classification task be transferred effectively into the GZSL task.

The comparison of the last four rows of Table 3 leads us to believe that all the proposed loss terms outlined in Fig. 1 and Algorithm 1 are important for achieving the state-of-the-art performance.

Excluding any one of them (corresponding to the extreme values λ = 0, λ = 1, κ = 0) leads to performance deterioration. Finally, an interesting observation can be made by comparing line 3 of Table 3 with performance of algorithms in Table 1 . In this case our algorithm does not use any class labels and relies on training using exclusively the retrieval losses that can be calculated only based on the pairwise relationships between texts and images. We can see that using this type of supervision alone already results in a very high-quality representation. The representation is competitive against the latest GAN/VAE based approaches on CUB and is state-of-the-art on FLOWERS. This opens up new exploration avenues showing that in the case when very fine-grained modality outputs are available (image and text description pairs being a very prominent example), the high-quality representations may be learned without relying on manually supplied class labels.

5. CONCLUSIONS:

We propose and empirically validate two improvements to the process of learning fine-grained crossmodal representations. First, we confirm the hypothesis that in the context of paired images and texts, a deep metric learning approach can be driven by an instance-based retrieval loss resulting in competitive generalized zero shot classification results. Combined with an additional class label based image/text crossentropy term this results in state-of-the-art performance on two well known datasets, CUB and FLOWERS. This is an interesting result demonstrating that high-quality deep representations can be trained relying largely on pairwise relationships between modalities. On top of that, we propose a solution to one of the prominent problems in GZSL: classifying instances of unseen classes as seen ones. We mathematically analyze and empirically validate the method of adjusting a single scalar that transcends in its effectiveness advanced dataset augmentation and training approaches based on GANs and VAEs.

:



Figures and Tables:

Figure 1: The architecture and training diagram describing the proposed method. Each batch consists of randomly sampled instances, i.e. pairs of images and their corresponding texts. Images are embedded via ResNet and texts are embedded via a CNN/LSTM stack. Image and text features are projected via a fully connected layer into the same dimensional space. In this space, distances between text and image features from different instances are computed. The negative distances are fed into softmax to train on both the image and the text retrieval tasks. The image retrieval task consists of retrieving the image corresponding to the given text of the same instance and the text retrieval task is vice versa. In addition to that, image and text embeddings are trained on auxiliary image and text classification tasks on the class labels corresponding to instances.

a monotone non-increasing function of α and we can reduce it by increasing α (please refer to Appendix A for a proof). Consider now P{ y α ∈ Y tr |y v ∈ Y tr }, which is a probability that we classify an image v from one of the seen classes as still one of the seen classes. Using exactly the same chain of arguments as in Appendix A, it is straightforward to show that the probability is a non-increasing function of α. Hence the probabilityP{ y α ∈ Y ts |y v ∈ Y tr } = 1 -P{ y α ∈ Y tr |y v ∈ Y tr } is a non-decreasing function of α.Therefore, as α increases, we expect more classification errors in classifying images from seen classes, because some of them will be classified as one of the unseen classes.To sum up, given the arguments presented above we expect that by varying α > 0 we can balance the error rate P{ y α ∈ Y tr |y v ∈ Y ts } of leaking the unseen class images into seen class classification decision and the error rate P{ y α ∈ Y ts |y v ∈ Y tr } of leaking the seen class images into unseen class classification decision. This is possible as we just showed above that P{ y α ∈ Y tr |y v ∈ Y ts } is a non-increasing function of α, while P{ y α ∈ Y ts |y v ∈ Y tr } is a non-decreasing one. It is also important to emphasize that α is applied only to distances between the query embedding and the prototypes of seen classes and it is constant over seen classes. Therefore, the application of α does not at all affect the classification error rates either within Y tr or within Y ts . Varying α balances exclusively the classification errors arising from transitions between seen and unseen class labels. We study the empirical aspects of balancing α in Section 4.4.

Figure 2: Harmonic mean Top-1 accuracy on seen and unseen, H, against the value of α on the validation set. The curves represent the mean and 95% confidence intervals over 10 optimization runs. Results are stable over different runs. H exhibits a distinct inverted U-shape w.r.t. α.

Figure3: Harmonic mean Top-1 accuracy on seen and unseen, H, against λ, the relative weight of image and text retrieval loss terms. λ = 0 corresponds to the case of image retrieval loss having weight 1 and text retrieval loss having weight 0. Mean over 10 optimization runs.

of image, text and label tuples, we are interested in finding representations f φ : V → Z of image, parameterized by φ, and f θ : T → Z of text, parameterized by θ, in a common embedding space Z . Furthermore, GZSL problem is defined using the sets of seen Y tr and unseen Y ts classes, such that Y = Y tr ∪ Y ts and Y tr ∩Y ts = ∅. The training set will then only contain the seen classes, i.e.S tr = {(v n , t n , y n ) | v n ∈ V , t n ∈ T , y n ∈ Y tr }and the task is to build a classifier function g : Z × Z → Y . This is different from the ZSL scenario focusing on g : Z × Z → Y ts .

Table content: <table/>

Generalized zero-shot Top-1 classification accuracy.

Table content: <table><tr><td/><td>CUB</td><td/><td/><td>FLOWERS</td><td/></tr><tr><td>u</td><td>s</td><td>H</td><td>u</td><td>s</td><td>H</td></tr></table>

Zero-shot Top-1 classification accuracy.

Table content: <table/>

Generalized zero-shot Top-1 classification accuracy, ablation study.

Table content: <table><tr><td/><td/><td/><td/><td>CUB</td><td/><td/><td>FLOWERS</td><td/></tr><tr><td>α</td><td>λ</td><td>κ</td><td>u</td><td>s</td><td>H</td><td>u</td><td>s</td><td>H</td></tr><tr><td colspan="9">0.0 0.5 0.5 38.3 65.3 48.3 55.1 84.6 66.7</td></tr><tr><td colspan="9">0.0 0.5 0.0 39.3 57.5 46.7 54.0 78.1 63.8</td></tr><tr><td/><td colspan="8">0.5 0.0 53.8 49.6 51.6 71.7 67.2 69.4</td></tr><tr><td/><td colspan="8">0.0 0.5 47.4 36.6 41.3 51.5 60.5 55.6</td></tr><tr><td/><td colspan="8">1.0 0.5 53.9 53.8 53.8 69.5 73.9 71.6</td></tr><tr><td/><td colspan="8">0.5 0.5 59.3 52.6 55.8 73.0 73.6 73.3</td></tr></table>

Peer review (iclr2020):
The paper proposes to to use four different losses to train a joint text-image embedding space for zero-shot learning. The four losses consist of a classification loss given text descriptions, a classification loss given images, two contrastive losses given pairs of text and images. The paper also discusses how to balance seen and unseen classes, and it seems that, empirically, embeddings for seen classes are closer together while the embeddings for the unseen classes are further apart. A scaling factor that makes sure these distances are comparable for seen and unseen classes is introduced, and the paper gives an explanation for such a scaling. The final performance on the CUB and FLOWERS data set is impressive. I am giving a score of 3. The engineering effort in the paper is appreciated, but the novelty is lacking. The losses introduced in the paper, broadly classified as contrastive loss, has been around since (Frome et al., 2013). See also (Karpathy et al., 2014; Wang et al., 2016) and in particular (Xian et al., 2016). The subtle differences between all these losses are whether there is a margin, whether the loss is smooth (i.e., a softmax as opposed a max), and how the negative samples are selected. I am surprised that the loss function being used by many are considered a contribution in the paper. The property of metric scaling is certainly interesting, but it does not answer why metric scaling is needed in the first place. Overall, the novelty is limited. Below are some minor points and questions for the paper. ... anchor embeddings learned in one modality as prototypes ... --> the word *prototype* is used extensively in the rest of the paper. i know this is a common term in the zero-shot learning community, but it might be good to give a formal definition early or at least give an informal definition. Algorithm 1 --> what is the choice of d in the experiments? ... the probability of image v_i and text t_j to belong to the same object distance --> isn*t this the constrastive loss? what does it mean for the two to belong to the same object distance? equations (2) --> it seems that v_i and t_i are paired, but it is unclear from the text. equation (1) and (2) --> have you considered other (distributions of) negative samples? ... P{hat{y} in Y^{tr} | y_v in Y^{ts}} is significantly greater than P{hat{y} in Y^{tr} | y_v in Y^{tr}} --> what*s the definition of P{hat{y} | y_v}? the notation is confusing and hinders the understanding of the rest of the discussion. it*s probably easier to just talk about distances and use the nearest neighbor argument. Figure 2 --> what*s different between the 10 different runs? random seeds? Figure 3 --> why is the performance significantly lower when lambda=0? can this be related to how negative samples are selected? References: DeViSE: a deep visual-semantic embedding model A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, T. Mikolov Neurips, 2013 Deep fragment embeddings for bidirectional image sentence mapping A. Karpathy, A. Joulin, F. Li Neurips, 2014 Learning deep structure-preserving image-text embeddings L. Wang, Y. Li, S. Lazebnik CVPR, 2016 Latent embeddings for zero-shot classification Y. Xian, Z. Akata, G. Sharma, Q. Nguyen, M. Hein, B. Schiele CVPR, 2016