CLAREL: classification via retrieval loss for zero-shot learning
Authors: Boris N. Oreshkin
Negar Rostamzadeh
Pedro O. Pinheiro
Christopher Pal
Venue:  (2020)
Keywords: zero-shot learning
representation learning
fine-grained classification

Abstract: We address the problem of learning fine-grained cross-modal representations. We propose an instance-based deep metric learning approach in joint visual and textual space. The key novelty of this paper is that it shows that using per-image semantic supervision leads to substantial improvement in zero-shot performance over using class-only supervision. On top of that, we provide a probabilistic justification for a metric rescaling approach that solves a very common problem in the generalized zero-shot learning setting, i.e., classifying test images from unseen classes as one of the classes seen during training. We evaluate our approach on two fine-grained zero-shot learning datasets: CUB and FLOWERS. We find that on the generalized zero-shot classification task CLAREL consistently outperforms the existing approaches on both datasets.

Under review as a conference paper at ICLR 2020
CLAREL: CLASSIFICATION VIA RETRIEVAL LOSS FOR HERO-SHOT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We address the problem of learning fine-grained cross-modal representations. We propose an instance- based deep metric learning approach in joint visual and textual space. The key novelty of this paper is that it shows that using per-image semantic supervision leads to substantial improvement in zero-shot performance over using class-only supervision. On top of that, we provide a probabilistic justification for a metric rescaling approach that solves a very common problem in the generalized zero-shot learning setting, i.e., classifying test images from unseen classes as one of the classes seen during training. We evaluate our approach on two fine- grained zero-shot learning datasets: CUB and FLOWERS. We find that on the generalized zero-shot classification task CLAREL consistently outperforms the existing approaches on both datasets.
INTRODUCTION
Deep learning-basee based approaches have demonstrated superior flexibility and generalization capabilities in information processing on a wide variety of tasks, such as vision, speech and language (LeCun et al. 2015). However, it has been widely realized that the transfer of deep representations to real-worts applications is challenging due to the typical reliance on massive the low-labeled data regime, especially in the zero-shot (Wang et al., 2019) and the few-shot (Wang) & Yao 2019) setups, have recently received significant attention in the literature. In the problem of zero-shot learning (ZSL), the objective is recognize categories that have not been seen during the training (Larochelle et al. 20088. This is typically done by relying on anchor embeddings learned in one modality as prototypes and by associating query embedding from the other modality with the closest prototype. In the generalized ZSL (GZSL) case (Xian et al.] (2018c), the objective is more challenging as recognition performed in the joint space of seen and unseen categories. ZSL, as well as its generalized counterpart, provide a viable framework to learn cross-modal representations that are flexible and adaptive. For example, in this paradigm, the adaptation to a new classification task based on text/image representation space alignment could be easy as defining/appending/nnee mooiiiiiiiooiggggggggggggggggggg a set of text sentences to define classes of new classifiers. This is an especially relevant problem as machine learning is challenged with the long tail of classes, and the idea of learning from pairs of images and sentences, abundant on the web, looks like a natural solution. Therefore, in this paper we specifically target the fine-grained scenario of paired images and their respective text descriptions. The uniqueness of this scenario is in the fact that the co-occurance of image and text provides a rich source of information. The ways of leveraging this source have not been sufficiently explored in the context of GZSL. Although we focus exclusively on the GZSL recognition setup in this paper, we believe that the research in this direction has potential to enable zero-shot flexibility of high-level tasks such as segmentation or conditional image generation (Zhang et al., 2018). The contributions of this work can be characterized under the following two themes.
datasets. Learning in
wider array
Instance-basee training loss. Most prominent zero-shot learning approaches rely heavily on class- level modality alignment (Xian et al., (018c). We propose new composite loss function that balances rittace-bbsed pairwise image/text retrieval loss and the usual classifier loss. The retrieval loss term does not use class labels. We demonstrate that the class-level information is important, but in the fine-graines text/image pairing scenarios, most of the GZSL accuracy can be extracted from the instance- based retrieval loss. To the best of our knowledge, this type of training has not been used in the GZSL. literature. Its impressive performance opens up new promising research directions.

---

Under review as a conference paper at ICLR 2020
image classtter
softmax
test classifier
FC
lest retrioval sodtmax
BesNet
FC
SEa SE rerrieval
solmaa
small gray bind with black bead orange
CNNLSTM
Figure 1: The architecture and training diagram describing the proposed method. Each batch consists of randomly sampled instances, i.e. pairs of images and their corresponding texts. Images are embedded via ResNet and texts are embedded via a CNN/LSTM stack. Image and text features are projected via a fully connected layer into the same dimensional space. this space, distances between text and image features from different instances are computed. The negative distances are fed into softmax train both the image and the text retrieval tasks. The image retrieval task consists of retrieving the image corresponding to the given text of the same instance and the text retrieval task vice versa. addition to that, image and text embeddings are trained on auxiliary image and text classification tasks the class labels corresponding instances.
Metric space rescaling. Metric-based ZSL approaches rely on distances between prototypes and query embeddings during inference. They are known suffer from imbalanced performance seen and unseen classes (Liu et al.) 2018). Previous work proposed to use a heuristic trick, calibrated stacking (Chao et al., 2016)) or calibration (Das & Lee 2019), to solve the problem. We refer to this technique as metric rescaling in our work, and provide a sound probabilistic justification for it.
PROPOSED METHOD
In this paper. we specifically target the fine-gr une-grained visual description scenario, as defined by Reed al. (2016). this setting, the dataset consists of a number of images from a given set of classes and each image is accompanied by number of textual descriptions. The task is to learn joint representation space for images and texts that can be used for zero-shot recognition. the zero-shot multimodal representation learning problem can then be defined follows. Given image, text and label interested in finding representations fø V + Z of image, parameterized by d. and fo T + Z of text, parameterized by f, in a common embedding space Z. Furthermore, GZSL problem is defined using the sets of seen y and unseen y classes, such that ) " U and nt = 0. The training set will then only contain the seen classes, i.e. S = { [wn, fn. Dn Un E build classifier function g Z
instance of
training set = {(r,, la, yn) ) Un V, in T, V/ Vnn = 1...N) tuples,
V, fn E T. Vhe y*J and the task
This is different
from the ZSL scenario focusing on g Z x Z + y1s
To build g. most approaches joint representation learning rely class labeling train represen- tation. For example, all the methods reviewed by Xian et al. (2018c) require the access to class labels train time. We hypothesise that in the fine- ane-grained learning scenario, such as the one described by Reed et 2216)) lot of information can be extracted simply from pairwise image/text co- occurrences. The class labels really only become critically necessary when we define class prototypes, i.e. at zero-shot test time. Following this intuition, we define a composite loss function that relies both on the pairwise relationships and on class labels. The high-level description of the proposed framework is depicted in Figure [ The framework enables with the effects train-time availability of class labels on the quality of zero-shot representations. The framework is based projecting texts and images into common space and then learning representation based on a mixture of four loss functions: a pairwise text retrieval loss, pairwise image retrieval loss, text classifier loss and an image classifier loss (see Algorithm T).
among other things, experiment
2

---

Under review as a conference paper at ICLR 2020
Algorithm Loss calculation for single optimization iteration of the proposed method. N is the number instances in the training set S"r B is the number instances per batch, C' is the number of classes in the train set. RANDOMSAMPLE(S B) denotes set of B elements chosen uniformly at random from a set S, without replacement.
Input: Training set S = (11 1, U1).... ................................ (uN: IN. yv) À E [0.1], n [0,
Output: The loss [ 8) for randomly sampled training batch. I + RANDOMSAMPLEI
N), N)
D Select B instance indices for batch
JTc (T)( (Ic()) - 0,0
> Initialize classification losses
for i I do
Ze 2t, t- fo()), ft(
Embed images and texts Image classifier probabilities > Text classifier probabilities
pí softmax W 12u b1)
pr <- softmax W 1 + br)
(r) JTc ( B crossentropy (pr, yi)
Text classification loss
JIc(Ø) t (1c(0) + crossentropy
Image classification loss
end for
(o 0), (In(d, 0) t 0,0
Initialize retrieval losses
for i in I do
JTR(O, 0) JTR(O, ())
1
d( d(zu, Zt,) log Σ
expl d(z (zu zt,))
>> Text retrieval loss
B
jEI
JIR(0,0) + JIR(Ø, 1
dl d(zu, (112),21.)00) + log
Σ
exp( (-d(-d(( (ze Zos))
Image retrieval loss
JEI
end for
J[b, 0) AJrn(c ( 0) + (1 {)JIn(ő,0)
Add retrieval loss the total loss b. Add classification loss to the total loss
[áb - k: J(o - + Jic(6))
2.1 RETRIEVAL LOSS FUNCTION
Pairwise cross-modal loss function is based solely on the pairwise relationships between texts and images. We choose to use the metric learning approach to capture the relationship between images and texts. Now, suppose d is a metric d: Z Z + 1R, is an image and 7 {tyy } is collection of arbitrary texts sampled uniformly at random, of which text tú belongs to 24-- We propose the following model for the probability of image 2ú and text tj to belong to the same object instance:
( fo(t)))
Pi = jiu,, ty. Exper exp( exp(=d(f(())) (e
(1)
The learning is then based on the following cross-entropy loss defined on the batch of size B:
B
JTR(O, 0)
Σ
big log j/ 4, fl (o
B
s.j=1
where f a binary indicator of the true match (hi, = 1, if = and O otherwise). Note that the expression above has the interpretation of the text retrieval loss. attains its smallest value when for each image in the batch we manage to assign probability I to its respective text and 0 to all other texts. This can be further expanded as:
B
d(So(oe)) log JTn(Ø,0) = Σ
Σ Ey
exp(-
(3)
i=1
Exchanging the order of image and text in the probability model [ leads to the image retrieval loss, Jın(Ø, 8). The two losses mixed using parameter À [o, 1] as shown in Algorithm
The pairwise retrieval loss functions are responsible for the modality alignment. In addition to those, propose include, mentioned above, the usual image and text classifier losses. These losses are responsible for reducing the intraclass variability representations. The classifier losses are added to the retrieval losses using mixing parameter [0, 1] as shown in Algorithm
3

---

Under review as a conference paper at ICLR 2020
2.2 BALANCING ACCURACY ON THE SEEN AND UNSEEN CLASSES
Let us define class prototypes (b)) each based on the set of texts Ty belonging to class y. {p(y) [T.1 ZuET, (t.),) y E J)). In the context of GZSL, the nearest neighbor decision rule for given image v and its features Zu (v) has the following form:
ÿ arg min d(z.. p(y))
yEy
The most acute problem faced in this setup is the accuracy imbalance between seen and unseen classes. very representative case clearly outlining the imbalance problem is presented in Table 6 of (Xian) et all, 2018c). where accuracy the seen classes is always significantly greater than the accuracy on unseen ones. In order to measure and control the imbalance, three metrics are commonly used to assess the classification performance in the GZSL scenario: the Top-1 accuracy on the seen categories (s), the Top-1 accuracy on the unseen categories (u) and their harmonic mean, H = u u.s/ s/(u + s). The main metric to assess GZSL performance is then H, which quantifies both and
To formalize the problem, we first introduce yo. the true class label of image v. Mathematically. the main GZSL pain point is that ( y [yye E ) is significantly greater than [ E y** [y, )" In other words, the problem is that a given image is more likely to be confused with one of the seen classes if it belongs to an unseen class than vice versa. Our approach to solving the problem is based on the following probabilistic representation of the event space for the decision rule in Equation
T E )d yo E ) =P min d(z., p(y)) < min d(z., p(y)) | yo E )
(5)
Rephrasing, the most acute GZSL error happens when the prototype of one of the seen classes is closer to image embedding from unseen class than any of the prototypes of the unseen classes.
To rectify the situation we propose the following very direct solution to balance [ū E y"T yo E y** and (7 E )²* E )² We introduce a positive scalar a R+ and scale all the distances corresponding to the seen prototypes by + a. This gives rise to the following scaled distance da:
[ (1 a)d(z p(v)), if e y"r (((z..(()))
dz p(y)
(6)
otherwise
The misclassification between unseen as seen classes for the classifier Da. based on (6) is then:
P(U. E [1 E y =P (a + a) min ((Zu,(()) < min ((z.))()))lu E yo (7) weyer
weyi,
and it has the following property: for any 0 S an S 0z P(ū., E ) E yo3 > P(mnz e 2* Mo }x*}, Lee. Plua E "" yts} is monotone non non non-incregsing function of a and can reduce it by increasing a (please refer to Appendix A for proof). Consider now Pf e "" yo E )rr), which is probability that we classify an image v from one of the seen classes as still one of the seen classes. Using exactly the same chain of arguments as in Appendix A it is show that the probability is non-increasing function of a. Hence the probability P(O.. E yod [y E y" = =1 - ( E y 13v E } is a non-decreasing function of a. Therefore, as a increases, we expect more classification errors in classifying images from seen classes, because some of them will be classified as one of the unseen classes.
straightforward
To sum up, given the arguments presented above we expect that by varying a >> 0 we can balance the error rate P( E yd yr E y of leaking the unseen class images into seen class classification decision and the error rate (T. E )² E )²r) of leaking the seen class images into unseen class classification decision. This is possible as we just showed above that (ū. E y yo E ) a non-increasing function of a, while PP E y 1y E ydrr is a non-decreasing one. It is also important to emphasize that is applied only to distances between the query embedding and the prototypes of seen classes and is constant over seen classes. Therefore, the application of a does not at all affect the classification error rates either within "er or within yts Varying a balances exclusively the classification errors arising from transitions between seen and unseen class labels. We study the empirical aspects of balancing a in Section 4.4
4

---

Under review as a conference paper at ICLR 2020
Table Generalized zero-shot Top-1 classification accuracy.
CUB
FLOWERS
S
H
u
H
CADA-VAE (Schönfeld al.
2019
n/a
n/a
53.4
n/a
n/a
n/a
I-CLSWGAN (Xian al. 20 8d)
50.3
58.3
54.0
59.0
73.8
65.6
E-VÆGANND2 (Xian el al.
2019
48,4
60.1
53.6
56.8
74.9
64.6
cycle-UUWGAN (Felix et
2018
47.9
59.3
53.0
61.6
69.2
65.2
COSMO+f COSMO+f-CLSWGAN Atzmon & Chechik 2019)
n/a
n/a
n/a
59.6
81.4
68.8
CLAREL (Ours)
59.3
52.6
55.8
73.0
73.6
73.3
Table 2: Zero-shot Top-1 classification accuracy.
CUB
FLOWERS
CADA-VAE (Schönfeld al. 2019
n/a
n/a
F-CLSWGAN (Xian et al. 2018d)
57.3
67.2
f-VAEGAN-DA (Xian al. 2019
61.0
67.7
cycle-(UWGGAN (Felix et al. CLAREL (Ours)
2018)
58.6
70.3
66.7
76.8
RELATED WORK
ZSL approaches aim at recognizing objects belonging to classes unseen during training Larochelle et al. 2008 Palatucci et al., 2009). This has been extended to the GZSL framework in which the
decision space consists of both seen and unseen classes (Socher al. 2013 Xian et al.) 2018ch. The
classical zero-shot approaches build a joint -isuanesemanti space, relying
linear cross-modal
compatibility function (e.g. dot-product between query embedding and semantic prototypes or
variation of hinge loss) (Frome et al. 2013 Akata et al. [015: 2016: Reed et al. (016). Non-linear
variants of the compatibility has also been explored (Xian et al. 2016)] SScher et al. (003)) Extending
previously proposed cross-modal transfer approaches based auto-encoders (Hubert Tsai et al. 2017) and cross-domain learning (Gretton et al. 2007), more recent line of work (Schönfeld et al.
2019 Xian al. 2018d| 2019: Felix et al. 2018]] Verma al., 2018) relies combining these approaches and their variations with dataset augmentation tools such as GAN (Goodfellow et al. 2014) and VAE (Kingma & Welling 2014). It is argued that the of those tools helps to resolve one of the prominent problems in GZSL scenario: classifying images from unseen classes as one of the seen classes. There exist approaches that try to tackle this same problem via temperature calibration (Liu et al. (018)) originally proposed by Hinton et al. (2015). Chao et al. (2016); Das & Lee (2019) proposed an approach seen/unseen accuracy balancing that very similar ours, based on heuristic arguments. We extend this line of work here by providing a probabilistic justification for the balaneing effect observed when applying metric rescaling. Atzmmnn & Chechik (2019) propose more sophisticated way to deal with seen/unseen imbalance via adaptive confidence smoothing and gating, yet as authors note much simpler to train than the existing GAN-base zero-shot approaches. In this work, introduce arguably the simplest zero-shot representation training approach of all, and we demonstrate that when the image level text information available, achieves the state-of-the-ar results GZSL task well-known datasets.
EXPERIMENTAL RESULTS
4.1 DATASETS
We focus on learning embeddings for fine-grained visual descriptions and test them in ZSL/GZSL scenario. To test the quality of trained embeddings we focus on datasets that provide paired images and text descriptions, such Caltech-UCSI BaacchhUUS--Biiii Birds (CUB) (Welinder et al. (0100) and Oxford Flowers (FLOWERS) (Nilsback Zisserman 2008), that were augmented with textual descriptions by Reed all 2016). We use the GZSL. splits proposed by Xian et al. (2018c). The attribute-base datasets,
5

---

Under review as a conference paper at ICLR 2020
60
58
56
64
54
62
52
60
0.00
0.01
0.02
0.03
0.04
0.00
0.01
0.02
0.03
0.04
(a) CUB
(b) FLOWERS
Figure 2: Harmonic mean Top-1 accuracy on seen and unseen, H. against the value of a on the validation set. The curves represent the mean and 95% confidence intervals over 10 optimization runs. Results are stable over different runs. H exhibits a distinct inverted U-shape w.r.t. a.
such as SUN (Patterson et al. 2014) and AWA (ammpert et al.) 2014) do not contain this information and do not have notion of entity of class in them. They are out of the scope of the current paper.
4.2 ARCHITECTURE AND TRAINING DETAILS
Below, we provide more detailed description of parameters used to build and train the architecture depicted in Figure T We use exactly the same hyperparameter settings for CUB and FLOWERS. The text feature extractor is built by cascading two ResNet blocks, followed by BiLSTM. Each ResNet block has 3 convolutional/batch norm layers. The number of filters the ResNet blocks is 128 and 256, BiLSTM has 512 filters for forward and backward branches (1024 total). All variables the convolutional stack (including the batch normalization parameters 7 and 3) are L22penalized with weight 0.001. The image feature extractor is ResNet-101 101 with fixed weights pretrained on the split ImageNet proposed by Xian et al. (2018c). In this work we use precomputed image features, available in (Xian et al. 2018a) for CUB and in (Xian et al., 2018b) for FLOWERS. Image and text features are projected in the common embedding space of size 1024 with FC layers with no non-linearity. They are preceded with dropout of 0.25. The trainable components of the model trained for I 150k batches of size 32 using SGD with initial learning rate of 0.1 that is annealed by factor of 10 every 50k batches. For each batch, we sample 32 instances, each instance includes vector of precomputed ResNet-101 features and 10 text descriptions corresponding to it. according the original dataset definition Reed et al. (2016). All 10 text descriptions are processed via the CNN/LSTM stack and the resulting embeddings are average pooled to create a vector representation of length 1024.
4.3 KEY RESULTS
Our key empirical results compared in Table and in Table 2 against the latest state of the art. Our results are based on the settings of À = 0.5, α = 0.5 and a selected on the validation sets of CUB and FLOWERS datasets. Please refer to Section 4.5 for the analysis of stability with respect to the choices of À and к and Sections 2.2 and 4.4 for more details on the selection of a. The combination of the proposed training method and the rebalancing of the metric space results in the state-of-the-art performance. Most of the current methods rely on the dataset augmentation techniques based on GANs, VAES or combinations thereof. Those are clearly complementary w.r.t. our method and their addition to the training procedure is likely to further boost the performance of our proposed approach. However, this outside of the scope of the current work. Moreover, the proposed method is ttate-of-the-an on FLOWERS even when compared against (Atzmon & Chechik 2019) that uses both more sophisticated GAN based embedding learning approach and more sophisticated seen/unseen error rate balancing based on COSMO. It is important to note that Atzmmnn Chechik (019) did not report the sentence level results on CUB. Yet. when applied attributes together with f-CLSWGAN (Xian al. 2018d) COSMO resulted in 0.8% performance drop and when applied with LAGO (Azzmon & Chechik 2018 it achieved 0.5 % improvement over
the attribute based state of the art.

---

Under review as a conference paper at ICLR 2020
80
60
Accurapy
55
70
550
65
45
55
48.0
0.2
0.4
0.6
98.0
0.6
0.8
(a) CUB
(b) FLOWERS
Figure 3: Harmonic mean Top-1 accuracy seen and unseen, H. against X, the relative weight of image and text retrieval loss terms. À corresponds to the case image retrieval loss having weight and text retrieval loss having weight 0. Mean over 10 optimization runs.
80
Top-1 Accurary
55
70
60
$8.0
0.4
%..0
(a) CUB
(b) FLOWERS
Figure 4: The plot of the harmonic mean 1 accuracy on seen and unseen, H. against K, the relative weight of the retrieval and the classification loss terms. к corresponds to the case of classification loss having weight O. The curves represent the mean over 10 optimization runs.
4.4 ON THE SEEN/UNSEEN ACCURACY BALANCING
Figure demonstrates the plot of harmonic mean Top-1 accuracy, H, against the value of the validation sets of CUB and FLOWERS datasets. The validation set is constructed by further splitting the train set both datasets. For example, CUB has train set of 5875 images from 100 seen classes and a validation set of 2946 images from 50 unseen classes. We further divide the train set into 4700 validation images (4700 + 1175 5875) and all the 2946 images from 50 classes as the unseen validation set. Once the value of a: is determined we train the representation on the full train+val subset and report results on the test split (the usual practice in GZSL), We confirm on the validation set that H exhibits an inverted U-shape behavior as a function of a, which was theoretically predicted in Section 2.2 Therefore, can be selected on the validation set and then applied to re-scale the metric space to balance the accuracy seen and unseen classes during test time as described in Section 2.2
train images from 100 seen classes, 1175
4.5 ABLATION STUDY
Figure presents the results of the ablation study the importance image and text retrieval losses. We see that all of the Top-1 accuracies (H, S, u) are stable in the range of X E [0.2, 0.9], when both losses have tangible weight. Removing either text or image retrieval losses (setting to 0 or respectively) leads to performance drop in both cases. Removing the text retrieval loss (case X = 0) results in the most significant drop. This due to the fact that text retrieval task more tightly related to the GZSL task. At the batch level, retrieving the right text given image equivalent to identifying the correct class encoded by a text prototype during ZSL inference step. The image retrieval task is not directly related to solving the ZSL problem and yet it does yield positive regularizing effect on both CUB and FLOWERS.
Figure 4 shows the results of the ablation study of the interplay between the retrieval loss and the classification loss. We observe, just as in the case with A, that there exists a reasonably flat and stable range of [0.2, 0.6). The range for is bit smaller. k results in the catastrophic performance

---

Under review as a conference paper at ICLR 2020
Table Generalized zero-shot Top-1 classification accuracy, ablation study.
CUB
FLOWERS
a
X
K
H
u
H
0.0
0.5
0.5
38.3
65.3
48.3
55.1
84.6
66.7
0.0
0.5
0.0
39.3
57.5
46.7
54.0
78.1
63.8
V
0.5
0.0
53.8
49.6
51.6
71.7
67.2
69.4
V
0.0
0.5
47.4
36.6
41.3
51.5
60.5
55.6
V
1.0
0.5
53.9
53.8
53.8
69.5
73.9
71.6
0.5
0.5
59.3
52.6
55.8
73.0
73.6
73.3
drop: the classification losses by themselves do not enforce any modality alignment (please refer to clearly demonstrating this).
Fig. T and Algorithm |
Table 3 studies the effects of different loss terms on the harmonic mean Top-1 accuracy H. The best result is achieved when all loss terms are active and when the metric space rescaling is (the case of A = 0.5, K = 0.5 and a is checked, the last line in the table). Comparing this with the case when there is no metric space rescaling (first line with a 0), we see that the rescaling helps to decrease the gap between seen and unseen classification accuracy. For CUB, the discrepancy reduction is from around 30% around 6%, for FLOWERS it is from around 30% to around 1%. We would like to stress that we only use images and texts from the training set to achieve that. Going to the second line in the table (the image/text classification loss inactive, A = 0) and comparing it to the first one, assess the effect of the image/text classification loss. It barely affects the performance on unseen set, but it significantly boosts the classification accuracy on the seen set (around 8% on both datasets). This is logical: adding classifier loss results in better classifier of the test images from the seen classes. This alone does not make it a better GZSL classifier, however. Only when applied together with metric space rescaling, this results in the performance boost (please refer to lines I and 6 in Table Our interpretation is that the addition of the image/text classifier loss helps to reduce the intraclat lass variability in embeddings and provides for tighter clustering. However, this also leads to overfit on the classification task. This is accounted for by metric rescaling that enables the learnings from the image/text classification task be transferred effectively into the GZSL task.
The comparison of the last four rows of Table leads us to believe that all the proposed loss terms outlined Fig. T and Algorithm [ are important for achieving the state-of-the-an performance. Excluding any one of them (corresponding to the extreme values X 0, X 1, 0) leads to performance deterioration. Finally, an interesting observation can be made by comparing line 3 of Table 3 with performance of algorithms in Table D In this case our algorithm does not use any class labels and relies on training using exclusively the retrieval losses that can be calculated only based on the pairwise relationships between texts and images. We type of supervision alone already results in a very high-qualig representation. The representation competitive against the latest GAN/VAE based approaches on CUB and is state-of-the-art FLOWERS. This opens up new exploration avenues showing that in the case when very fine-grained modality outputs available (image and text description pairs being very prominent example), the high-quality representations may be learned without relying on manually supplied class labels.
that using this
5 CONCLUSIONS
We propose and empirically validate two improvements to the process of learning fine-grained cross- modal representations. First, we confirm the hypothesis that in the context of paired images and texts, a deep metric learning approach can be driven by an instance-based retrieval loss resulting in competitive generalized zero shot classification results. Combined with an additional class label based image/text crossentropy term this results in state-of-the-a performance on two well known datasets, CUB and FLOWERS. This is an interesting result demonstrating that high-quality deep representations can be trained relying largely on pairwise relationships between modalities. On top of that, we propose a solution to one of the prominent problems in GZSL: classifying instances of unseen classes seen ones. We mathematically analyze and empirically validate the method of adjusting a single scalar that transcends in its effectiveness advanced dataset augmentation and training approaches based GANs and VAEs.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embeddings for fine-grained image classification. In CVPR, 2015.
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for image classification. TPAMI, 2016.
Yuval Atzmon and Gal Chechik. Probabilistic AND-OR attribute grouping for zero-shot learning, In UAI, 2018.
Yuval Atzmon and Gal Chechik. Adaptive confidence smoothing for generalized zero-shot learning. In CVPR, 2019.
Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis of generalized zero-shot learning for object recognition the wild. In ECCV (2), pp. 52-68, 2016.
Debasmit Das and Lee. Zero-shot image recognition using relational matching, adaptation and calibration. In International Joint Conference on Neural Networks, 2019.
Rafael Felix, Vijay Kumar B G, Ian Reid, and Gustavo Careeir.. Multi-modal cycle-consistent
generalized zero-shot learning. In ECCV, 2018.
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Mare Aurelio Ranzato, and Tomas Mikolov. DeViSE: A deep visual-semantic embedding model. In NIPS, pp. 2121-2129, 2013.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pp. 2672-2680, 2014.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kernel method for the two-sample-rrobeem. In NIPS, 2007.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NIPS Workshop, 2015.
Yao-Hung Hubert Tsai, Liang-Kand Huang, and Ruslan Salakhutdinov. Learning robust visual- semantic embeddings. In CVPR, 2017.
Diederik P. Kingma and Max Welling. Auto-enes Auto-encoding variational bayes. In ICLR, 2014.
C. H. Lampert, H. Nickisch, and S. Harmeling. Attribute-based classification for zero-shot visual object categorization. TPAMI, 2014.
Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks. AAAI, 2008.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 2015.
Shichen Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Generalized zero-shot learning with deep calibration network. In NIPS, 2018.
M. Nilsback and A. Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference Computer Vision, Graphics Image Processing, 2008.
Mark Palatucci, Dean Pomerleau, Geoffrey E Hinton, and Tom M Mitchell. Zero-shot learning with semantic output codes. NIPS, 2009.
Genevieve Patterson, Chen Xu, Hang Su, and James Hays. The sun attribute database: Beyond categories for deeper scene understanding. IJCV, 2014.
Scott E. Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. Learning deep representations of fine-grained visual descriptions. In CVPR, 2016.
Edgar Schönfeld, Sayna Ebrahimi, Samarth Sinha. Trevor Darrell, and Zeynep Akata. Generalized zero-and few-shot learning via aligned variational autoencoders. CVPR, 2019.

---

Under review as a conference paper at ICLR 2020
Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. In NIPS, 2013.
Vinay Kumar Verma, Gundeep Arora, Ashish Mishra, and Piyush Rai. Generalized zero-shot learning via synthesized examples. In CVPR, 2018.
Wei Wang, Vincent W. Zheng, Han Yu, and Chunyan Miao. A survey of zero-shot learning: Settings, methods, and applications. ACM Trans. Intell. Syst. Technol., 2019.
Yaqing Wang and Quanming Yao. Few-shot learning: A survey. In arXiv, 2019.
Welinder, S. Branson, T. Mita, C. Wah, Schroff, S. Belongie, and P. Perona. Caltech-UCSD
Birds 200. Technical report, California Institute of Technology, 2010.
Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh N. Nguyen, Matthias Hein, and Bernt Schiele. Latent embeddings for zero-shot classification. In CVPR, 2016.
Yonggin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. Pretrained CUB features, 2018a. httphhttt////taaaeee
d2 mp: -iw.attss.....mmii.........mmm wpg httpp.....e/iii/illll///
Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. Pretrained FLOWERS
features, 2018b. URL http::tttt:/ttat wttp:/etttt..d..mmmi...... www.iinn..........mmmmmm.mm.. de hwtpp//ewwww.e//a/eeeeeeeee 8xi
Zip
Yongqin Xian, H. Christoph Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning: A comprehensive evaluation of the good, the bad and the ugly. TPAMI, 2018c.
Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In CVPR, 2018d.
Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. f-vaegan-d2: A feature generating framework for any-shot learning. CVPR, 2019.
H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas. StackGAN++: Realistic image synthesis with stacked generative ad adversarial networks. PAMI, 2018.
A THE ANALYSIS OF ERROR RATES
We show that PV E ye E ydg > P(U. E y E yos Let define der = minyeyer d(zu, p(v)) and des = mingey d(z., p(y)), then Equation can be rewritten as: Pfya E e"'M ly E e = {(1 + a),er le E
Let us consider the probability of event d, < 8is and decompose it as follows:
P ( < dealyo E y P( + a)o,r < a1)+)ooll E y(r]
= P ((1 aader < di U s (1 + a) < (1 + a) E
= -P ((1+a))6g < duab, E y²d} + P f. < + a1 < (1 +1+ (1+a) d E yoo
- PP [1 a)o., < des n di. < (1 + a) < a1+ E y
=P aa)+r < 6vel E ) +P (0 S (1+ < a1 E )²)
The transitions are based on the relationship between probabilities of arbitrary events A and B,
P(AUB] = P[A} + PPBB PPAA B}, and in our case P(A n B} = O. This implies that:
dis
P(0a E yd yd+} P( ydr ly E yd -
s dr Scalye e yds
(1+a)
yd ly E )d* < PP
We have just shown that for non-negative the probability of misclassifying image from an unseen class as one of the seen classes is smaller for the decision rule Da than for the original decision rule ÿ. In fact, we can make stronger claim. Since &ts and der are non-negative, it is increases, and hence probability that
clear that the length of interval [5t./(1 + a), des) increases non-decraasing with increasing a. Thus we have for any 0 S an S oz. P(D., E y (v. E yr 2 P(ūe, E yx lBb E ) iee. P(ū. E y yo E yts) is monotone non-increasing function of and we can reduce it by increasing a.
d,, falls in this interval
10


Reviews:
[{"review_id": "rkxBq85pYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents two main contributions: first, a simple retrieval-based objective for learning joint text and image representations, and second, a metric-scaling term that improves performance on unseen classes in the generalized zero-shot (GZSL) setting. They evaluate on two datasets, CUB and FLOWERS, consisting of images paired with text descriptions, and show that their relatively simple technique outperforms complex GAN and VAE-based models.\n\nWeak accept, see considerations below. The results seem solid, and the architecture is quite simple compared to other models on this task. However, there are some gaps in the analysis, and the paper would benefit from a more careful comparison to the literature.\n\nThe results are strong, and it’s encouraging to see a simple approach like this outperform more complex models. However, it’s not clear how novel the retrieval model is; as implemented, it’s similar to a sampled softmax or noise contrastive estimation - the paper would benefit from a more careful comparison of their approach to established alternatives, and a discussion of why this approach should learn better representations than a VAE or GAN. It’s also not clear how novel the metric scaling technique is. It undeniably works well, but the use seems remarkably similar to (Das & Lee 2019, Section II.D).\n\nThe observation (Section 4.5) that the retrieval-only model performs well without any class labels is interesting. How does this compare to other unsupervised approaches? It also appears that the version with the metric scaling here still performs better - is this because the retrieval model is trained only on examples from the “seen” classes? Does this gap go away if (unlabeled) examples from the unseen classes are available at training time?\n\nA few other notes:\n* Section 4.5: “Our interpretation is that the addition of the image/text classifier loss helps to reduce the intraclass variability in embeddings and provides for tighter clustering.” - could you test this directly, or provide some visualization, such as t-SNE plots?\n* The image representations are initialized from an ImageNet model, but text is trained from scratch. Why not use a pretrained text encoder like BERT?\n* Table 3: it would help to have text labels for these rows, or a reminder in the caption of what \\lambda = 0 and \\kappa = 0 mean (otherwise, one needs to flip/scroll back to page 4)\n* Figure 2: what are the actual values of u and s that contribute to this plot?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rkx6ViXCKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles zero-shot and generalised zero-shot learning by using the per-image semantic information. An instance-based loss is introduced to align images and their corresponding text in the same embedding space. To solve the extreme imbalanced issue of generalized zero-shot learning, the authors propose to scale the prediction scores of seen classes by a constant factor. They demonstrate technic contributions on CUB and Flowers datasets and the results achieve the state-of-the-art.\nThis paper should be rejected because (1) motivation is not well justified. This paper fails to convince me that it is practical to use per-image semantic information. Zero-shot learning aims to reduce the cost of annotation, but the per-image semantic information used in this paper conversely increases the annotation efforts. More specifically, this paper directly applies the per-image text descriptions proposed by Reed et al. '16, in which they annotated each image of CUB and Flowers by 10 different sentences. Due to the lack of such expensive annotations, this paper can be only evaluated on CUB and Flowers datasets,  results on other popular zero-shot learning datasets e.g., AWA, SUN, ImageNet, are essentially missing.\n(2) novelty is limited. Using per-image semantic information is not new in zero-shot learning at all. Reed et al. '16 studied this problem and showed that per-image text-description can surpass per-image attributes in zero-shot learning. The loss function of this paper is also similar to  Reed et al. '16, which used the max-margin loss to align image and text pairs, v.s. the cross-entropy loss of this paper. Claiming that they are the first to use per-image semantic information in GZSL is not convincing because GZSL was not introduced at the time of Reed et al. '16. Metric scaling is not novel either. It is, in fact, equivalent to the calibration technic proposed by Chao et al. '16. The theory of metric scaling is redundant in my point of view because it is obvious that rescaling decreases the seen class scores. \n(3) experiments are insufficient to support the contribution. Their experiments are not comparing apple to apple. Specifically, the competing methods are using per-class semantic information while their approach uses per-image semantic information which is unfair because per-image semantic information includes much more supervision. The authors should have compared the results of those methods trained with per-image semantic information. I would expect all the approach will benefit from per-image side information. \n\n\n\nPost-rebuttal comments:  In the author responses, the authors have written long stories to fight against my reviews. But unfortunately,  a large part of the responses is not addressing my concerns. The authors repeatedly argue that their main contribution is to show that image-level supervision is an effective way to tackle (generalized) zero-shot learning. However, this contribution is not significant because Reed et al. '16 has demonstrated this point in the scenario of zero-shot learning. Extending this idea to generalized zero-shot learning is not a sufficient contribution for ICLR.  \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gFTZm0YB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to to use four different losses to train a joint text-image embedding space for zero-shot learning. The four losses consist of a classification loss given text descriptions, a classification loss given images, two contrastive losses given pairs of text and images. The paper also discusses how to balance seen and unseen classes, and it seems that, empirically, embeddings for seen classes are closer together while the embeddings for the unseen classes are further apart. A scaling factor that makes sure these distances are comparable for seen and unseen classes is introduced, and the paper gives an explanation for such a scaling. The final performance on the CUB and FLOWERS data set is impressive.\n\nI am giving a score of 3. The engineering effort in the paper is appreciated, but the novelty is lacking. The losses introduced in the paper, broadly classified as contrastive loss, has been around since (Frome et al., 2013). See also (Karpathy et al., 2014; Wang et al., 2016) and in particular (Xian et al., 2016). The subtle differences between all these losses are whether there is a margin, whether the loss is smooth (i.e., a softmax as opposed a max), and how the negative samples are selected. I am surprised that the loss function being used by many are considered a contribution in the paper. The property of metric scaling is certainly interesting, but it does not answer why metric scaling is needed in the first place. Overall, the novelty is limited.\n\nBelow are some minor points and questions for the paper.\n\n... anchor embeddings learned in one modality as prototypes ...\n--> the word \"prototype\" is used extensively in the rest of the paper. i know this is a common term in the zero-shot learning community, but it might be good to give a formal definition early or at least give an informal definition.\n\nAlgorithm 1\n--> what is the choice of d in the experiments?\n\n... the probability of image v_i and text t_j to belong to the same object distance\n--> isn't this the constrastive loss? what does it mean for the two to belong to the same object distance?\n\nequations (2)\n--> it seems that v_i and t_i are paired, but it is unclear from the text.\n\nequation (1) and (2)\n--> have you considered other (distributions of) negative samples?\n\n... P{\\hat{y} \\in Y^{tr} | y_v \\in Y^{ts}} is significantly greater than P{\\hat{y} \\in Y^{tr} | y_v \\in Y^{tr}}\n--> what's the definition of P{\\hat{y} | y_v}? the notation is confusing and hinders the understanding of the rest of the discussion. it's probably easier to just talk about distances and use the nearest neighbor argument.\n\nFigure 2\n--> what's different between the 10 different runs? random seeds?\n\nFigure 3\n--> why is the performance significantly lower when lambda=0? can this be related to how negative samples are selected?\n\nReferences:\n\nDeViSE: a deep visual-semantic embedding model\nA. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, T. Mikolov\nNeurips, 2013\n\nDeep fragment embeddings for bidirectional image sentence mapping\nA. Karpathy, A. Joulin, F. Li\nNeurips, 2014\n\nLearning deep structure-preserving image-text embeddings\nL. Wang, Y. Li, S. Lazebnik\nCVPR, 2016\n\nLatent embeddings for zero-shot classification\nY. Xian, Z. Akata, G. Sharma, Q. Nguyen, M. Hein, B. Schiele\nCVPR, 2016\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper demonstrates that per-image semantic supervision, as opposed to class-only supervision, can benefit zero-shot learning performance in certain contexts.  Evaluations are conducted using CUB and FLOWERS fine-grained zero-shot data sets.  In terms of evaluation, the paper received mixed final scores (two reject, one accept).

During the rebuttal period, both reject reviewers considered the author responses, but in the end did not find the counterarguments sufficiently convincing.  For example, one reviewer maintained that in its present form, the paper appeared too shallow without additional experiments and analyses to justify the suitability of the contrastive loss used for obtaining embeddings applied to zero-shot learning.  Another continued to believe post-rebuttal that reference Reed et al., (2016) undercut the novelty of the proposed approach.

And consistent with these sentiments, even the reviewer who voted for acceptance alluded to the limited novelty of the proposed approach; however, the author response merely states that a future revision will clarify the novelty.  But this then requires another round of reviewing to determine whether the contribution is sufficiently new, especially given that all reviewers raised this criticism in one way or another.  Furthermore, the rebuttal also mentions the inclusion of some additional experiments, but again, we don't know how these will turn out.

Based on these considerations then, the AC did not see sufficient justification for accepting a paper with aggregate scores that are otherwise well below the norm for successful ICLR submissions.
Decision: Reject