Year: 2019

Abstract:
We address the problem of learning fine-grained cross-modal representations. We propose an instance-based deep metric learning approach in joint visual and textual space. The key novelty of this paper is that it shows that using per-image semantic supervision leads to substantial improvement in zero-shot performance over using class-only supervision. On top of that, we provide a probabilistic justification for a metric rescaling approach that solves a very common problem in the generalized zero-shot learning setting, i.e., classifying test images from unseen classes as one of the classes seen during training. We evaluate our approach on two finegrained zero-shot learning datasets: CUB and FLOWERS. We find that on the generalized zero-shot classification task CLAREL consistently outperforms the existing approaches on both datasets.

heading: 1 INTRODUCTION
text: Deep learning-based approaches have demonstrated superior flexibility and generalization capabilities in information processing on a wide variety of tasks, such as vision, speech and language (LeCun et al., 2015). However, it has been widely realized that the transfer of deep representations to real-world applications is challenging due to the typical reliance on massive hand-labeled datasets. Learning in the low-labeled data regime, especially in the zero-shot (Wang et al., 2019) and the few-shot (Wang & Yao, 2019) setups, have recently received significant attention in the literature. In the problem of zero-shot learning (ZSL), the objective is to recognize categories that have not been seen during the training (Larochelle et al., 2008). This is typically done by relying on anchor embeddings learned in one modality as prototypes and by associating a query embedding from the other modality with the closest prototype. In the generalized ZSL (GZSL) case (Xian et al., 2018c), the objective is more challenging as recognition is performed in the joint space of seen and unseen categories. ZSL, as well as its generalized counterpart, provide a viable framework to learn cross-modal representations that are flexible and adaptive. For example, in this paradigm, the adaptation to a new classification task based on text/image representation space alignment could be as easy as defining/appending/modifying a set of text sentences to define classes of new classifiers. This is an especially relevant problem as machine learning is challenged with the long tail of classes, and the idea of learning from pairs of images and sentences, abundant on the web, looks like a natural solution. Therefore, in this paper we specifically target the fine-grained scenario of paired images and their respective text descriptions. The uniqueness of this scenario is in the fact that the co-occurance of image and text provides a rich source of information. The ways of leveraging this source have not been sufficiently explored in the context of GZSL. Although we focus exclusively on the GZSL recognition setup in this paper, we believe that the research in this direction has potential to enable zero-shot flexibility in a wider array of high-level tasks such as segmentation or conditional image generation (Zhang et al., 2018). The contributions of this work can be characterized under the following two themes.
Instance-based training loss. Most prominent zero-shot learning approaches rely heavily on classlevel modality alignment (Xian et al., 2018c). We propose a new composite loss function that balances instance-based pairwise image/text retrieval loss and the usual classifier loss. The retrieval loss term does not use class labels. We demonstrate that the class-level information is important, but in the fine-grained text/image pairing scenarios, most of the GZSL accuracy can be extracted from the instance-based retrieval loss. To the best of our knowledge, this type of training has not been used in the GZSL literature. Its impressive performance opens up new promising research directions.
Metric space rescaling. Metric-based ZSL approaches rely on distances between prototypes and query embeddings during inference. They are known to suffer from imbalanced performance on seen and unseen classes (Liu et al., 2018). Previous work proposed to use a heuristic trick, calibrated stacking (Chao et al., 2016) or calibration (Das & Lee, 2019), to solve the problem. We refer to this technique as metric rescaling in our work, and provide a sound probabilistic justification for it.

heading: 2 PROPOSED METHOD
text: In this paper, we specifically target the fine-grained visual description scenario, as defined by Reed et al. (2016). In this setting, the dataset consists of a number of images from a given set of classes and each image is accompanied by a number of textual descriptions. The task is to learn a joint representation space for images and texts that can be used for zero-shot recognition. An instance of the zero-shot multimodal representation learning problem can then be defined as follows. Given a training set S = {(vn, tn, yn) | vn ∈ V , tn ∈ T , yn ∈ Y , n = 1 . . . N} of image, text and label tuples, we are interested in finding representations fφ : V → Z of image, parameterized by φ, and fθ : T → Z of text, parameterized by θ, in a common embedding space Z . Furthermore, GZSL problem is defined using the sets of seen Ytr and unseen Yts classes, such that Y = Ytr ∪ Yts and Ytr∩Yts = ∅. The training set will then only contain the seen classes, i.e. Str = {(vn, tn, yn) | vn ∈ V , tn ∈ T , yn ∈ Ytr} and the task is to build a classifier function g : Z ×Z → Y . This is different from the ZSL scenario focusing on g : Z × Z → Yts. To build g, most approaches to joint representation learning rely on class labeling to train a representation. For example, all the methods reviewed by Xian et al. (2018c) require the access to class labels at train time. We hypothesise that in the fine-grained learning scenario, such as the one described by Reed et al. (2016), a lot of information can be extracted simply from pairwise image/text cooccurrences. The class labels really only become critically necessary when we define class prototypes, i.e. at zero-shot test time. Following this intuition, we define a composite loss function that relies both on the pairwise relationships and on the class labels. The high-level description of the proposed framework is depicted in Figure 1. The framework enables us, among other things, to experiment with the effects of train-time availability of class labels on the quality of zero-shot representations. The framework is based on projecting texts and images into a common space and then learning a representation based on a mixture of four loss functions: a pairwise text retrieval loss, a pairwise image retrieval loss, a text classifier loss and an image classifier loss (see Algorithm 1).
Algorithm 1 Loss calculation for a single optimization iteration of the proposed method. N is the number of instances in the training set Str, B is the number of instances per batch, C is the number of classes in the train set. RANDOMSAMPLE(S, B) denotes a set of B elements chosen uniformly at random from a set S, without replacement. Input: Training set Str = {(v1, t1, y1), . . . , (vN , tN , yN )}, λ ∈ [0, 1], κ ∈ [0, 1]. Output: The loss J(φ, θ) for a randomly sampled training batch. I ← RANDOMSAMPLE({1, . . . , N}, B) . Select B instance indices for batch JTC(θ), JIC(φ)← 0, 0 . Initialize classification losses for i in I do zvi , zti ← fφ(vi), fθ(ti) . Embed images and texts pI ← softmax(WIzvi + bI) . Image classifier probabilities pT ← softmax(WT zti + bT ) . Text classifier probabilities JTC(θ)← JTC(θ) + 1B crossentropy(pT , yi) . Text classification loss JIC(φ)← JIC(φ) + 1B crossentropy(pI , yi) . Image classification loss
end for JTR(φ, θ), JIR(φ, θ)← 0, 0 . Initialize retrieval losses for i in I do
JTR(φ, θ)← JTR(φ, θ) + 1
B d(zvi , zti) + log∑ j∈I exp(−d(zvi , ztj ))  . Text retrieval loss JIR(φ, θ)← JIR(φ, θ) + 1
B d(zvi , zti) + log∑ j∈I exp(−d(zti , zvj ))  . Image retrieval loss end for J(φ, θ)← λJTR(φ, θ) + (1− λ)JIR(φ, θ) . Add retrieval loss to the total loss J(φ, θ)← (1− κ)J(φ, θ) + κ2 (JTC(θ) + JIC(φ)) . Add classification loss to the total loss

heading: 2.1 RETRIEVAL LOSS FUNCTION
text: Pairwise cross-modal loss function is based solely on the pairwise relationships between texts and images. We choose to use the metric learning approach to capture the relationship between images and texts. Now, suppose d is a metric d : Z × Z → R+, vi is an image and τ = {tj′} is a collection of arbitrary texts sampled uniformly at random, of which text tj belongs to vi. We propose the following model for the probability of image vi and text tj to belong to the same object instance:
pφ,θ(i = j|vi, tj , τ) = exp(−d(fφ(vi), fθ(tj)))∑
tj′∈τ exp(−d(fφ(vi), fθ(tj′))) . (1)
The learning is then based on the following cross-entropy loss defined on the batch of size B:
JTR(φ, θ) = − 1
B B∑ i,j=1 `i,j log pφ,θ(i = j|vi, tj , {tj′}Bj′=1), (2)
where `i,j is a binary indicator of the true match (`i,j = 1, if i = j and 0 otherwise). Note that the expression above has the interpretation of the text retrieval loss. It attains its smallest value when for each image in the batch we manage to assign probability 1 to its respective text and 0 to all other texts. This can be further expanded as:
JTR(φ, θ) = 1
B B∑ i=1
Ñ d(fφ(vi), fθ(ti)) + log
[ ∑ tj′∈τ
exp(−d(fφ(vi), fθ(tj′))) ]é . (3)
Exchanging the order of image and text in the probability model (1) leads to the image retrieval loss, JIR(φ, θ). The two losses are mixed using parameter λ ∈ [0, 1] as shown in Algorithm 1. The pairwise retrieval loss functions are responsible for the modality alignment. In addition to those, we propose to include, as mentioned above, the usual image and text classifier losses. These losses are responsible for reducing the intraclass variability of representations. The classifier losses are added to the retrieval losses using a mixing parameter κ ∈ [0, 1] as shown in Algorithm 1.

heading: 2.2 BALANCING ACCURACY ON THE SEEN AND UNSEEN CLASSES
text: Let us define class prototypes p(y), each based on the set of texts Ty belonging to class y, {p(y) = 1 |Ty| ∑ ti∈Ty fθ(ti) | y ∈ Y}. In the context of GZSL, the nearest neighbor decision rule for a given image v and its features zv = fφ(v) has the following form:
ŷ = argmin y∈Y d(zv,p(y)) . (4)
The most acute problem faced in this setup is the accuracy imbalance between seen and unseen classes. A very representative case clearly outlining the imbalance problem is presented in Table 6 of (Xian et al., 2018c), where accuracy on the seen classes is always significantly greater than the accuracy on unseen ones. In order to measure and control the imbalance, three metrics are commonly used to assess the classification performance in the GZSL scenario: the Top-1 accuracy on the seen categories (s), the Top-1 accuracy on the unseen categories (u) and their harmonic mean, H = u · s/(u+ s). The main metric to assess GZSL performance is then H, which quantifies both u and s.
To formalize the problem, we first introduce yv , the true class label of image v. Mathematically, the main GZSL pain point is that P{ŷ ∈ Ytr|yv ∈ Yts} is significantly greater than P{ŷ ∈ Yts|yv ∈ Ytr}. In other words, the problem is that a given image is more likely to be confused with one of the seen classes if it belongs to an unseen class than vice versa. Our approach to solving the problem is based on the following probabilistic representation of the event space for the decision rule in Equation (4):
P{ŷ ∈ Ytr|yv ∈ Yts} = P ß min y∈Ytr d(zv,p(y)) < min y∈Yts d(zv,p(y)) | yv ∈ Yts ™ . (5)
Rephrasing, the most acute GZSL error happens when the prototype of one of the seen classes is closer to an image embedding from an unseen class than any of the prototypes of the unseen classes.
To rectify the situation we propose the following very direct solution to balance P{ŷ ∈ Ytr|yv ∈ Yts} and P{ŷ ∈ Yts|yv ∈ Ytr}. We introduce a positive scalar α ∈ R+ and scale all the distances corresponding to the seen prototypes by 1 + α. This gives rise to the following scaled distance dα:
dα(zv,p(y)) = ® (1 + α)d(zv,p(y)), if y ∈ Ytr d(zv,p(y)), otherwise . (6)
The misclassification between unseen as seen classes for the classifier ŷα, based on (6) is then: P{ŷα ∈ Ytr|yv ∈ Yts} = P ß (1 + α) min
y∈Ytr d(zv,p(y)) < min y∈Yts d(zv,p(y))|yv ∈ Yts
™ , (7)
and it has the following property: for any 0 ≤ α1 ≤ α2, P{ŷα1 ∈ Ytr|yv ∈ Yts} ≥ P{ŷα2 ∈ Ytr|yv ∈ Yts}, i.e. P{ŷα ∈ Ytr|yv ∈ Yts} is a monotone non-increasing function of α and we can reduce it by increasing α (please refer to Appendix A for a proof). Consider now P{ŷα ∈ Ytr|yv ∈ Ytr}, which is a probability that we classify an image v from one of the seen classes as still one of the seen classes. Using exactly the same chain of arguments as in Appendix A, it is straightforward to show that the probability is a non-increasing function of α. Hence the probability P{ŷα ∈ Yts|yv ∈ Ytr} = 1− P{ŷα ∈ Ytr|yv ∈ Ytr} is a non-decreasing function of α. Therefore, as α increases, we expect more classification errors in classifying images from seen classes, because some of them will be classified as one of the unseen classes.
To sum up, given the arguments presented above we expect that by varying α > 0 we can balance the error rate P{ŷα ∈ Ytr|yv ∈ Yts} of leaking the unseen class images into seen class classification decision and the error rate P{ŷα ∈ Yts|yv ∈ Ytr} of leaking the seen class images into unseen class classification decision. This is possible as we just showed above that P{ŷα ∈ Ytr|yv ∈ Yts} is a non-increasing function of α, while P{ŷα ∈ Yts|yv ∈ Ytr} is a non-decreasing one. It is also important to emphasize that α is applied only to distances between the query embedding and the prototypes of seen classes and it is constant over seen classes. Therefore, the application of α does not at all affect the classification error rates either within Ytr or within Yts. Varying α balances exclusively the classification errors arising from transitions between seen and unseen class labels. We study the empirical aspects of balancing α in Section 4.4.

heading: 3 RELATED WORK
text: ZSL approaches aim at recognizing objects belonging to classes unseen during training (Larochelle et al., 2008; Palatucci et al., 2009). This has been extended to the GZSL framework in which the decision space consists of both seen and unseen classes (Socher et al., 2013; Xian et al., 2018c). The classical zero-shot approaches build a joint visual-semantic space, relying on a linear cross-modal compatibility function (e.g. dot-product between query embedding and semantic prototypes or a variation of a hinge loss) (Frome et al., 2013; Akata et al., 2015; 2016; Reed et al., 2016). Non-linear variants of the compatibility has also been explored (Xian et al., 2016; Socher et al., 2013). Extending previously proposed cross-modal transfer approaches based on auto-encoders (Hubert Tsai et al., 2017) and cross-domain learning (Gretton et al., 2007), more recent line of work (Schönfeld et al., 2019; Xian et al., 2018d; 2019; Felix et al., 2018; Verma et al., 2018) relies on combining these approaches and their variations with dataset augmentation tools such as GAN (Goodfellow et al., 2014) and VAE (Kingma & Welling, 2014). It is argued that the use of those tools helps to resolve one of the prominent problems in GZSL scenario: classifying images from unseen classes as one of the seen classes. There exist approaches that try to tackle this same problem via temperature calibration (Liu et al., 2018) originally proposed by Hinton et al. (2015). Chao et al. (2016); Das & Lee (2019) proposed an approach to seen/unseen accuracy balancing that is very similar to ours, based on heuristic arguments. We extend this line of work here by providing a probabilistic justification for the balancing effect observed when applying metric rescaling. Atzmon & Chechik (2019) propose a more sophisticated way to deal with seen/unseen imbalance via adaptive confidence smoothing and gating, yet as authors note it is much simpler to train than the existing GAN-based zero-shot approaches. In this work, we introduce arguably the simplest zero-shot representation training approach of all, and we demonstrate that when the image level text information is available, it achieves the state-of-the-art results on GZSL task on two well-known datasets.

heading: 4 EXPERIMENTAL RESULTS

heading: 4.1 DATASETS
text: We focus on learning embeddings for fine-grained visual descriptions and test them in ZSL/GZSL scenario. To test the quality of trained embeddings we focus on datasets that provide paired images and text descriptions, such as Caltech-UCSD-Birds (CUB) (Welinder et al., 2010) and Oxford Flowers (FLOWERS) (Nilsback & Zisserman, 2008), that were augmented with textual descriptions by Reed et al. (2016). We use the GZSL splits proposed by Xian et al. (2018c). The attribute-based datasets,
such as SUN (Patterson et al., 2014) and AWA (Lampert et al., 2014) do not contain this information and do not have a notion of entity of a class in them. They are out of the scope of the current paper.

heading: 4.2 ARCHITECTURE AND TRAINING DETAILS
text: Below, we provide more detailed description of parameters used to build and train the architecture depicted in Figure 1. We use exactly the same hyperparameter settings for CUB and FLOWERS. The text feature extractor is built by cascading two ResNet blocks, followed by a BiLSTM. Each ResNet block has 3 convolutional/batch norm layers. The number of filters in the ResNet blocks is 128 and 256, BiLSTM has 512 filters for forward and backward branches (1024 total). All variables in the convolutional stack (including the batch normalization parameters γ and β) are L2-penalized with weight 0.001. The image feature extractor is a ResNet-101 with fixed weights pretrained on the split of ImageNet proposed by Xian et al. (2018c). In this work we use precomputed image features, available in (Xian et al., 2018a) for CUB and in (Xian et al., 2018b) for FLOWERS. Image and text features are projected in the common embedding space of size 1024 with FC layers with no non-linearity. They are preceded with a dropout of 0.25. The trainable components of the model are trained for 150k batches of size 32 using SGD with initial learning rate of 0.1 that is annealed by a factor of 10 every 50k batches. For each batch, we sample 32 instances, each instance includes a vector of precomputed ResNet-101 features and 10 text descriptions corresponding to it, according to the original dataset definition Reed et al. (2016). All 10 text descriptions are processed via the CNN/LSTM stack and the resulting embeddings are average pooled to create a vector representation of length 1024.

heading: 4.3 KEY RESULTS
text: Our key empirical results are compared in Table 1 and in Table 2 against the latest state of the art. Our results are based on the settings of λ = 0.5, κ = 0.5 and α selected on the validation sets of CUB and FLOWERS datasets. Please refer to Section 4.5 for the analysis of stability with respect to the choices of λ and κ and Sections 2.2 and 4.4 for more details on the selection of α. The combination of the proposed training method and the rebalancing of the metric space results in the state-of-the-art performance. Most of the current methods rely on the dataset augmentation techniques based on GANs, VAEs or combinations thereof. Those are clearly complementary w.r.t. our method and their addition to the training procedure is likely to further boost the performance of our proposed approach. However, this is outside of the scope of the current work. Moreover, the proposed method is state-of-the-art on FLOWERS even when compared against (Atzmon & Chechik, 2019) that uses both more sophisticated GAN based embedding learning approach and a more sophisticated seen/unseen error rate balancing based on COSMO. It is important to note that Atzmon & Chechik (2019) did not report the sentence level results on CUB. Yet, when applied on attributes together with f-CLSWGAN (Xian et al., 2018d) COSMO resulted in 0.8% performance drop and when applied with LAGO (Atzmon & Chechik, 2018) it achieved 0.5 % improvement over the attribute based state of the art.

heading: 4.4 ON THE SEEN/UNSEEN ACCURACY BALANCING
text: Figure 2 demonstrates the plot of harmonic mean Top-1 accuracy, H, against the value of α on the validation sets of CUB and FLOWERS datasets. The validation set is constructed by further splitting the train set on both datasets. For example, CUB has a train set of 5875 images from 100 seen classes and a validation set of 2946 images from 50 unseen classes. We further divide the train set into 4700 train images from 100 seen classes, 1175 seen validation images (4700 + 1175 = 5875) and we use all the 2946 images from 50 classes as the unseen validation set. Once the value of α is determined we train the representation on the full train+val subset and report results on the test split (the usual practice in GZSL). We confirm on the validation set that H exhibits an inverted U-shape behavior as a function of α, which was theoretically predicted in Section 2.2. Therefore, α can be selected on the validation set and then applied to re-scale the metric space to balance the accuracy on seen and unseen classes during test time as described in Section 2.2.

heading: 4.5 ABLATION STUDY
text: Figure 3 presents the results of the ablation study on the importance of image and text retrieval losses. We see that all of the Top-1 accuracies (H, s, u) are stable in the range of λ ∈ [0.2, 0.9], when both losses have tangible weight. Removing either text or image retrieval losses (setting λ to 0 or 1 respectively) leads to performance drop in both cases. Removing the text retrieval loss (case λ = 0) results in the most significant drop. This is due to the fact that the text retrieval task is more tightly related to the GZSL task. At the batch level, retrieving the right text given an image is equivalent to identifying the correct class encoded by a text prototype during ZSL inference step. The image retrieval task is not directly related to solving the ZSL problem and yet it does yield a positive regularizing effect on both CUB and FLOWERS.
Figure 4 shows the results of the ablation study of the interplay between the retrieval loss and the classification loss. We observe, just as in the case with λ, that there exists a reasonably flat and stable range of κ ∈ [0.2, 0.6]. The range for κ is a bit smaller. κ = 1 results in the catastrophic performance
drop: the classification losses by themselves do not enforce any modality alignment (please refer to Fig. 1 and Algorithm 1 clearly demonstrating this).
Table 3 studies the effects of different loss terms on the harmonic mean Top-1 accuracy H. The best result is achieved when all loss terms are active and when the metric space rescaling is on (the case of λ = 0.5, κ = 0.5 and α is checked, the last line in the table). Comparing this with the case when there is no metric space rescaling (first line with α = 0), we see that the rescaling helps to decrease the gap between seen and unseen classification accuracy. For CUB, the discrepancy reduction is from around 30% to around 6%, for FLOWERS it is from around 30% to around 1%. We would like to stress that we only use images and texts from the training set to achieve that. Going to the second line in the table (the image/text classification loss is inactive, κ = 0) and comparing it to the first one, we assess the effect of the image/text classification loss. It barely affects the performance on unseen set, but it significantly boosts the classification accuracy on the seen set (around 8% on both datasets). This is logical: adding a classifier loss results in a better classifier of the test images from the seen classes. This alone does not make it a better GZSL classifier, however. Only when applied together with metric space rescaling, this results in the performance boost (please refer to lines 1 and 6 in Table 3). Our interpretation is that the addition of the image/text classifier loss helps to reduce the intraclass variability in embeddings and provides for tighter clustering. However, this also leads to overfit on the classification task. This is accounted for by metric rescaling that enables the learnings from the image/text classification task be transferred effectively into the GZSL task.
The comparison of the last four rows of Table 3 leads us to believe that all the proposed loss terms outlined in Fig. 1 and Algorithm 1 are important for achieving the state-of-the-art performance. Excluding any one of them (corresponding to the extreme values λ = 0, λ = 1, κ = 0) leads to performance deterioration. Finally, an interesting observation can be made by comparing line 3 of Table 3 with performance of algorithms in Table 1. In this case our algorithm does not use any class labels and relies on training using exclusively the retrieval losses that can be calculated only based on the pairwise relationships between texts and images. We can see that using this type of supervision alone already results in a very high-quality representation. The representation is competitive against the latest GAN/VAE based approaches on CUB and is state-of-the-art on FLOWERS. This opens up new exploration avenues showing that in the case when very fine-grained modality outputs are available (image and text description pairs being a very prominent example), the high-quality representations may be learned without relying on manually supplied class labels.

heading: 5 CONCLUSIONS
text: We propose and empirically validate two improvements to the process of learning fine-grained crossmodal representations. First, we confirm the hypothesis that in the context of paired images and texts, a deep metric learning approach can be driven by an instance-based retrieval loss resulting in competitive generalized zero shot classification results. Combined with an additional class label based image/text crossentropy term this results in state-of-the-art performance on two well known datasets, CUB and FLOWERS. This is an interesting result demonstrating that high-quality deep representations can be trained relying largely on pairwise relationships between modalities. On top of that, we propose a solution to one of the prominent problems in GZSL: classifying instances of unseen classes as seen ones. We mathematically analyze and empirically validate the method of adjusting a single scalar that transcends in its effectiveness advanced dataset augmentation and training approaches based on GANs and VAEs.

heading: A THE ANALYSIS OF ERROR RATES
text: We show that P{ŷ ∈ Ytr|yv ∈ Yts} ≥ P{ŷα ∈ Ytr|yv ∈ Yts}. Let us define δtr ≡ miny∈Ytr d(zv,p(y)) and δts ≡ miny∈Yts d(zv,p(y)), then Equation (7) can be rewritten as:
P{ŷα ∈ Ytr|yv ∈ Yts} = P { (1 + α)δtr < δts|yv ∈ Yts } . (8)
Let us consider the probability of event δtr < δts and decompose it as follows: P { δtr < δts|yv ∈ Yts } = P { (1 + α)δtr < (1 + α)δts|yv ∈ Yts } = P { (1 + α)δtr < δts ∪ δts ≤ (1 + α)δtr < (1 + α)δts|yv ∈ Yts
} = P { (1 + α)δtr < δts|yv ∈ Yts } + P { δts ≤ (1 + α)δtr < (1 + α)δts|yv ∈ Yts
} − P { (1 + α)δtr < δts ∩ δts ≤ (1 + α)δtr < (1 + α)δts|yv ∈ Yts
} = P { (1 + α)δtr < δts|yv ∈ Yts } + P { δts ≤ (1 + α)δtr < (1 + α)δts|yv ∈ Yts } .
The transitions are based on the relationship between probabilities of arbitrary events A and B, P{A ∪B} = P{A}+ P{B} − P{A ∩B}, and in our case P{A ∩B} = 0. This implies that: P{ŷα ∈ Ytr|yv ∈ Yts} = P{ŷ ∈ Ytr|yv ∈ Yts} − P ß
δts (1 + α)
≤ δtr < δts|yv ∈ Yts ™
≤ P{ŷ ∈ Ytr|yv ∈ Yts}. (9)
We have just shown that for a non-negative α the probability of misclassifying an image from an unseen class as one of the seen classes is smaller for the decision rule ŷα than for the original decision rule ŷ. In fact, we can make a stronger claim. Since δts and δtr are non-negative, it is clear that the length of interval [δts/(1 + α), δts) increases as α increases, and hence probability that δtr falls in this interval is non-decreasing with increasing α. Thus we have for any 0 ≤ α1 ≤ α2, P{ŷα1 ∈ Ytr|yv ∈ Yts} ≥ P{ŷα2 ∈ Ytr|yv ∈ Yts}, i.e. P{ŷα ∈ Ytr|yv ∈ Yts} is a monotone non-increasing function of α and we can reduce it by increasing α.

Review summaries:
This paper tackles zero-shot and generalised zero-shot learning by using the per-image semantic information. An instance-based loss is introduced to align images and their corresponding text in the same embedding space. To solve the extreme imbalanced issue of generalized zero-shot learning, the authors propose to scale the prediction scores of seen classes by a constant factor. They demonstrate technic contributions on CUB and Flowers datasets and the results achieve the state-of-the-art.

The paper proposes to to use four different losses to train a joint text-image embedding space for zero-shot learning. The four losses consist of a classification loss given text descriptions, a classification loss given images, two contrastive losses given pairs of text and images. The paper also discusses how to balance seen and unseen classes, and it seems that, empirically, embeddings for seen classes are closer together while the embeddings for the unseen classes are further apart. A scaling factor that makes sure these distances are comparable for seen and unseen classes is introduced, and the paper gives an explanation for such a scaling. The final performance on the CUB and FLOWERS data set is impressive.