CLAREL: classification via retrieval loss for zero-shot learning

1 INTRODUCTION . Deep learning-based approaches have demonstrated superior flexibility and generalization capabilities in information processing on a wide variety of tasks , such as vision , speech and language ( LeCun et al. , 2015 ) . However , it has been widely realized that the transfer of deep representations to real-world applications is challenging due to the typical reliance on massive hand-labeled datasets . Learning in the low-labeled data regime , especially in the zero-shot ( Wang et al. , 2019 ) and the few-shot ( Wang & Yao , 2019 ) setups , have recently received significant attention in the literature . In the problem of zero-shot learning ( ZSL ) , the objective is to recognize categories that have not been seen during the training ( Larochelle et al. , 2008 ) . This is typically done by relying on anchor embeddings learned in one modality as prototypes and by associating a query embedding from the other modality with the closest prototype . In the generalized ZSL ( GZSL ) case ( Xian et al. , 2018c ) , the objective is more challenging as recognition is performed in the joint space of seen and unseen categories . ZSL , as well as its generalized counterpart , provide a viable framework to learn cross-modal representations that are flexible and adaptive . For example , in this paradigm , the adaptation to a new classification task based on text/image representation space alignment could be as easy as defining/appending/modifying a set of text sentences to define classes of new classifiers . This is an especially relevant problem as machine learning is challenged with the long tail of classes , and the idea of learning from pairs of images and sentences , abundant on the web , looks like a natural solution . Therefore , in this paper we specifically target the fine-grained scenario of paired images and their respective text descriptions . The uniqueness of this scenario is in the fact that the co-occurance of image and text provides a rich source of information . The ways of leveraging this source have not been sufficiently explored in the context of GZSL . Although we focus exclusively on the GZSL recognition setup in this paper , we believe that the research in this direction has potential to enable zero-shot flexibility in a wider array of high-level tasks such as segmentation or conditional image generation ( Zhang et al. , 2018 ) . The contributions of this work can be characterized under the following two themes . Instance-based training loss . Most prominent zero-shot learning approaches rely heavily on classlevel modality alignment ( Xian et al. , 2018c ) . We propose a new composite loss function that balances instance-based pairwise image/text retrieval loss and the usual classifier loss . The retrieval loss term does not use class labels . We demonstrate that the class-level information is important , but in the fine-grained text/image pairing scenarios , most of the GZSL accuracy can be extracted from the instance-based retrieval loss . To the best of our knowledge , this type of training has not been used in the GZSL literature . Its impressive performance opens up new promising research directions . Metric space rescaling . Metric-based ZSL approaches rely on distances between prototypes and query embeddings during inference . They are known to suffer from imbalanced performance on seen and unseen classes ( Liu et al. , 2018 ) . Previous work proposed to use a heuristic trick , calibrated stacking ( Chao et al. , 2016 ) or calibration ( Das & Lee , 2019 ) , to solve the problem . We refer to this technique as metric rescaling in our work , and provide a sound probabilistic justification for it . 2 PROPOSED METHOD . In this paper , we specifically target the fine-grained visual description scenario , as defined by Reed et al . ( 2016 ) . In this setting , the dataset consists of a number of images from a given set of classes and each image is accompanied by a number of textual descriptions . The task is to learn a joint representation space for images and texts that can be used for zero-shot recognition . An instance of the zero-shot multimodal representation learning problem can then be defined as follows . Given a training set S = { ( vn , tn , yn ) | vn ∈ V , tn ∈ T , yn ∈ Y , n = 1 . . . N } of image , text and label tuples , we are interested in finding representations fφ : V → Z of image , parameterized by φ , and fθ : T → Z of text , parameterized by θ , in a common embedding space Z . Furthermore , GZSL problem is defined using the sets of seen Ytr and unseen Yts classes , such that Y = Ytr ∪ Yts and Ytr∩Yts = ∅ . The training set will then only contain the seen classes , i.e . Str = { ( vn , tn , yn ) | vn ∈ V , tn ∈ T , yn ∈ Ytr } and the task is to build a classifier function g : Z ×Z → Y . This is different from the ZSL scenario focusing on g : Z × Z → Yts . To build g , most approaches to joint representation learning rely on class labeling to train a representation . For example , all the methods reviewed by Xian et al . ( 2018c ) require the access to class labels at train time . We hypothesise that in the fine-grained learning scenario , such as the one described by Reed et al . ( 2016 ) , a lot of information can be extracted simply from pairwise image/text cooccurrences . The class labels really only become critically necessary when we define class prototypes , i.e . at zero-shot test time . Following this intuition , we define a composite loss function that relies both on the pairwise relationships and on the class labels . The high-level description of the proposed framework is depicted in Figure 1 . The framework enables us , among other things , to experiment with the effects of train-time availability of class labels on the quality of zero-shot representations . The framework is based on projecting texts and images into a common space and then learning a representation based on a mixture of four loss functions : a pairwise text retrieval loss , a pairwise image retrieval loss , a text classifier loss and an image classifier loss ( see Algorithm 1 ) . Algorithm 1 Loss calculation for a single optimization iteration of the proposed method . N is the number of instances in the training set Str , B is the number of instances per batch , C is the number of classes in the train set . RANDOMSAMPLE ( S , B ) denotes a set of B elements chosen uniformly at random from a set S , without replacement . Input : Training set Str = { ( v1 , t1 , y1 ) , . . . , ( vN , tN , yN ) } , λ ∈ [ 0 , 1 ] , κ ∈ [ 0 , 1 ] . Output : The loss J ( φ , θ ) for a randomly sampled training batch . I ← RANDOMSAMPLE ( { 1 , . . . , N } , B ) . Select B instance indices for batch JTC ( θ ) , JIC ( φ ) ← 0 , 0 . Initialize classification losses for i in I do zvi , zti ← fφ ( vi ) , fθ ( ti ) . Embed images and texts pI ← softmax ( WIzvi + bI ) . Image classifier probabilities pT ← softmax ( WT zti + bT ) . Text classifier probabilities JTC ( θ ) ← JTC ( θ ) + 1B crossentropy ( pT , yi ) . Text classification loss JIC ( φ ) ← JIC ( φ ) + 1B crossentropy ( pI , yi ) . Image classification loss end for JTR ( φ , θ ) , JIR ( φ , θ ) ← 0 , 0 . Initialize retrieval losses for i in I do JTR ( φ , θ ) ← JTR ( φ , θ ) + 1 B d ( zvi , zti ) + log∑ j∈I exp ( −d ( zvi , ztj ) ) . Text retrieval loss JIR ( φ , θ ) ← JIR ( φ , θ ) + 1 B d ( zvi , zti ) + log∑ j∈I exp ( −d ( zti , zvj ) ) . Image retrieval loss end for J ( φ , θ ) ← λJTR ( φ , θ ) + ( 1− λ ) JIR ( φ , θ ) . Add retrieval loss to the total loss J ( φ , θ ) ← ( 1− κ ) J ( φ , θ ) + κ2 ( JTC ( θ ) + JIC ( φ ) ) . Add classification loss to the total loss 2.1 RETRIEVAL LOSS FUNCTION . Pairwise cross-modal loss function is based solely on the pairwise relationships between texts and images . We choose to use the metric learning approach to capture the relationship between images and texts . Now , suppose d is a metric d : Z × Z → R+ , vi is an image and τ = { tj′ } is a collection of arbitrary texts sampled uniformly at random , of which text tj belongs to vi . We propose the following model for the probability of image vi and text tj to belong to the same object instance : pφ , θ ( i = j|vi , tj , τ ) = exp ( −d ( fφ ( vi ) , fθ ( tj ) ) ) ∑ tj′∈τ exp ( −d ( fφ ( vi ) , fθ ( tj′ ) ) ) . ( 1 ) The learning is then based on the following cross-entropy loss defined on the batch of size B : JTR ( φ , θ ) = − 1 B B∑ i , j=1 ` i , j log pφ , θ ( i = j|vi , tj , { tj′ } Bj′=1 ) , ( 2 ) where ` i , j is a binary indicator of the true match ( ` i , j = 1 , if i = j and 0 otherwise ) . Note that the expression above has the interpretation of the text retrieval loss . It attains its smallest value when for each image in the batch we manage to assign probability 1 to its respective text and 0 to all other texts . This can be further expanded as : JTR ( φ , θ ) = 1 B B∑ i=1 Ñ d ( fφ ( vi ) , fθ ( ti ) ) + log [ ∑ tj′∈τ exp ( −d ( fφ ( vi ) , fθ ( tj′ ) ) ) ] é . ( 3 ) Exchanging the order of image and text in the probability model ( 1 ) leads to the image retrieval loss , JIR ( φ , θ ) . The two losses are mixed using parameter λ ∈ [ 0 , 1 ] as shown in Algorithm 1 . The pairwise retrieval loss functions are responsible for the modality alignment . In addition to those , we propose to include , as mentioned above , the usual image and text classifier losses . These losses are responsible for reducing the intraclass variability of representations . The classifier losses are added to the retrieval losses using a mixing parameter κ ∈ [ 0 , 1 ] as shown in Algorithm 1 .

Summary: The paper proposes to to use four different losses to train a joint text-image embedding space for zero-shot learning. The four losses consist of a classification loss given text descriptions, a classification loss given images, two contrastive losses given pairs of text and images. The paper also discusses how to balance seen and unseen classes, and it seems that, empirically, embeddings for seen classes are closer together while the embeddings for the unseen classes are further apart. A scaling factor that makes sure these distances are comparable for seen and unseen classes is introduced, and the paper gives an explanation for such a scaling. The final performance on the CUB and FLOWERS data set is impressive.