CLAREL: classification via retrieval loss for zero-shot learning

1 INTRODUCTION . Deep learning-based approaches have demonstrated superior flexibility and generalization capabilities in information processing on a wide variety of tasks , such as vision , speech and language ( LeCun et al. , 2015 ) . However , it has been widely realized that the transfer of deep representations to real-world applications is challenging due to the typical reliance on massive hand-labeled datasets . Learning in the low-labeled data regime , especially in the zero-shot ( Wang et al. , 2019 ) and the few-shot ( Wang & Yao , 2019 ) setups , have recently received significant attention in the literature . In the problem of zero-shot learning ( ZSL ) , the objective is to recognize categories that have not been seen during the training ( Larochelle et al. , 2008 ) . This is typically done by relying on anchor embeddings learned in one modality as prototypes and by associating a query embedding from the other modality with the closest prototype . In the generalized ZSL ( GZSL ) case ( Xian et al. , 2018c ) , the objective is more challenging as recognition is performed in the joint space of seen and unseen categories . ZSL , as well as its generalized counterpart , provide a viable framework to learn cross-modal representations that are flexible and adaptive . For example , in this paradigm , the adaptation to a new classification task based on text/image representation space alignment could be as easy as defining/appending/modifying a set of text sentences to define classes of new classifiers . This is an especially relevant problem as machine learning is challenged with the long tail of classes , and the idea of learning from pairs of images and sentences , abundant on the web , looks like a natural solution . Therefore , in this paper we specifically target the fine-grained scenario of paired images and their respective text descriptions . The uniqueness of this scenario is in the fact that the co-occurance of image and text provides a rich source of information . The ways of leveraging this source have not been sufficiently explored in the context of GZSL . Although we focus exclusively on the GZSL recognition setup in this paper , we believe that the research in this direction has potential to enable zero-shot flexibility in a wider array of high-level tasks such as segmentation or conditional image generation ( Zhang et al. , 2018 ) . The contributions of this work can be characterized under the following two themes . Instance-based training loss . Most prominent zero-shot learning approaches rely heavily on classlevel modality alignment ( Xian et al. , 2018c ) . We propose a new composite loss function that balances instance-based pairwise image/text retrieval loss and the usual classifier loss . The retrieval loss term does not use class labels . We demonstrate that the class-level information is important , but in the fine-grained text/image pairing scenarios , most of the GZSL accuracy can be extracted from the instance-based retrieval loss . To the best of our knowledge , this type of training has not been used in the GZSL literature . Its impressive performance opens up new promising research directions . Metric space rescaling . Metric-based ZSL approaches rely on distances between prototypes and query embeddings during inference . They are known to suffer from imbalanced performance on seen and unseen classes ( Liu et al. , 2018 ) . Previous work proposed to use a heuristic trick , calibrated stacking ( Chao et al. , 2016 ) or calibration ( Das & Lee , 2019 ) , to solve the problem . We refer to this technique as metric rescaling in our work , and provide a sound probabilistic justification for it . 2 PROPOSED METHOD . In this paper , we specifically target the fine-grained visual description scenario , as defined by Reed et al . ( 2016 ) . In this setting , the dataset consists of a number of images from a given set of classes and each image is accompanied by a number of textual descriptions . The task is to learn a joint representation space for images and texts that can be used for zero-shot recognition . An instance of the zero-shot multimodal representation learning problem can then be defined as follows . Given a training set S = { ( vn , tn , yn ) | vn ∈ V , tn ∈ T , yn ∈ Y , n = 1 . . . N } of image , text and label tuples , we are interested in finding representations fφ : V → Z of image , parameterized by φ , and fθ : T → Z of text , parameterized by θ , in a common embedding space Z . Furthermore , GZSL problem is defined using the sets of seen Ytr and unseen Yts classes , such that Y = Ytr ∪ Yts and Ytr∩Yts = ∅ . The training set will then only contain the seen classes , i.e . Str = { ( vn , tn , yn ) | vn ∈ V , tn ∈ T , yn ∈ Ytr } and the task is to build a classifier function g : Z ×Z → Y . This is different from the ZSL scenario focusing on g : Z × Z → Yts . To build g , most approaches to joint representation learning rely on class labeling to train a representation . For example , all the methods reviewed by Xian et al . ( 2018c ) require the access to class labels at train time . We hypothesise that in the fine-grained learning scenario , such as the one described by Reed et al . ( 2016 ) , a lot of information can be extracted simply from pairwise image/text cooccurrences . The class labels really only become critically necessary when we define class prototypes , i.e . at zero-shot test time . Following this intuition , we define a composite loss function that relies both on the pairwise relationships and on the class labels . The high-level description of the proposed framework is depicted in Figure 1 . The framework enables us , among other things , to experiment with the effects of train-time availability of class labels on the quality of zero-shot representations . The framework is based on projecting texts and images into a common space and then learning a representation based on a mixture of four loss functions : a pairwise text retrieval loss , a pairwise image retrieval loss , a text classifier loss and an image classifier loss ( see Algorithm 1 ) . Algorithm 1 Loss calculation for a single optimization iteration of the proposed method . N is the number of instances in the training set Str , B is the number of instances per batch , C is the number of classes in the train set . RANDOMSAMPLE ( S , B ) denotes a set of B elements chosen uniformly at random from a set S , without replacement . Input : Training set Str = { ( v1 , t1 , y1 ) , . . . , ( vN , tN , yN ) } , λ ∈ [ 0 , 1 ] , κ ∈ [ 0 , 1 ] . Output : The loss J ( φ , θ ) for a randomly sampled training batch . I ← RANDOMSAMPLE ( { 1 , . . . , N } , B ) . Select B instance indices for batch JTC ( θ ) , JIC ( φ ) ← 0 , 0 . Initialize classification losses for i in I do zvi , zti ← fφ ( vi ) , fθ ( ti ) . Embed images and texts pI ← softmax ( WIzvi + bI ) . Image classifier probabilities pT ← softmax ( WT zti + bT ) . Text classifier probabilities JTC ( θ ) ← JTC ( θ ) + 1B crossentropy ( pT , yi ) . Text classification loss JIC ( φ ) ← JIC ( φ ) + 1B crossentropy ( pI , yi ) . Image classification loss end for JTR ( φ , θ ) , JIR ( φ , θ ) ← 0 , 0 . Initialize retrieval losses for i in I do JTR ( φ , θ ) ← JTR ( φ , θ ) + 1 B d ( zvi , zti ) + log∑ j∈I exp ( −d ( zvi , ztj ) ) . Text retrieval loss JIR ( φ , θ ) ← JIR ( φ , θ ) + 1 B d ( zvi , zti ) + log∑ j∈I exp ( −d ( zti , zvj ) ) . Image retrieval loss end for J ( φ , θ ) ← λJTR ( φ , θ ) + ( 1− λ ) JIR ( φ , θ ) . Add retrieval loss to the total loss J ( φ , θ ) ← ( 1− κ ) J ( φ , θ ) + κ2 ( JTC ( θ ) + JIC ( φ ) ) . Add classification loss to the total loss 2.1 RETRIEVAL LOSS FUNCTION . Pairwise cross-modal loss function is based solely on the pairwise relationships between texts and images . We choose to use the metric learning approach to capture the relationship between images and texts . Now , suppose d is a metric d : Z × Z → R+ , vi is an image and τ = { tj′ } is a collection of arbitrary texts sampled uniformly at random , of which text tj belongs to vi . We propose the following model for the probability of image vi and text tj to belong to the same object instance : pφ , θ ( i = j|vi , tj , τ ) = exp ( −d ( fφ ( vi ) , fθ ( tj ) ) ) ∑ tj′∈τ exp ( −d ( fφ ( vi ) , fθ ( tj′ ) ) ) . ( 1 ) The learning is then based on the following cross-entropy loss defined on the batch of size B : JTR ( φ , θ ) = − 1 B B∑ i , j=1 ` i , j log pφ , θ ( i = j|vi , tj , { tj′ } Bj′=1 ) , ( 2 ) where ` i , j is a binary indicator of the true match ( ` i , j = 1 , if i = j and 0 otherwise ) . Note that the expression above has the interpretation of the text retrieval loss . It attains its smallest value when for each image in the batch we manage to assign probability 1 to its respective text and 0 to all other texts . This can be further expanded as : JTR ( φ , θ ) = 1 B B∑ i=1 Ñ d ( fφ ( vi ) , fθ ( ti ) ) + log [ ∑ tj′∈τ exp ( −d ( fφ ( vi ) , fθ ( tj′ ) ) ) ] é . ( 3 ) Exchanging the order of image and text in the probability model ( 1 ) leads to the image retrieval loss , JIR ( φ , θ ) . The two losses are mixed using parameter λ ∈ [ 0 , 1 ] as shown in Algorithm 1 . The pairwise retrieval loss functions are responsible for the modality alignment . In addition to those , we propose to include , as mentioned above , the usual image and text classifier losses . These losses are responsible for reducing the intraclass variability of representations . The classifier losses are added to the retrieval losses using a mixing parameter κ ∈ [ 0 , 1 ] as shown in Algorithm 1 .

Summary: This paper tackles zero-shot and generalised zero-shot learning by using the per-image semantic information. An instance-based loss is introduced to align images and their corresponding text in the same embedding space. To solve the extreme imbalanced issue of generalized zero-shot learning, the authors propose to scale the prediction scores of seen classes by a constant factor. They demonstrate technic contributions on CUB and Flowers datasets and the results achieve the state-of-the-art.