Zero-Shot Recognition through Image-Guided Semantic Classification

1 INTRODUCTION . As a feasible solution for addressing the limitations of supervised classification methods , zeroshot learning ( ZSL ) aims to recognize objects whose instances have not been seen during training ( Larochelle et al. , 2008 ; Palatucci et al. , 2009 ) . Unseen classes are recognized by associating seen and unseen classes through some form of semantic space ; therefore , the knowledge learned from seen classes is transferred to unseen classes . In the semantic space , each class has a corresponding vector representation called a class prototype . Class prototypes can be obtained from human-annotated attributes that describe visual properties of objects ( Farhadi et al. , 2009 ; Lampert et al. , 2014 ) or from word embeddings learned in an unsupervised manner from text corpus ( Mikolov et al. , 2013 ; Pennington et al. , 2014 ; Devlin et al. , 2018 ) . A majority of ZSL methods can be viewed using the visual-semantic embedding framework , as displayed in Figure 1 ( a ) . Images are mapped from the visual space to the semantic space in which all classes reside , or images and labels are projected to a latent space ( Yang & Hospedales , 2015 ; Liu et al. , 2018 ) . Then , the inference is performed in this common space ( Akata et al. , 2013 ; Frome et al. , 2013 ; Socher et al. , 2013 ) , typically using cosine similarity or Euclidean distance . Another perspective of embedding-based methods is to construct an image classifier for each unseen class by learning the correspondence between a binary one-versus-rest image classifier ( i.e. , visual representation of a class ) and its class prototype in the semantic space ( i.e. , semantic representation of a class ) ( Wang et al. , 2019 ) . Once this correspondence function is learned , a binary one-versus-rest image classifier can be constructed for an unseen class with its prototype ( Wang et al. , 2019 ) . For example , a commonly used choice for such correspondence is the bilinear function ( Frome et al. , 2013 ; Akata et al. , 2013 ; 2015 ; Romera-Paredes & Torr , 2015 ; Li et al. , 2018 ) . Considerable efforts have been made to extend the linear function to nonlinear ones ( Xian et al. , 2016 ; Wang et al. , 2017 ; Elhoseiny et al. , 2017 ; Qiao et al. , 2016 ) . Figure 1 ( b ) illustrates this perspective . Learning the correspondence between an image classifier and a class prototype has the following drawbacks . First , the assumption of using a single image classifier for each class is restrictive because the manner for separating classes in both visual and semantic spaces would not be unique . We argue that semantic classification should be conducted dynamically conditioned on an input image . For example , the visual attribute wheel may be useful for classifying most car images . Nevertheless , cars with missing wheels should also be correctly recognized using other visual attributes . Therefore , instance-specific semantic classifiers are more preferable than category-specific ones because the classifier weights can be adaptively determined based on image content . Second , the scale of training data for learning the correspondence is constrained to be the number of class labels . In other words , a training set with C labels has only C visual-semantic classifier pairs to build the correspondence . This may hinder the robustness of deep models that usually require large-scale training data . Finally , although class embedding has rich semantic meanings , each class is represented by only a single class prototype to determine where images of that class collapse inevitably ( MarcoBaroni , 2016 ; Fu et al. , 2015 ) . The mapped semantic representations from images may collapse to hubs , which are close to many other points in the semantic space , rather than being similar to the true class label ( MarcoBaroni , 2016 ) . In this paper , we present a new method , named Image-Guided Semantic Classification ( IGSC ) , to address these problems . IGSC aims to learn the correspondence between an image and its corresponding label classifier , as illustrated in Figure 1 ( c ) . In contrast to existing methods focusing on the learning of visual ( or semantic ) representations ( Zhang et al. , 2016 ; Frome et al. , 2013 ; Socher et al. , 2013 ) , IGSC analyzes the input image and seeks for combinations of variables in the semantic space ( e.g. , combinations of attributes ) that distinguish a class ( belonging to the input ) from other classes . The proposed IGSC method has the following characteristics : • IGSC learns the correspondence between an image in the visual space and a classifier in the semantic space . The correspondence can be learned with training pairs in the scale of training images rather than that of classes . • IGSC performs learning to learn in an end-to-end manner . Label classification is conducted by a semantic classifier whose weights are generated on the fly . This model is simple yet powerful because of its adaptive nature . • IGSC unifies visual attribute detection and label classification . This is achieved via the design of a conditional network ( the proposed classifier learning method ) , in which label classification is the main task of interest and the conditional input image provides additional information of a specific situation . • IGSC alleviates the hubness problem . The correspondence between an image and a semantic classifier learned from seen classes can be transferred to recognize unseen concepts . We evaluated IGSC with experiments conducted on four public benchmark datasets , including SUN ( Patterson & Hays , 2012 ) , CUB ( Patterson & Hays , 2012 ) , AWA2 ( Lampert et al. , 2014 ) , and aPY ( Farhadi et al. , 2009 ) . Experimental results demonstrated that the proposed method achieved promising performance , compared with current state-of-the-art methods.The remainder of the paper is organized as follows : We briefly review related work in Section 2 . Section 3 presents the proposed framework . The experimental results and conclusions are provided in Sections 4 and 5 , respectively . 2 RELATED WORK . Zero-shot learning has evolved rapidly during the last decade , and therefore documenting the extensive literature with limited pages is rarely possible . In this section , we review a few representative zero-shot learning methods and refer readers to ( Xian et al. , 2019a ; Wang et al. , 2019 ) for a comprehensive survey . One pioneering main stream of ZSL uses attributes to infer the label of an image belonging to one of the unseen classes ( Lampert et al. , 2014 ; Al-Halah et al. , 2016 ; Norouzi et al. , 2014 ; Jayaraman & Grauman , 2014 ; Kankuekul et al. , 2012 ) . The attributes of an image are predicted , then the class label is inferred by searching the class which attains the most similar set of attributes . For example , the Direct Attribute Prediction ( DAP ) model ( Lampert et al. , 2009 ) estimates the posterior of each attribute for an image by learning probabilistic attribute classifiers . A test sample is then classified by each attribute classifier alternately , and the class label is predicted by probabilistic estimation . Similar to the attribute-based methods , the proposed method has the merits of modeling the relationships among classes . However , IGSC unifies these two steps : attribute classifier learning and inferring from detected attributes to the class . Furthermore , attribute classifiers are jointly learned in IGSC . A broad family of ZSL methods apply an embedding framework that directly learns a mapping from the visual space to the semantic space ( Palatucci et al. , 2009 ; Akata et al. , 2013 ; 2015 ; RomeraParedes & Torr , 2015 ) . The visual-to-semantic mapping can be linear ( Frome et al. , 2013 ) or nonlinear ( Socher et al. , 2013 ) . For example , DeViSE ( Frome et al. , 2013 ) learns a linear mapping between the image and semantic spaces using an efficient ranking loss formulation . Cross-Modal Transfer ( CMT ) ( Socher et al. , 2013 ) uses a neural network with two hidden layers to learn a nonlinear projection from image feature space to word vector space . More recently , deep neural network models are proposed to mirror learned semantic relations among classes in the visual domain from the image ( Annadani & Biswas , 2018 ) or from the part ( Zhu et al. , 2018a ) levels . IGSC is also an embedding-based ZSL method . IGSC differs significantly from existing methods in that IGSC learns the correspondence between an image and its semantic classifier , enabling the possibility of using different classification manners to separate class prototypes in the semantic space . Recent ZSL models adopt the generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) or other generative models for synthesizing unseen examples ( Bucher et al. , 2017 ; Long et al. , 2017 ; Jiang et al. , 2018 ; Verma et al. , 2018 ; Xian et al. , 2018 ; Zhu et al. , 2018b ; Xian et al. , 2019b ; Verma et al. , 2020 ; Yu et al. , 2020 ; Ma & Hu , 2020 ) or for reconstructing training images ( Chen et al. , 2018 ) . The synthesized images obtained at the training stage can be fed to conventional classifiers so that ZSL is converted into the conventional supervised learning problem ( Long et al. , 2017 ) . The transformation from attributes to image features require involving generative models such as denoising autoencoders ( Bucher et al. , 2017 ) , GAN ( Xian et al. , 2018 ; Zhu et al. , 2018b ) or their variants ( Verma et al. , 2018 ; Felix et al. , 2018 ; Xian et al. , 2019b ; Yu et al. , 2020 ; Ma & Hu , 2020 ) . Despite outstanding performances reported in the papers , these works leverage some form of the unseen class information during training . In view of real-world applications involving recognition in-the-wild , novel classes including the image samples as well as the semantic representations may not be available in model learning . The proposed method is agnostic to all unseen class information during training . Furthermore , the proposed method is much simpler in the architecture design and has a much smaller model size , compared with the generative methods . 3 APPROACH . 3.1 PROBLEM DESCRIPTION . Given a training set S = { ( xn , yn ) , n = 1 . . . N } , with yn ∈ Ys being a class label in the seen class set , the goal of ZSL is to learn a classifier f : X → Y which can generalize to predict any image x to its correct label , which is not only in Ys but also in the unseen class set Yu . In the prevailing family of compatibility learning ZSL ( Xian et al. , 2019a ; Ba et al. , 2015 ) , the prediction is made via : ŷ = f ( x ; W ) = argmax y∈Y F ( x , y ; W ) . ( 1 ) In particular , if Y = Yu , this is the conventional ZSL setting ; if Y = Ys∪Yu , this is the generalized zero-shot learning ( GZSL ) setting , which is more practical for real-world applications . The compatibility function F ( · ) —parameterized by W—is used to associate visual and semantic information . In the visual space , each image x has a vector representation , denoted by θ ( x ) . Similarly , each class label y has a vector representation in the semantic space ( called the class prototype ) , denoted by φ ( y ) . In short , θ ( x ) and φ ( y ) are the image and class embeddings , both of which are given .

Summary: The authors tackle the problem of zero-shot learning, that is, the recognition of classes and categories for which no visual data are available, but only semantic embedding, providing a description of the classes in terms of auxiliary textual descriptions. To this aim, authors propose a method dubbed  Image-Guided Semantic Classification in which a two-stream network (fed by either visual and semantic embedding) learns a compatibility function whose recognition performance is enhanced by means of calibrated stacking (Chao et al. 2016).