Zero-Shot Recognition through Image-Guided Semantic Classification

1 INTRODUCTION . As a feasible solution for addressing the limitations of supervised classification methods , zeroshot learning ( ZSL ) aims to recognize objects whose instances have not been seen during training ( Larochelle et al. , 2008 ; Palatucci et al. , 2009 ) . Unseen classes are recognized by associating seen and unseen classes through some form of semantic space ; therefore , the knowledge learned from seen classes is transferred to unseen classes . In the semantic space , each class has a corresponding vector representation called a class prototype . Class prototypes can be obtained from human-annotated attributes that describe visual properties of objects ( Farhadi et al. , 2009 ; Lampert et al. , 2014 ) or from word embeddings learned in an unsupervised manner from text corpus ( Mikolov et al. , 2013 ; Pennington et al. , 2014 ; Devlin et al. , 2018 ) . A majority of ZSL methods can be viewed using the visual-semantic embedding framework , as displayed in Figure 1 ( a ) . Images are mapped from the visual space to the semantic space in which all classes reside , or images and labels are projected to a latent space ( Yang & Hospedales , 2015 ; Liu et al. , 2018 ) . Then , the inference is performed in this common space ( Akata et al. , 2013 ; Frome et al. , 2013 ; Socher et al. , 2013 ) , typically using cosine similarity or Euclidean distance . Another perspective of embedding-based methods is to construct an image classifier for each unseen class by learning the correspondence between a binary one-versus-rest image classifier ( i.e. , visual representation of a class ) and its class prototype in the semantic space ( i.e. , semantic representation of a class ) ( Wang et al. , 2019 ) . Once this correspondence function is learned , a binary one-versus-rest image classifier can be constructed for an unseen class with its prototype ( Wang et al. , 2019 ) . For example , a commonly used choice for such correspondence is the bilinear function ( Frome et al. , 2013 ; Akata et al. , 2013 ; 2015 ; Romera-Paredes & Torr , 2015 ; Li et al. , 2018 ) . Considerable efforts have been made to extend the linear function to nonlinear ones ( Xian et al. , 2016 ; Wang et al. , 2017 ; Elhoseiny et al. , 2017 ; Qiao et al. , 2016 ) . Figure 1 ( b ) illustrates this perspective . Learning the correspondence between an image classifier and a class prototype has the following drawbacks . First , the assumption of using a single image classifier for each class is restrictive because the manner for separating classes in both visual and semantic spaces would not be unique . We argue that semantic classification should be conducted dynamically conditioned on an input image . For example , the visual attribute wheel may be useful for classifying most car images . Nevertheless , cars with missing wheels should also be correctly recognized using other visual attributes . Therefore , instance-specific semantic classifiers are more preferable than category-specific ones because the classifier weights can be adaptively determined based on image content . Second , the scale of training data for learning the correspondence is constrained to be the number of class labels . In other words , a training set with C labels has only C visual-semantic classifier pairs to build the correspondence . This may hinder the robustness of deep models that usually require large-scale training data . Finally , although class embedding has rich semantic meanings , each class is represented by only a single class prototype to determine where images of that class collapse inevitably ( MarcoBaroni , 2016 ; Fu et al. , 2015 ) . The mapped semantic representations from images may collapse to hubs , which are close to many other points in the semantic space , rather than being similar to the true class label ( MarcoBaroni , 2016 ) . In this paper , we present a new method , named Image-Guided Semantic Classification ( IGSC ) , to address these problems . IGSC aims to learn the correspondence between an image and its corresponding label classifier , as illustrated in Figure 1 ( c ) . In contrast to existing methods focusing on the learning of visual ( or semantic ) representations ( Zhang et al. , 2016 ; Frome et al. , 2013 ; Socher et al. , 2013 ) , IGSC analyzes the input image and seeks for combinations of variables in the semantic space ( e.g. , combinations of attributes ) that distinguish a class ( belonging to the input ) from other classes . The proposed IGSC method has the following characteristics : • IGSC learns the correspondence between an image in the visual space and a classifier in the semantic space . The correspondence can be learned with training pairs in the scale of training images rather than that of classes . • IGSC performs learning to learn in an end-to-end manner . Label classification is conducted by a semantic classifier whose weights are generated on the fly . This model is simple yet powerful because of its adaptive nature . • IGSC unifies visual attribute detection and label classification . This is achieved via the design of a conditional network ( the proposed classifier learning method ) , in which label classification is the main task of interest and the conditional input image provides additional information of a specific situation . • IGSC alleviates the hubness problem . The correspondence between an image and a semantic classifier learned from seen classes can be transferred to recognize unseen concepts . We evaluated IGSC with experiments conducted on four public benchmark datasets , including SUN ( Patterson & Hays , 2012 ) , CUB ( Patterson & Hays , 2012 ) , AWA2 ( Lampert et al. , 2014 ) , and aPY ( Farhadi et al. , 2009 ) . Experimental results demonstrated that the proposed method achieved promising performance , compared with current state-of-the-art methods.The remainder of the paper is organized as follows : We briefly review related work in Section 2 . Section 3 presents the proposed framework . The experimental results and conclusions are provided in Sections 4 and 5 , respectively . 2 RELATED WORK . Zero-shot learning has evolved rapidly during the last decade , and therefore documenting the extensive literature with limited pages is rarely possible . In this section , we review a few representative zero-shot learning methods and refer readers to ( Xian et al. , 2019a ; Wang et al. , 2019 ) for a comprehensive survey . One pioneering main stream of ZSL uses attributes to infer the label of an image belonging to one of the unseen classes ( Lampert et al. , 2014 ; Al-Halah et al. , 2016 ; Norouzi et al. , 2014 ; Jayaraman & Grauman , 2014 ; Kankuekul et al. , 2012 ) . The attributes of an image are predicted , then the class label is inferred by searching the class which attains the most similar set of attributes . For example , the Direct Attribute Prediction ( DAP ) model ( Lampert et al. , 2009 ) estimates the posterior of each attribute for an image by learning probabilistic attribute classifiers . A test sample is then classified by each attribute classifier alternately , and the class label is predicted by probabilistic estimation . Similar to the attribute-based methods , the proposed method has the merits of modeling the relationships among classes . However , IGSC unifies these two steps : attribute classifier learning and inferring from detected attributes to the class . Furthermore , attribute classifiers are jointly learned in IGSC . A broad family of ZSL methods apply an embedding framework that directly learns a mapping from the visual space to the semantic space ( Palatucci et al. , 2009 ; Akata et al. , 2013 ; 2015 ; RomeraParedes & Torr , 2015 ) . The visual-to-semantic mapping can be linear ( Frome et al. , 2013 ) or nonlinear ( Socher et al. , 2013 ) . For example , DeViSE ( Frome et al. , 2013 ) learns a linear mapping between the image and semantic spaces using an efficient ranking loss formulation . Cross-Modal Transfer ( CMT ) ( Socher et al. , 2013 ) uses a neural network with two hidden layers to learn a nonlinear projection from image feature space to word vector space . More recently , deep neural network models are proposed to mirror learned semantic relations among classes in the visual domain from the image ( Annadani & Biswas , 2018 ) or from the part ( Zhu et al. , 2018a ) levels . IGSC is also an embedding-based ZSL method . IGSC differs significantly from existing methods in that IGSC learns the correspondence between an image and its semantic classifier , enabling the possibility of using different classification manners to separate class prototypes in the semantic space . Recent ZSL models adopt the generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) or other generative models for synthesizing unseen examples ( Bucher et al. , 2017 ; Long et al. , 2017 ; Jiang et al. , 2018 ; Verma et al. , 2018 ; Xian et al. , 2018 ; Zhu et al. , 2018b ; Xian et al. , 2019b ; Verma et al. , 2020 ; Yu et al. , 2020 ; Ma & Hu , 2020 ) or for reconstructing training images ( Chen et al. , 2018 ) . The synthesized images obtained at the training stage can be fed to conventional classifiers so that ZSL is converted into the conventional supervised learning problem ( Long et al. , 2017 ) . The transformation from attributes to image features require involving generative models such as denoising autoencoders ( Bucher et al. , 2017 ) , GAN ( Xian et al. , 2018 ; Zhu et al. , 2018b ) or their variants ( Verma et al. , 2018 ; Felix et al. , 2018 ; Xian et al. , 2019b ; Yu et al. , 2020 ; Ma & Hu , 2020 ) . Despite outstanding performances reported in the papers , these works leverage some form of the unseen class information during training . In view of real-world applications involving recognition in-the-wild , novel classes including the image samples as well as the semantic representations may not be available in model learning . The proposed method is agnostic to all unseen class information during training . Furthermore , the proposed method is much simpler in the architecture design and has a much smaller model size , compared with the generative methods . 3 APPROACH . 3.1 PROBLEM DESCRIPTION . Given a training set S = { ( xn , yn ) , n = 1 . . . N } , with yn ∈ Ys being a class label in the seen class set , the goal of ZSL is to learn a classifier f : X → Y which can generalize to predict any image x to its correct label , which is not only in Ys but also in the unseen class set Yu . In the prevailing family of compatibility learning ZSL ( Xian et al. , 2019a ; Ba et al. , 2015 ) , the prediction is made via : ŷ = f ( x ; W ) = argmax y∈Y F ( x , y ; W ) . ( 1 ) In particular , if Y = Yu , this is the conventional ZSL setting ; if Y = Ys∪Yu , this is the generalized zero-shot learning ( GZSL ) setting , which is more practical for real-world applications . The compatibility function F ( · ) —parameterized by W—is used to associate visual and semantic information . In the visual space , each image x has a vector representation , denoted by θ ( x ) . Similarly , each class label y has a vector representation in the semantic space ( called the class prototype ) , denoted by φ ( y ) . In short , θ ( x ) and φ ( y ) are the image and class embeddings , both of which are given .

Summary: This paper proposes a simple yet effective method for zero-shot learning. In the method, a network is learned to predict the compatibility function weight given the input of the image. The predicted weight is then applied to semantic attributes and the final class label is predicted by the maximum compatibility score. The method is evaluated on benchmark datasets and illustrates competitive performance.