Adaptive and Generative Zero-Shot Learning

We address the problem of generalized zero-shot learning ( GZSL ) where the task is to predict the class label of a target image whether its label belongs to the seen or unseen category . Similar to ZSL , the learning setting assumes that all class-level semantic features are given , while only the images of seen classes are available for training . By exploring the correlation between image features and the corresponding semantic features , the main idea of the proposed approach is to enrich the semantic-to-visual ( S2V ) embeddings via a seamless fusion of adaptive and generative learning . To this end , we extend the semantic features of each class by supplementing image-adaptive attention so that the learned S2V embedding can account for not only inter-class but also intra-class variations . In addition , to break the limit of training with images only from seen classes , we design a generative scheme to simultaneously generate virtual class labels and their visual features by sampling and interpolating over seen counterparts . In inference , a testing image will give rise to two different S2V embeddings , seen and virtual . The former is used to decide whether the underlying label is of the unseen category or otherwise a specific seen class ; the latter is to predict an unseen class label . To demonstrate the effectiveness of our method , we report state-of-the-art results on four standard GZSL datasets , including an ablation study of the proposed modules . 1 INTRODUCTION . Different from conventional learning tasks , zero-shot learning ( ZSL ) by Lampert et al . ( 2009 ) ; Palatucci et al . ( 2009 ) ; Akata et al . ( 2013 ) explores the extreme case of performing inference only over samples of unseen classes . To make the scenario more realistic , generalized zero-shot learning ( GZSL ) ( Chao et al. , 2016 ; Xian et al. , 2017 ) is subsequently proposed so that inference can concern samples of both seen and unseen classes . Nevertheless . the learning setting in ZSL/GZSL is essentially the same where sample classes are divided into two categories , seen and unseen , but only those samples of seen classes are accessible to training . In addition , each of all the classes under consideration is characterized by semantic features such as attributes ( Xian et al. , 2018b ) or text descriptions ( Zhu et al. , 2018 ) to specify and relate seen and unseen classes . The lack of training samples from unseen classes has prompted generative approaches ( Chen et al. , 2018 ; Felix et al. , 2018 ; Kumar Verma et al. , 2018 ; Mishra et al. , 2018 ) to creating synthetic data from semantic features of unseen classes . The strategy could enable learning semantic-visual alignment on unseen classes implicitly , and thus improves the ability to classify unseen classes . However , such generative models are indeed trained on seen samples , and the quality of synthesized unseen samples is predominantly influenced by seen classes . If the number of training samples of each seen class is small , it is hard for generative models to adequately synthesize samples of unseen classes , leading to unsatisfactory zero-shot learning . To better address the issue , we propose to synthesize visual and semantic features of virtual classes rather than those of the unseen classes . An interesting analogy is that childhood experience and relevant study ( Greene , 1995 ) suggest the behavior of using human imagination to produce new object concepts could assist our cognitive capability . To mimic people utilizing imagination for exploring new knowledge , we create virtual classes by the integration of past ” experience ” ( seen classes ) . In detail , we extend the mixup technique by Zhang et al . ( 2018 ) to generate virtual classes , with a subtle difference that mixing is conducted on the semantic features ( in addition to the visual ones ) , instead of the class label vectors . In ZSL/GZSL , each seen or unseen class is typically described by a single semantic feature vector . The practice is useful in differentiating different classes in a principled way , but may not be sufficient to reflect the inter-class and intra-class visual discrepancies , not to mention the ambiguities caused by different backgrounds , view orientations , or occlusion in images . The concern of inefficient class-level representation can also be observed from how the semantic feature vectors are constructed . Take , for example , the Attribute Pascal and Yahoo ( aPY ) dataset ( Farhadi et al. , 2009 ) , where each instance is annotated by 64 attributes . The semantic features of each class in aPY are obtained by averaging the attribute vectors of all its instances . We are thus motivated to introduce an imageadaptive class representation , integrating the original semantic features for inter-class discrimination with an image-specific attention vector for intra-class variations . With the addition of virtual training data and the image-adaptive class representation , our method is designed to learn two classification experts : one for seen classes and the other for unseen classes . Both experts project the image-adaptive semantic feature vectors to the visual space and use cosine similarity to find the class label most similar to the given visual feature vector . The seen expert is trained with the provided training ( seen ) data , while the class prediction is over all possible classes , including seen and unseen . In inference , if its predicted class is not within the seen category . The testing sample is deemed to be from the unseen category , whose label is then decided by the unseen expert . The unseen expert is trained with the virtual data only , and the process indeed resembles meta-learning . However , the effectiveness of meta-learning is boosted by the design of the imageadaptive mechanism in that fine-tuning is not needed in performing zero-shot classification over unseen classes . We characterize the main contributions of this work as follows . • Instead of generating synthetic data of unseen classes , we propose to yield virtual classes and data by mixup interpolations . The virtual classes of synthetic data can then be seamlessly coupled with meta-learning to improve the inference on unseen testing samples . • We introduce the concept of representing each class with image-adaptive semantic features that could vary among intra-class samples . While the adaptive mechanism improves classifying the seen classes , it manifests the advantage in boosting the effect of meta-learning over virtual data to zero-shot inference over unseen classes . • We demonstrate state-of-the-art results of zero-shot learning over four popular benchmark datasets and justify the design of our method with a thorough ablation study . 2 RELATED WORK . We review relevant literature in this section . First , we describe generative approaches for ZSL/GZSL that synthesize unseen images for training . To improve GZSL performance , we propose to couple virtual class generation with meta-learning for mimicking the inference scenario . Next , we discuss attention approaches that extract discriminating features from images to help classification . 2.1 GENERATIVE APPROACHES FOR ZSL/GZSL . Arguably one of the most important problems in ZSL/GZSL is to prevent models from being biased to seen classes . Generative approaches ( Chen et al. , 2018 ; Felix et al. , 2018 ; Kumar Verma et al. , 2018 ; Mishra et al. , 2018 ; Schonfeld et al. , 2019 ; Paul et al. , 2019 ; Xian et al. , 2019 ) tackle the problem by synthesizing visual features of unseen classes from their semantic features with generative models like Generative Adversarial Networks ( GAN ; Goodfellow et al. , 2014 ) or Variational Autoencoders ( VAE ) ( VAE ; Kingma & Welling , 2014 ) . The synthetic visual features act as pseudo-examples of unseen classes , and effectively reduce the ZSL/GZSL problem to a supervised learning one . One key issue behind generative approaches comes from the insufficient amount of data to learn a good generative model . As a consequence , some semantic features that seem important during training may cause overfitting , and others that seem less important may be completely dropped . Therefore , several prior techniques propose new constraints or losses to preserve semantic features and regularize the generative model . For instance , ( Chen et al. , 2018 ) avoids the dropping of semantic information by disentangling the semantic space into two subspaces , one for classification and the other for reconstruction ; Felix et al . ( 2018 ) enforces visual-semantic feature consistency by requiring synthesized visual features to reconstruct semantic features accurately . However , even with the new constraints or losses , the success of generative approaches still highly depends on whether there is enough data/variation in the seen classes to synthesize diverse visual features . Otherwise , the yielded features will be too close to those few seen classes and can not help ZSL/GZSL much . For example , in AWA2 dataset ( Xian et al. , 2018a ) , the unseen samples of “ rat ” are easily misclassified as seen classes of “ mouse ” or “ hamster ” because synthetic visual features for “ rat ” are doomed to be confused with seen visual features of those similar classes . The phenomenon inspires us to generate synthetic visual features for not only the given unseen classes but also the virtual unseen classes . These virtual classes provide a wider spectrum of support for unseen scenarios to improve ZSL/GZSL performance . Our other focal effort is to connect the concept of meta-learning ( Vinyals et al. , 2016 ; Finn et al. , 2017 ) with virtual classes in the training phase of the unseen expert . The most closely related approaches in ZSL are Li et al . ( 2019b ) ; Yu et al . ( 2020 ) ; Verma et al . ( 2020 ) ; Sung et al . ( 2018 ) ; Hu et al . ( 2018 ) . Verma et al . ( 2020 ) combines meta-learning and generative models to conquer limits in generative models . Li et al . ( 2019b ) mimics the inference scenario by randomly selecting seen classes as ” fake ” new classes in each episode . Yu et al . ( 2020 ) randomly splits seen classes into two sets to train and refine the model . In contrast , we simulate the ZSL inference scenario in each episode by creating virtual classes from seen classes using mixup . Owing to the training scenario more resembling the inference setting in ZSL , our model achieves better S2V embedding on unseen classes and obtains state-of-the-art ZSL performance on most datasets . Besides , Sung et al . ( 2018 ) learns a deep distance metric to classify samples . Hu et al . ( 2018 ) utilizes the correction module to assist classification . On the contrary , our model does not need to learn complex relationship between classes and additional assistant module to achieve good performance . 2.2 ATTENTION . Attention mechanism is widely used in ZSL ( Ji et al. , 2018 ; Xie et al. , 2019 ; Huynh & Elhamifar , 2020 ; Min et al. , 2020 ; Liu et al. , 2019 ) . Highlighting important local features and reducing noisy feature influence generate a more effective mapping between visual and semantic domains . Ji et al . ( 2018 ) proposes S2GA which utilizes semantic features to emphasize most informative visual local features . ( Xie et al. , 2019 ) employs AREN and ACSE to focus on the most important region in images . Huynh & Elhamifar ( 2020 ) applies dense attribute-attention to find the most discriminating image parts and embed them to semantic features individually . DVBE ( Min et al. , 2020 ) utilizes spatial and channel attention to maximize inter-class margin . While the attention to visual features is proved useful by the above-mentioned research efforts , relatively few attempts assess the importance of attention on semantic features . In the ZSL setting , each class contains only one semantic feature vector such that intra-class variations are neglected from the single semantic representation . For example , the object of interest can be occluded partly in the image or in the front of different backgrounds , and a unique class-wise semantic feature vector is hard to reflect such variances . Thus , attention to semantic features is worth further exploring in ZSL . Along this line , the most relevant work to ours is LFGAA ( Liu et al. , 2019 ) , which considers semantic prediction of samples . Different from LFGAA , our method puts attention on discriminating dimensions of semantic features and maps them to the visual domain . The effect of the proposed attention mechanism is to attract the visual representation of ground-truth class and repel others . Besides , our other main difference from LFGAA is the shortcut design in the attention model . We add a shortcut to prevent the model from overfitting on training data and retain discriminating dimensions for unseen class classification .

Summary: The paper proposes a framework for the GZSL using the meta-learning and attention mechanism. The image-guided attention on the semantic space helps to adapt the better class specific semantic information. The modified semantic space projected to the visual space and in the visual space, cosine similarity is measured. The paper learns separate expert for the seen and unseen classes. The unseen class expert is trained with the pseudo negative samples with pseudo negative labels. Meta-learning based training helps to learn the model when only a few examples per class are available.