Isometric Propagation Network for Generalized Zero-shot Learning

1 INTRODUCTION . One primary challenge on the track from artificial intelligence to human-level intelligence is to improve the generalization capability of machine learning models to unseen problems . While most supervised learning methods focus on generalization to unseen data from training task/classes , zeroshot learning ( ZSL ) ( Larochelle et al. , 2008 ; Lampert et al. , 2014 ; Xian et al. , 2019a ) has a more ambitious goal targeting the generalization to new tasks and unseen classes . In the context of image classification , given images from some training classes , ZSL aims to classify new images from unseen classes with zero training image available . Without training data , it is impossible to directly learn a mapping from the input space to the unseen classes . Hence , recent works introduced learning to map between the semantic space and visual space ( Zhu et al. , 2019a ; Li et al. , 2017 ; Jiang et al. , 2018 ) so that the query image representation and the class representation can be mapped to a shared space for comparison . However , learning to align the representation usually leads to overfitting on seen classes ( Liu et al. , 2018 ) . One of the reasons is that in most zero-shot learning settings , the number of images per class is hundreds while the semantic information provided for one class is only limited to one vector . Thus , the mapping function is easily overfitting to the seen classes when trained using the small set of attributes . The learned model typically has imbalanced performance on samples from the seen classes and unseen classes , i.e. , strong when predicting samples from the seen classes but struggles in the prediction for samples in the unseen classes . In this paper , we propose “ Isometric Propagation Network ( IPN ) ” which learns to dynamically interact between the visual space and the semantic space . Within each space , IPN uses an attention module to generate a category graph , and then perform multiple steps of propagation on the graph . In every step , the distribution of the attention scores generated in each space are regularized to be isometric so that implicitly aligns the relationships between classes in two spaces . Due to different motifs of the generated graphs , the regularizer for the isometry between the two distributions are provided with sufficient training supervisions and can potentially generalize better on unseen classes . To get more diverse graph motifs , we also apply episodic training which samples different subset of classes as the nodes for every training episode rather than the whole set of seen classes all the time . To evaluate IPN , we compared it with several state-of-the-art ZSL methods on three popular ZSL benchmarks , i.e. , AWA2 ( Xian et al. , 2019a ) , CUB ( Welinder et al. , 2010 ) and aPY ( Farhadi et al. , 2009 ) . To test the generalization ability on more diverse unseen classes rather than similar classes to the seen classes , e.g. , all animals for AWA2 and all birds for CUB , we also evaluate IPN on two new large-scale datasets extracted from tieredImageNet ( Ren et al. , 2018 ) , i.e. , tieredImageNet-Mixed and tieredImageNet-Segregated . IPN consistently achieves state-of-the-art performance in terms of the harmonic mean of the accuracy on unseen and seen classes . We show the ablation study of each component in our model and visualize the final learned representation in each space in Figure 1 . It shows the class representations in each space learned by IPN are evenly distributed and connected smoothly between the seen and unseen classes . The two spaces can also reflect each other . 2 RELATED WORKS . Most ZSL methods can be categorized as either discriminative or generative . Discriminative methods generate the classifiers for unseen classes from its attributes by learning a mapping from a input image ’ s visual features to the associated class ’ s attributes on training data of seen classes ( Frome et al. , 2013 ; Akata et al. , 2015a ; b ; Romera-Paredes & Torr , 2015 ; Kodirov et al. , 2017 ; Socher et al. , 2013 ; Cacheux et al. , 2019 ; Li et al. , 2019b ) , or by relating similar classes according to their shared properties , e.g. , semantic features ( Norouzi et al. , 2013 ; Chi et al. , 2021 ) or phantom classes ( Changpinyo et al. , 2016 ) , etc . In contrast , generative methods employ the recently popular image generation models ( Goodfellow et al. , 2014 ) to synthesize images of unseen classes , which reduces ZSL to supervised learning ( Huang et al. , 2019 ) but usually increases the training cost and model complexity . In addition , the study of unsupervised learning ( Fan et al. , 2021 ; Liu et al. , 2021 ) , meta learning ( Santoro et al. , 2016 ; Liu et al. , 2019b ; a ; Ding et al. , 2020 ; 2021 ) , or architecture design ( Dong et al. , 2021 ; Dong & Yang , 2019 ) can also be used improve the representation ability of visual or semantic features . Alignment between two spaces in ZSL Frome et al . ( 2013 ) ; Socher et al . ( 2013 ) ; Zhang et al . ( 2017 ) proposed to transform representations either to the visual space or the semantic space for distance for comparison . Li et al . ( 2017 ) introduces how to optimize the semantic space so that the two spaces can be better aligned . Jiang et al . ( 2018 ) ; Liu et al . ( 2020 ) tries to map the class representations from two spaces to another common space . Comparatively , our IPN does not align two spaces by feature transformation but by regularizing the isometry of the intermediate output from two propagation model . Graph in ZSL Graph structures have already been explored for their benefits to some discriminative ZSL models , e.g. , Wang et al . ( 2018 ) ; Kampffmeyer et al . ( 2019 ) learn to generate the weights of a fully-connected classifier for a given class based on its semantic features computed on a pre- defined semantic graph using a graph convolutional networks ( GCN ) . They require ( 1 ) a pre-trained CNN whose last fully-connected layer provides the ground truth of the classifiers ZSL aims to generate ; and ( 2 ) an existing knowledge graph as the input of GCN . By contrast , our method learns to automatically generate the graph and the class prototypes composing the support set of a distancebased classifier . Hence , our method does not require a pre-trained CNN to provide a ground truth for the classifier . In addition , we train a modified graph attention network ( GAN ) ( rather than a GCN ) to generate the prototypes , so it learns both the class adjacency and edges by itself and can provide more accurate modeling of class dependency in scenarios when no knowledge graph is provided . Tong et al . ( 2019 ) and Xie et al . ( 2020 ) also leverage the idea of graph to transform the feature extracted from the CNN while our graph serves as the paths for propagation . It requires an off-the-shelf graph generated by ConceptNet ( Speer et al. , 2016 ) , while IPN applies prototype propagation and learns to construct the graph in an end-to-end manner . Attention in ZSL The attention module has been applied for zero-shot learning for localization . Liu et al . ( 2019c ) introduces localization on the semantic information and Xie et al . ( 2019 ) proposes localization on the visual feature . Zhu et al . ( 2019b ) further extends it to multi-localization guided by semantic . Different from them , our attention serves as generating the weights for propagation . Our episodic training strategy is inspired by the meta-learning method from ( Santoro et al. , 2016 ) . Similar methods have been broadly studied in meta-learning and few-shot learning approaches ( Finn et al. , 2017 ; Snell et al. , 2017 ; Sung et al. , 2018 ) but rarely studied for ZSL in previous works . In this paper , we show that training ZSL model as a meta-learner can improve the generalization performance and mitigate overfitting . 3 PROBLEM SETTING . We consider the following setting of ZSL . The training set is composed of data-label pairs ( x , y ) , where x ∈ X is a sample from class y , and y must be one of the seen classes in Yseen . During the test stage , given a test sample x of class y , the ZSL model achieved in training is expected to produce the correct prediction of y . In order to relate the unseen classes in test to the seen classes in training , we are also given the semantic embedding sy of each class y ∈ Yseen ∪ Yunseen . We further assume that seen and unseen classes are disjoint , i.e. , Yseen∩Yunseen = ∅ . The test samples come from both Yseen and Yunseen in generalized zero-shot learning . 4 ISOMETRIC PROPAGATION NETWORK . In this paper , we bridge the two spaces via class prototypes in both the semantic and visual spaces , i.e. , a semantic prototype P sy and a visual prototype P v y for each class y . In order to generate high-quality prototypes , we develop a learnable scheme called “ Isometric Propagation Network ( IPN ) ” to iteratively refine the prototypes by propagating them on a graph of classes so each prototype is dynamically determined by the samples and attributes of its related classes . In this section , we will first introduce the architecture of IPN . An illustration of IPN is given in Figure 2 . Then , we will elaborate the episodic training strategy , in which a consistency loss is minimized for isometric alignment between the two spaces . 4.1 PROTOTYPE INITIALIZATION . Given a task T defined on a subset of classes YT with training data DT , IPN first initializes the semantic and visual prototype for each class y ∈ YT . The semantic prototype of class y is transformed from its attributes sy . Specifically , we have drawn on the expert modules from ( Zhang & Shi , 2019 ) to initialize the semantic prototypes P sy [ 0 ] . In previous works ( Snell et al. , 2017 ; Santoro et al. , 2016 ) , the visual prototype of a class is usually computed by averaging the features of images from the class . In IPN , for each seen class y ∈ YT ∩ Yseen , we initialize its visual prototype P vy [ 0 ] by P vy [ 0 ] = W |DTy | ∑ ( xi , yi ) ∈DTy fcnn ( xi ) , D T y , { ( xi , yi ) ∈ DT : yi = y } , ( 1 ) where fcnn ( · ) is a backbone convolutional neural nets ( CNN ) extracting features from raw images , and W is a linear transformation aiming to change the dimension of the visual prototype to be the same as semantic prototypes . We argue that W is necessary because ( 1 ) the visual feature produced by the CNN is usually in high dimension , and reducing its dimension can save significant computations during propagation ; ( 2 ) keeping the dimensions of the visual and semantic prototypes the same can avoid the dominance of either of them in the model . During test , we initialize the visual prototype of each unseen class y ∈ YT∩Yunseen as the weighted sum of its K-nearest classes from Yseen in the semantic space1 , i.e. , P vy [ 0 ] = W · ∑ z∈Ny as ( P sy [ 0 ] , P s z [ 0 ] ) · P vz , ( 2 ) where W is the same transformation from Eq . ( 1 ) , and Ny is the set of the top-K ( K is a hyperparameter ) seen classes2 with the largest similarities cs ( P sy [ 0 ] , P s z [ 0 ] ) to class y in semantic space .

Summary: The authors propose a novel computational pipeline to tackle a well-known problem in zero-shot learning: although multiple visual instances are available for the classes and categories to be recognized, one and only one semantic embedding is available to describe the classes/categories while using side information like attributes or relevant textual information. To cope with that problem, authors learn visual and semantic prototypes which are then adopted to perform gradient descent over a graph in which the topological relationship among similar/dissimilar classes are preserved. In the experimental validation, the proposed method shows its superiority among a number of prior methods in zero-shot learning, including discriminative and generative methods.