Zero-Shot Learning with Common Sense Knowledge Graphs

1 INTRODUCTION . Deep neural networks require large amounts of labeled training data to achieve optimal performance . This is a severe bottleneck , as obtaining large amounts of hand-labeled data is an expensive process . Zero-shot learning is a training strategy which allows a machine learning model to predict novel classes without the need for any labeled examples for the new classes ( Romera-Paredes & Torr , 2015 ; Socher et al. , 2013 ; Wang et al. , 2019 ) . Zero-shot models learn parameters for seen classes along with their class representations . During inference , new class representations are provided for the unseen classes . Previous zero-shot learning systems have used hand-engineered attributes ( Akata et al. , 2015 ; Farhadi et al. , 2009 ; Lampert et al. , 2014 ) , pretrained embeddings ( Frome et al. , 2013 ) and learned embeddings ( e.g . sentence embeddings ) ( Xian et al. , 2016 ) as class representations . Class representations in a zero-shot learning framework should satisfy the following properties : ( 1 ) they should adapt to unseen classes without requiring additional human effort , ( 2 ) they should provide rich features such that the unseen classes have sufficient distinguishing characteristics among themselves , ( 3 ) they should be applicable to a range of downstream tasks . Previous approaches for class representations have various limitations . On one end of the spectrum , attribute-based methods provide rich features but the attributes have to be fixed ahead of time for the unseen classes . On the other end of the spectrum , pretrained embeddings such as GloVe ( Pennington et al. , 2014 ) and Word2Vec ( Mikolov et al. , 2013 ) offer the flexibility of easily adapting to new classes but rely on unsupervised training on large corpora—which may not provide distinguishing characteristics necessary for zero-shot learning . Many methods lie within the spectrum and learn class representations for zero-shot tasks from descriptions such as attributes , text , and image prototypes . Existing approaches that have achieved state-of-the-art performance make task-specific adjustments and can not exactly be adapted to tasks in different domains ( Liu et al. , 2019a ; Verma et al. , 2020 ) . Methods using graph neural networks on the ImageNet graph to learn class representations have achieved strong performance on zero-shot object classification ( Kampffmeyer et al. , 2019 ; Wang et al. , 2018 ) . These methods are general-purpose , since we show that they can be adapted to other tasks as well . However , the ImageNet graph may not provide rich features suitable for a wide range of downstream tasks . In our work , we propose to learn class representations from common sense knowledge graphs . Common sense knowledge graphs ( Liu & Singh , 2004 ; Speer et al. , 2017 ; Tandon et al. , 2017 ; Zhang et al. , 2020 ) are an untapped source of explicit high-level knowledge that requires little human effort to apply to a range of tasks . These graphs have explicit edges between related concept nodes and provide valuable information to distinguish between different concepts . However , adapting existing zero-shot learning frameworks to learn class representations from common sense knowledge graphs is challenging in several ways . GCNZ ( Wang et al. , 2018 ) learns graph neural networks with a symmetrically normalized graph Laplacian , which not only requires the entire graph structure during training but also needs retraining if the graph structure changes , i.e. , GCNZ is not inductive . Common sense knowledge graphs can be large ( 2 million to 21 million edges ) and training a graph neural network on the entire graph can be prohibitively expensive . DGP ( Kampffmeyer et al. , 2019 ) is an inductive method and aims to generate expressive class representations but assumes a directed acyclic graph such as WordNet . Common sense knowledge graphs do not have a directed acyclic graph structure . To address these limitations , we propose ZSL-KG , a general-purpose framework with a novel transformer graph convolutional network ( TrGCN ) to learn class representations . Graph neural networks learn to represent the structure of graphs by aggregating information from each node ’ s neighbourhood . Aggregation techniques used in GCNZ , DGP , and most other graph neural network approaches are linear , in the sense that they take a ( possibly weighted ) mean or maximum of the neighbourhood features . To capture the complex information in the common sense knowledge graph , TrGCN learns a transformer-based aggregator to compute the non-linear combination of the node neighbours . A few prior works have considered LSTM-based aggregators ( Hamilton et al. , 2017a ) as a way to increase the expressivity of graph neural networks , but their outputs can be sensitive to the ordering imposed on the nodes in each neighborhood . For example , on the Animals with Attributes 2 dataset , we find that when given the same test image 10 times with different neighborhood orderings , an LSTM-based graph neural network outputs inconsistent predictions 16 % of the time ( Appendix A ) . One recent work considers trying to make LSTMs less sensitive by averaging the outputs over permutations , but this significantly increases the computational cost and provides only a small boost to prediction accuracy ( Murphy et al. , 2019 ) . In contrast , TrGCN learns a transformer-based aggregator , which is non-linear and naturally permutation invariant . Additionally , our framework is inductive , i.e. , the graph neural network can be executed on graphs that are different from the training graph , which is necessary for inductive zero-shot learning under which the test classes are unknown during training . We demonstrate the effectiveness of our framework on three zero-shot learning tasks in vision and language : object classification , intent classification , and fine-grained entity typing . We report new state-of-the-art accuracies on six zero-shot benchmark datasets ( Xian et al. , 2018a ; Farhadi et al. , 2009 ; Deng et al. , 2009 ; Coucke et al. , 2018 ; Gillick et al. , 2014 ; Weischedel & Brunstein , 2005 ) . ZSL-KG outperforms the state-of-the-art specialized method for each task by an average 1.7 accuracy points . ZSL-KG also outperforms GCNZ , the best general-purpose method on average by 5.3 accuracy points . Our ablation study on ZSL-KG with alternate graph neural networks shows that our transformer-based aggregator adds up to 2.8 accuracy points improvement on these tasks . In summary , our main contributions are the following : 1 . We propose to learn class representations from common sense knowledge graphs for zeroshot learning . 2 . We present ZSL-KG , a general-purpose framework based on graph neural networks with a novel transformer-based architecture . Our proposed architecture learns non-linear combination of the nodes neighbourhood and generates expressive class representations . 3 . ZSL-KG achieves new state-of-the-art accuracies on Animals with Attributes 2 ( Xian et al. , 2018a ) , aPY ( Farhadi et al. , 2009 ) , ImageNet ( Deng et al. , 2009 ) , SNIPS-NLU ( Coucke et al. , 2018 ) , Ontonotes ( Gillick et al. , 2014 ) , and BBN ( Weischedel & Brunstein , 2005 ) . 2 BACKGROUND . In this section , we summarize zero-shot learning and graph neural networks . 2.1 ZERO-SHOT LEARNING . Zero-shot learning has several variations ( Wang et al. , 2019 ; Xian et al. , 2018a ) . Our work focuses on inductive zero-shot learning , under which we do not have access to the unseen classes during training . We train a zero-shot classifier by optimizing over the seen classes . But , unlike traditional methods , zero-shot classifiers are trained along with class representations such as attributes , pretrained embeddings , etc . Recent approaches learn a class encoder φ ( y ) ∈ Rd to produce vector-valued class representations from an initial input , such as a string or other identifier of the class . ( In our case , y is a node in a graph and its k-hop neighborhood . ) During inference , the class representations are used to label examples with the unseen classes by passing the examples through an example encoder θ ( x ) ∈ Rd and predicting the class whose representation has the highest inner product with the example representation . Recent work in zero-shot learning commonly uses one of two approaches to learn the class encoder φ ( y ) . One approach uses a bilinear similarity function defined by a compatibility matrix W ∈ Rd×d ( Frome et al. , 2013 ; Xian et al. , 2018a ) : f ( θ ( x ) , W , φ ( y ) ) = θ ( x ) TWφ ( y ) . ( 1 ) The bilinear similarity function gives a score for each example-class pair . The parameters of θ , W , and φ are learned by taking a softmax over f for all possible seen classes y ∈ YS and minimizing either the cross entropy loss or a ranking loss with respect to the true labels . In other words , f should give a higher score for the correct class ( es ) and lower scores for the incorrect classes . W is often constrained to be low rank , to reduce the number of learnable parameters ( Obeidat et al. , 2019 ; Yogatama et al. , 2015 ) . Lastly , other variants of the similarity function add minor variations such as non-linearities between factors of W ( Socher et al. , 2013 ; Xian et al. , 2016 ) . The other common approach is to first train a neural network classifier in a supervised fashion . The final fully connected layer of this network has a vector representation for each seen class , and the remaining layers are used as the example encoder θ ( x ) . Then , the class encoder φ ( y ) is trained by minimizing the L2 loss between the representations from supervised learning and φ ( y ) ( Kampffmeyer et al. , 2019 ; Socher et al. , 2013 ; Wang et al. , 2018 ) . The class encoder that we propose in Section 3 can be plugged into either approach . 2.2 GRAPH NEURAL NETWORKS . The basic idea behind graph neural networks is to learn node embeddings that reflect the structure of the graph ( Hamilton et al. , 2017b ) . Consider the graph G = ( V , E , R ) , where V is the set of vertices with node features Xv and ( vi , r , vj ) ∈ E are the labeled edges and r ∈ R are the relation types . Graph neural networks learn node embeddings by iterative aggregation of the k-hop neighbourhood . Each layer of a graph neural network has two main components AGGREGATE and COMBINE ( Xu et al. , 2019 ) : a ( l ) v = AGGREGATE ( l ) ( { h ( l−1 ) u ∀u ∈ N ( v ) } ) ( 2 ) where a ( l ) v ∈ Rdl−1 is the aggregated node feature of the neighbourhood , h ( l−1 ) u is the node feature in neighbourhood N ( . ) of node v including a self loop . The aggregated node is passed to the COMBINE to generate the node representation h ( l ) v ∈ Rdl for the l-th layer : h ( l ) v = COMBINE ( l ) ( h ( l−1 ) v , a ( l ) v ) ( 3 ) h ( 0 ) v = xv where xv is the initial feature vector for the node . Previous works on graph neural networks for zero-shot learning have used GloVe ( Pennington et al. , 2014 ) to represent the initial features ( Kampffmeyer et al. , 2019 ; Wang et al. , 2018 ) .

Summary: This paper tackles zero-shot learning by leveraging the large-scale knowledge graph i.e., ConceptNet to propagate the knowledge learned from seen classes to unseen classes. The authors propose a novel propagation rule that aggregates node embeddings by the self-attention technique. It is infeasible to run GCN on such large-scale knowledge graph. Therefore they reduce the knolwedge graph by adopting a neighborhood sampling strategy based on random walks. The method is evaluated on multiple zero-shot learning tasks including object classification, intent classification and fine-grained entity typing. The SOTA results are achieved.