PanRep: Universal node embeddings for heterogeneous graphs

1 INTRODUCTION . Learning node representations from heterogeneous graph data powers the success of many downstream machine learning tasks such as node classification ( Kipf & Welling , 2017 ) , and link prediction ( Wang et al. , 2017 ) . Graph neural networks ( GNNs ) learn node embeddings by applying a sequence of nonlinear operations parametrized by the graph adjacency matrix and achieve stateof-the-art performance in the aforementioned downstream tasks . The era of big data provides an opportunity for machine learning methods to harness large datasets ( Wu et al. , 2013 ) . Nevertheless , typically the labels in these datasets are scarce due to either lack of information or increased labeling costs ( Bengio et al. , 2012 ) . The lack of labeled data points hinders the performance of supervised algorithms , which may not generalize well to unseen data and motivates unsupervised learning . Unsupervised node embeddings may be used for downstream learning tasks , while the specific tasks are typically not known a priori . For example , node representations of the Amazon book graph can be employed for recommending new books as well as classifying a book ’ s genre . This work aspires to provide universal node embeddings , which will be applied in multiple downstream tasks and achieve comparable performance to their supervised counterparts . Although unsupervised learning has numerous applications , limited labels of the downstream task may be available . Refining the unsupervised universal representations with these labels could further increase the representation power of the embeddings . This can be achieved by fine-tuning the unsupervised model . Natural language processing methods have achieved state-of-the-art performance by applying such a fine-tuning framework ( Devlin et al. , 2018 ) . Fine-tuning pretrained models is beneficial compared to end-to-end supervised learning since the former typically generalizes better especially when labeled data are limited and decreases the inference time since typically just a few fine-tuning iterations typically suffice for the model to converge ( Erhan et al. , 2010 ) . This work introduces a framework for unsupervised learning of universal node representations on heterogenous graphs termed PanRep1 . It consists of a GNN encoder that maps the heterogenous graph data to node embeddings and four decoders , each capturing different topological and node feature 1Pan : Pangkosmios ( Greek for universal ) and Rep : Representation properties . The cluster and recover ( CR ) decoder exploits a clustering prior of the node attributes . The motif ( Mot ) decoder captures structural node properties that are encoded in the network motifs . The meta-path random walk ( MRW ) decoder promotes embedding similarity among nodes participating in a MRW and hence captures intermediate neighborhood structure . Finally , the heterogeneous information maximization ( HIM ) decoder aims at maximizing the mutual information among node local and the global representations per node type . These decoders model general properties of the graph data related to node homophily ( Gleich , 2015 ; Kloster & Gleich , 2014 ) or node structural similarity ( Rossi & Ahmed , 2014 ; Donnat et al. , 2018 ) . PanRep is solely supervised by the decoders and has no knowledge of the labels of the downstream task . The universal embeddings learned by PanRep are employed as features by models such as SVM ( Suykens & Vandewalle , 1999 ) or DistMult ( Yang et al. , 2014 ) to be trained for the downstream tasks . To further accommodate the case where limited labels are available for some downstream tasks we propose fine-tuning PanRep ( PanRep-FT ) . In this operational setting , PanRep-FT is optimized adhering to a task-specific loss . PanRep can be considered as a pretrained model for extracting node embeddings of heterogenous graph data . Figure 1 illustrates the two novel models . The contribution of this work is threefold . C1 . We introduce a novel problem formulation of universal unsupervised learning and design a tailored learning framework termed PanRep . We identify the following general properties of the heterogenous graph data : ( i ) the clustering of local node features , ( ii ) structural similarity among nodes , ( iii ) the local and intermediate neighborhood structure , ( iv ) and the mutual information among same-type nodes . We develop four novel decoders to model the aforementioned properties . C2 . We adjust the unsupervised universal learning framework to account for possible limited labels of the downstream task . PanRep-FT refines the universal embeddings and increases the model generalization capability . C3.We compare the proposed models to state-of-the-art supervised and unsupervised methods for node classification and link prediction . PanRep outperforms all unsupervised and certain supervised methods in node classification , especially when the labeled data for the supervised methods is small . PanRep-FT outperforms even supervised approaches in node classification and link prediction , which corroborates the merits of pretraining models . Finally , we apply our method on the drug-repurposing knowledge graph ( DRKG ) for discovering drugs for Covid-19 and identify several drugs used in clinical trials as possible drug candidates . 2 RELATED WORK . Unsupervised learning . Representation learning amounts to mapping nodes in an embedding space where the graph topological information and structure is preserved ( Hamilton et al. , 2017 ) . Typically , representation learning methods follow the encoder-decoder framework advocated by PanRep . Nevertheless , the decoder is typically attuned to a single task based on e.g. , matrix factorization ( Tang et al. , 2015 ; Ahmed et al. , 2013 ; Cao et al. , 2015 ; Ou et al. , 2016 ) , random walks ( Grover & Leskovec , 2016 ; Perozzi et al. , 2014 ) , or kernels on graphs ( Smola & Kondor , 2003 ) . Recently , methods relying on GNNs are increasingly popular for representation learning tasks ( Wu et al. , 2020 ) . GNNs typically rely on random walk-based objectives ( Grover & Leskovec , 2016 ; Hamilton et al. , 2017 ) or on maximizing the mutual information among node representations ( Veličković et al. , 2018b ) . Relational GNNs methods extend representation learning to heterogeneous graphs ( Dong et al. , 2017 ; Shi et al. , 2018 ; Shang et al. , 2016 ) . Relative to these contemporary works PanRep introduces multiple decoders to learn universal embeddings for heterogeneous graph data capturing the clustering of local node features , structural similarity among nodes , the local and intermediate neighborhood structure , and the mutual information among same-type nodes . Supervised learning . Node classification is typically formulated as a semi-supervised learning ( SSL ) task over graphs , where the labels for a subset of nodes are available for training ( Belkin et al. , 2004 ) . GNNs achieve state-of-the-art performance in SSL by utilizing regular graph convolution ( Kipf & Welling , 2017 ) or graph attention ( Veličković et al. , 2018a ) , while these models have been extended in heterogeneous graphs ( Schlichtkrull et al. , 2018 ; Fu et al. , 2020 ; Wang et al. , 2019b ) . Similarly , another prominent supervised downstream learning task is link prediction with numerous applications in recommendation systems ( Wang et al. , 2017 ) and drug discovery ( Zhou et al. , 2020 ; Ioannidis et al. , 2020 ) . Knowledge-graph ( KG ) embedding models rely on mapping the nodes and edges of the KG to a vector space by maximizing a score function for existing KG edges ( Wang et al. , 2017 ; Yang et al. , 2014 ; Zheng et al. , 2020 ) . RGCN models ( Schlichtkrull et al. , 2018 ) have been successful in link prediction and contrary to KG embedding models can further utilize node features . The universal embeddings extracted from PanRep without labeled supervision offer a strong competitive to these supervised approaches for both node classification and link prediction tasks . Pretraining . Pretraining models provides a significant performance boost compared to traditional approaches in natural language processing ( Devlin et al. , 2018 ; Radford et al. , 2019 ) and computer vision ( Donahue et al. , 2014 ; Girshick et al. , 2014 ) . Pretraining offers increased generalization capability especially when the labeled data is scarce and increased inference speed relative to end-toend training ( Devlin et al. , 2018 ) . Parallel to our work ( Hu et al. , 2020 ; Qiu et al. , 2020 ) proposed pretraining models for GNNs . These contemporary works supervise the models only using attribute or edge generation schemes without accounting for higher order structural similarity or maximizing the mutual information of the embeddings . Recently , ( Hu et al. , 2019 ) introduced a framework for pretraining GNNs for graph classification . Different than ( Hu et al. , 2019 ) that focuses on graph representations , PanRep aims at node prediction tasks and obtains node representations via capturing properties related to node homophily ( Gleich , 2015 ) or node structural similarity ( Rossi & Ahmed , 2014 ) . PanRep is a novel pretrained model for node classification and link prediction that requires significantly less labeled points to reach the performance of its fully supervised counterparts . 3 DEFINITIONS AND PROBLEM FORMULATION . A heterogeneous graph with T node types and R relation types is defined as G : = { { Vt } Tt=1 , { Er } Rr=1 } . The node types represent the different entities and the relation types represent how these entities are semantically associated to each other . For example , in the IMDB network , the node types correspond to actors , directors , movies , etc. , whereas the relation types correspond to directed-by and played-in relations . The number of nodes of type t is denoted by Nt and its associated nodal set by Vt : = { nt } Ntn=1 . The total number of nodes in G is N : = ∑t t=1Nt . The rth relation type , Er : = { ( nt , n′t′ ) ∈ Vt × Vt′ } , holds all interactions of a certain type among Vt and Vt′ and may represent that a movie is directed-by a director . Heterogenous graphs are typically used to represent KGs ( Wang et al. , 2017 ) . Each node nt is also associated with an F × 1 feature vector xnt . This feature may be a natural language embedding of the title of a movie . The nodal features are collected in a N × F matrix X . Note that certain node types may not have features and for these we use an embedding layer to represent their features . Unsupervised learning . Given G and X , the goal of representation learning is to estimate a function g such that H : = g ( X , G ) , where H ∈ RN×D represents the node embeddings and D is the size of the embedding space.Note that in estimating g , no labeled information is available . Universal representation learning . The universal representations H should perform well on different downstream tasks . Different node classification and link prediction tasks may arise by considering different number of training nodes and links and different label types , e.g. , occupation label or education level label . Consider I downstream task , for the universal representations H it holds that L ( i ) ( f ( i ) ( H ) , T ( i ) ) ≤ , i = 1 , . . . , I , ( 1 ) where L ( i ) , f ( i ) , and T ( i ) represent the loss function , learned classifier , and training set ( node labels or links ) for task i , respectively and is the largest error for all tasks . The goal of unsupervised universal representation learning is to learn H such that is small . While learning H , PanRep does not have knowledge of { L ( i ) , f ( i ) , T ( i ) } i . Nevertheless , by utilizing the novel decoder scheme PanRep achieves superior performance even compared to supervised approaches across tasks .

Summary: This paper proposes a universal and unsupervised GNN-based representation learning (node embedding pretraining) model named PanRep for heterogeneous graphs, which benefits a variety of downstream tasks such as node classification and link prediction. More specifically, employing an encoder similar to R-GCN, PanRep utilizes four different types of universal supervision signals for heterogeneous graphs, i.e. cluster and recover, motif, metapath random walk and heterogeneous info maximization, to better characterize the graph structures. PanRep can be further fine-tuned in a semi-supervised manner with limited labeled data, known as PanRep-FT, for specific applications. Experiments on benchmark datasets have shown the effectiveness and performance gain over other unsupervised and some supervised baseline approaches. One example use case is applied to COVID-19 drug repurposing to identify potential treatment candidates.