P^2IR: Universal Deep Node Representation via Partial Permutation Invariant Set Functions

1 INTRODUCTION . Graph node representation learning ( or graph embedding in some literature ) is to learn the numerical representation of each node in a graph by vectors in a Euclidean space , where the geometric relationship reflects the structure of the original graph . Nodes that are “ close ” in the graph are embedded to have similar vector representations ( Cai et al. , 2017 ) . The learned node vectors benefit a number of graph analysis tasks , such as node classification ( Bhagat et al. , 2011 ) , link prediction ( LibenNowell & Kleinberg , 2007 ) , community detection ( Fortunato , 2010 ) , and many others ( Hamilton et al. , 2017b ) . In order to preserve the node geometric relations in an embedded space , the similarity/proximity/distance of a node to its neighborhood is generally taken as input to different graph embedding approaches . For example , matrix-factorization approaches work on pre-defined pairwise similarity measures ( e.g. , different order of adjacency matrix ) ( Zhang et al. , 2018 ) . Deepwalk ( Perozzi et al. , 2014 ) , node2vec ( Grover & Leskovec , 2016 ) and other recent approaches ( Dong et al. , 2017 ) consider flexible , stochastic measure of node similarity by the node co-occurrence on short random walks over the graph ( Goyal & Ferrara , 2017 ) . Neighborhood autoencoder methods compress the information about a node ’ s local neighborhood that is described as a neighborhood vector containing the node ’ s pairwise similarity to all other nodes in the graph ( Wang et al. , 2016 ; Cao et al. , 2016 ; Bojchevski & Günnemann , 2018 ) . Neural network based approaches such as graph convolutional networks ( GCN ) and GraphSAGE apply convolution like functions on its surrounding neighborhood for aggregating neighborhood information ( Kipf & Welling , 2016a ; Hamilton et al. , 2017a ; Tu et al. , 2018 ) . However , most existing methods either explicitly or implicitly restrict the dependence form of each node to its neighbors and also the depth of neighbors . Therefore , some important topology structures within the graph and subtle dependence between neighbors may not be captured in the embedded space . For example , the methods in the family of stochastic walk lose the control of influence of further neighbors in a network since they usually utilize a fixed hyper-parameter as a window size , or need to specify the degree of neighborhood proximity ; GCN losses the flexibility of neighborhood information since it fixes the two-layer aggregation function for the representation computation . In this work , we propose a Partial Permutation Invariant Representation method ( P2IR ) by developing a new notion of partial permutation invariant set function , that can • learn node representation via a universal graph embedding function f , without pre-defining pairwise similarity , specifying random walk parameters , or choosing aggregation functions among element-wise mean , a max-pooling neural network , or LSTMs ; • capture the arbitrary relationship of each node to its neighbors ; • automatically decide the significance of nodes at different distances ; • be generally applied to any graphs from simple homogeneous graphs to heterogeneous graphs with complicated types of nodes . Evaluation results on benchmark data sets show that the proposed P2IR outperforms the state-of-theart approaches on producing node vectors for classification tasks . 2 RELATED WORK . The main difference among various graph embedding methods lies in how they define the “ closeness ” between two nodes ( Cai et al. , 2017 ) . First-order proximity , second-order proximity or even high-order proximity have been widely studied for capturing the structural relationship between nodes ( Tang et al. , 2015b ; Yang et al. , 2017 ; Zhu et al. , 2018 ) . Comprehensive reviews of graph embedding have been done by Cai et al . ( 2017 ) ; Hamilton et al . ( 2017b ) ; Goyal & Ferrara ( 2017 ) ; Yang et al . ( 2017 ) . In this section , we discuss the relevant graph embedding approaches in terms of how node closeness to neighboring nodes is measured , for highlighting our contribution on utilizing neighboring nodes in a most general manner . Matrix Analysis on Graph Embedding As early as 2011 , a spectral clustering method ( Tang & Liu , 2011 ) was proposed to take the eigenvalue decomposition of a normalized Laplacian matrix of a graph as an effective approach to obtain the embeddings of nodes . Other similar approaches choose different similar matrix ( from Laplacian matrix ) to make a trade-off between modeling the “ first-order similarity ” and “ higher-order similarity ” ( Cao et al. , 2015 ; Ou et al. , 2016 ; Rossi et al. , 2018 ) . Node content information can also be easily fused in the pairwise similarity measure , e.g. , in TADW ( Yang et al. , 2015 ) , as well as node label information , which resulting in semi-supervised graph embedding methods , e.g. , MMDW ( Tu et al. , 2016 ) . Recently , an arbitrary-order proximity preserved network embedding method is introduced in ( Zhang et al. , 2018 ) based on matrix eigen-decomposition , which is applied to a pre-defined high-order proximity matrix . For heterogeneous networks , Huang et al . ( 2017 ) proposed a label involved matrix analysis to learn the classification result of each vertex within a semi-supervised framework . Random Walk on a Graph to Node Representation Learning Both deepwalk ( Perozzi et al. , 2014 ) and node2vec ( Grover & Leskovec , 2016 ) are outstanding graph embedding methods to solve the node representation learning problem . They convert the graph structures into a sequential context format with random walk ( Lovász , 1993 ) . Thanks to the invention of Mikolov et al . ( 2013 ) for word representation learning of sentences , deepwalk inherited the learning framework for words representation learning in paragraphs to generate the representation of nodes in random walk context . And then node2vec evolved such the idea with additional hyper-parameters tuning for the trade-off between DFS and WFS to control the direction of random walk . Struc2vec ( Ribeiro et al. , 2017 ) also utilizes the multilayer graph to construct the graph node representations . Gao & Huang ( 2018 ) propose a self-paced network embedding by introducing a dynamic negative sampling method to select difficult negative context nodes in the training process . Planetoid ( Yang et al. , 2016 ) proposed a semi-supervised learning framework by guiding random walk with available node label information . The heterogeneity of graph nodes is often handled by a heterogeneous random walk procedure ( Dong et al. , 2017 ) , or selected relation pairs ( Chang et al. , 2015 ) . Tang et al . ( 2015a ) considered the predictive text embedding problem on a large-scale heterogeneous text network and the proposed method is also based on pre-defined heterogeneous random walks . Neighborhood Encoders to Graph Embedding There are also methods focusing on aggregating or encoding the neighbors ’ information to generate node embeddings . DNGR ( Cao et al. , 2016 ) and SDNE ( Wang et al. , 2016 ) introduce the autoencoder to construct the similarity function between the neighborhood vectors and the embedding of the target node . DNGR defines neighborhood vectors based on random walks and SDNE introduces adjacency matrix and Laplacian eigenmaps to the definition of neighborhood vectors . GraphWave ( Donnat et al. , 2018 ) learns the representation of each node ’ s neighborhood via leveraging heat wavelet diffusion patterns . Although the idea of autoencoder is a great improvement , these methods are painful when the scale of the graph is up to millions of nodes . Therefore , methods with neighborhood aggregation and convolutional encoders are involved to construct a local aggregation for node embedding , such as GCN ( Kipf & Welling , 2016a ; b ; Schlichtkrull et al. , 2018 ; van den Berg et al. , 2017 ) , FastGCN ( Chen et al. , 2018 ) , column networks ( Pham et al. , 2017 ) , the GraphSAGE algorithm ( Hamilton et al. , 2017a ) , GAT ( Velickovic et al. , 2017 ) and a recent DRNE ( Tu et al. , 2018 ) method using layer normalized LSTM to approximate the embedding of a target node by the aggregation of its neighbors ’ embeddings . The main idea of these methods is involving an iterative or recursive aggregation procedure e.g. , convolutional kernels or pooling procedures to generate the embedding vectors for all nodes and such aggregation procedures are shared by all nodes in a graph . The above-mentioned methods work differently on using neighboring nodes for node representation learning . They require on pre-defining pairwise similarity measure between nodes , or specifying random walk parameters , or choosing aggregation functions . In practice , it takes non-trivial effort to tune these parameters or try different measures , especially when graphs are complicated with nodes in multiple types , i.e. , heterogeneous graphs . This work hence targets on making neighboring nodes play their roles in a most general manner such that their contributions are learned but not user-defined . The resultant embedding method has the flexibility to work on any types of homogeneous and heterogeneous graph . Our proposed method P2IR has a natural advantage on avoiding any manual manipulation of random walking strategies or designs for the relationships between different types of nodes . To the invention of set functions by Zaheer et al . ( 2017 ) , all existing valid mapping strategies from neighborhood to the target nodes can be represented by the set functions which are learnt by P2IR automatically . 3 THE PROPOSED P2IR METHOD In this section , we first formally define the problem , followed by introducing a new notion - partial permutation invariant function set function . This section ends up with the overall P2IR method . 3.1 PROBLEM SETUP . We target on designing graph embedding models for the most general graph that may include K different types of nodes ( k=1 for homogeneous graphs ) . Formally , a graph G = { V , E } , where the node set V = ∪Kk=1Vk , i.e. , V is composed of K disjoint types of nodes . One instance of such a graph is the academic publication network , which includes different types of nodes for papers , publication venues , author names , author affiliations , research domains etc . Given such a graph G , our goal is to learn the embedding vector for each node in this graph . 3.2 THE PROPOSED UNIVERSAL GRAPH EMBEDDING MODEL . As we know , the position of a node in the embedded space is collaboratively determined by its neighboring nodes . Therefore , the key issue to address for graph embedding is how to model the dependence of each node to its neighbors . Most existing approaches need to pre-specify the neighbors ( usually via the distance on the graph ) , and either explicitly or implicitly define a specific form to characterize the dependence between each node and its neighbors . We propose a universal graph embedding model that neither requires to pre-define neighbors nor to specify the dependence form between each node and its neighbors . The embedding vector xv of node v ∈ Vk can be represented by its neighbors ’ embedding vectors via a function f xv = f ( Xv1 , X v 2 , · · · , XvK ) , ∀v ∈ Vk , ∀k ∈ { 1 , 2 , . . . , K } whereXvk is a matrix with column vectors corresponding to the embedding of node v ’ s neighbors in type k. Note that the neighbors can include step-1 ( or immediate ) neighbors , step-2 neighbors , and even further neighbors . How to characterize an arbitrary dependence to neighbors ? The key observation is that all neighboring nodes reachable from a target node at the same step are not distinguishable from the view of the target node . Therefore , function f ( · ) should be a partially permutation invariant function . That is , if we swap any elements in aXvk , the function value remains the same . Unfortunately , the set function is not simply learn-able because the permutation property is hard to guarantee directly . One straightforward idea to represent the partially permutation invariant function is to define it in the following form f ( Xv1 , · · · , XvK ) : = ∑ P1∈P|V1| ∑ P2∈P|V2| · · · ∑ PK∈P|VK | T ( Xv1P1 , · · · , XvKPK ) ( 1 ) where Pn denotes the set of n-dimensional permutation matrices . Xv1P1 is to permute the columns inXv1 . It is easy to verify that the function defined in equation 1 is partially permutation invariant , but it is intractable because it involves ∏N k=1 ( |Vk| ! ) “ sum ” items . Our solution of learning function f is based on the following important theorem , which gives a neat way to represent any partial permutation invariant function . The proof is provided in the Appendix . Theorem 3.1 . [ Partial permutation invariant function representation theorem ] Let f be a continuous real-valued function defined on a compact set X with the following form f x1,1 , x1,2 , · · · , x1 , N1︸ ︷︷ ︸ G1 , x2,1 , x2,2 , · · · , x2 , N2︸ ︷︷ ︸ G2 , · · · , xK,1 , xK,2 , · · · , xK , NK︸ ︷︷ ︸ GK .

Summary: The paper proposes a formulation for taking care of neighborhood of different distances for graph embedding. It makes use of a notion called permutation invariant function which defined as a function where if we swap any features in the inputs, the function value remains the same. Given this, they make two contributions to make the consideration of neighborhood of different distances for graph embedding possible. First, they make the assumption that the contribution of neighbours of same steps should be the same and thus permutable in defining how the embedding function of a node is depending on this neighbours. Another one is the use of 1-d NN for estimating the contribution from 1-step, 2-step and up to infinite-step. Then, the overall problem formulation is defined and can be learned using SDG.