Dual Graph Representation Learning

Graph representation learning embeds nodes in large graphs as low-dimensional vectors and benefit to many downstream applications . Most embedding frameworks , however , are inherently transductive and unable to generalize to unseen nodes or learn representations across different graphs . Inductive approaches , such as GraphSAGE , neglect different contexts of nodes and can not learn node embeddings dually . In this paper , we present a context-aware unsupervised dual encoding framework , CADE , to generate representation of nodes by combining real-time neighborhood structure with neighbor-attentioned representation , and preserving extra memory of known nodes . Experimently , we exhibit that our approach is effective by comparing to state-of-the-art methods . 1 INTRODUCTION . The study of real world graphs , such as social network analysis ( Hamilton et al . ( 2017a ) ) , molecule screening ( Duvenaud et al . ( 2015 ) ) , knowledge base reasoning ( Trivedi et al . ( 2017 ) ) , and biological protein-protein networks ( Zitnik & Leskovec ( 2017 ) ) , evolves with the development of computing technologies . Learning vector representations of graphs is effective for a variety of prediction and graph analysis tasks ( Grover & Leskovec ( 2016 ) ; Tang et al . ( 2015 ) ) . High-dimensional information about neighbors of nodes are represented by dense vectors , which can be fed to off-the-shelf approaches to tasks , such as node classification ( Wang et al . ( 2017 ) ; Bhagat et al . ( 2011 ) ) , link prediction ( Perozzi et al . ( 2014 ) ; Wei et al . ( 2017 ) ) , node clustering ( Nie et al . ( 2017 ) ; Ding et al . ( 2001 ) ) , recommender systems ( Ying et al . ( 2018a ) ) and visualization ( Maaten & Hinton ( 2008 ) ) . There are mainly two types of models for graph representation learning . Transductive approaches ( Perozzi et al . ( 2014 ) ; Grover & Leskovec ( 2016 ) ; Tang et al . ( 2015 ) ) are able to learn representations of existing nodes but unable to generalize to new nodes . However , in real-world evolving graphs such as social networks , new users will join and must be represented . Inductive approaches were proposed to address this issue . GraphSAGE ( Hamilton et al . ( 2017b ) ) , a hierarchical sampling and aggregating framework , successfully leverages feature information to generate embeddings of the new nodes . However , GraphSAGE has its own faults . Firstly , it samples all neighborhood nodes randomly and uniformly ; secondly , it treats the output of encoder as the final representation of node . Based on the hierarchical framework of GraphSAGE , GAT ( Velickovic et al . ( 2017 ) ) uses given class labels to guide attention over neighborhood so as to aggregate useful feature information . However , without knowledge of ground-truth class labels , it is difficult for unsupervised approaches to apply attention . To address this issue , we introduce a dual encoding framework for unsupervised inductive representation learning of graphs . Instead of learning self-attention over neighborhoods of nodes , we exploit the bi-attention between representations of two nodes that co-occur in a short random-walk ( which we call a positive pair ) . In Figure 1 , we illustrate how nodes are embedded into low-dimensional vectors , where each node v has an optimal embeddings ov . Yet the direct output of encoder zv of GraphSAGE could be located anywhere . Specifically , given feature input from both sides of a positive pair ( v , vp ) , a neural network is trained to encode the pair into K different embeddings zkv and z k vp through different sampled neighborhoods or different encoding functions . Then , a bi-attention layer is applied to generate the most adjacent matches zv|vp and zvp|v , which will be referred as dual-representations . By putting most attention on the pair of embeddings with smallest difference , dual representation of nodes with less deviation will be generated , which can be visualized as zv|· in Figure 1 . GraphSAGE naively assumes that unseen graph structure should be ( easily ) represented by known graphs data . We combine the ground truth structure and the learned dual-encoder to generate final representation . Unseen nodes can be represented based on their neighborhood structure . Current inductive approaches have no direct memory of the training nodes . We combine the idea of both transductive and inductive approaches via associating an additive global embedding bias to each node , which can be seen as a memorable global identification of each node in training sets . Our contributions can be summarized as follows : • we introduce a dual encoding framework to produce context-aware representation for nodes , and conduct experiments to demonstrate its efficiency and effectiveness ; • we apply bi-attention mechanism for graph representation dual learning , managing to learn dual representation of nodes more precisely ; • we combine the training of transductive global bias with inductive encoding process , as memory of nodes that are already used for training . 2 RELATED WORK . Following ( Cai et al . ( 2018 ) , Kinderkhedia ( 2019 ) and Goyal & Ferrara ( 2018 ) ) , there are mainly two types of approaches : 2.1 NETWORK EMBEDDING . For unsupervised embedding learning , DeepWalk ( Perozzi et al . ( 2014 ) ) and node2vec ( Grover & Leskovec ( 2016 ) ) are based on random-walks extending the Skip-Gram model ; LINE ( Tang et al . ( 2015 ) ) seeks to preserve first- and second-order proximity and trains the embedding via negative sampling ; SDNE ( Wang et al . ( 2016 ) ) jointly uses unsupervised components to preserve secondorder proximity and expolit first-order proximity in its supervised components ; TRIDNR ( Pan et al . ( 2016 ) ) , CENE ( Sun et al . ( 2016 ) ) , TADW ( Yang et al . ( 2015 ) ) , GraphSAGE ( Hamilton et al . ( 2017b ) ) utilize node attributes and potentially node labels . Convolutional neural networks are also applied to graph-structured data . For instance , GCN ( Kipf & Welling ( 2017 ) ) proposed an simplified graph convolutional network . These graph convolutional network based approaches are ( semi ) supervised . Recently , inductive graph embedding learning ( Hamilton et al . ( 2017b ) Velickovic et al . ( 2017 ) Bojchevski & Günnemann ( 2017 ) Derr et al . ( 2018 ) Gao et al . ( 2018 ) , Li et al . ( 2018 ) , Wang et al . ( 2018 ) and Ying et al . ( 2018b ) ) produce impressive performance across several large-scale benchmarks . 2.2 ATTENTION . Attention mechanism in neural processes have been extensively studied in neuroscience and computational neuroscience ( Itti et al . ( 1998 ) ; Desimone & Duncan ( 1995 ) ) and frequently applied in deep learning for speech recognition ( Chorowski et al . ( 2015 ) ) , translation ( Luong et al . ( 2015 ) ) , question answering ( Seo et al . ( 2016 ) ) and visual identification of objects ( Xu et al . ( 2015 ) ) . Inspired by ( Seo et al . ( 2016 ) and Abu-El-Haija et al . ( 2018 ) ) , we construct a bi-attention layer upon aggregators to capture useful parts of the neighborhood . 3 MODEL . Let G = { V , E , X } be an undirected graph , where a set of nodes V are connected by a set of edges E , and X ∈ R|V |×f is the attribute matrix of nodes . A global embedding bias matrix is denoted by B ∈ R|V |×d , where a row of B represents the d-dimensional global embedding bias of a node . The hierarchical layer number , the embedding output of the l-th layer and the final output embedding are denoted by L , hl and z , respectively . 3.1 CONTEXT-AWARE INDUCTIVE EMBEDDING ENCODING . The embedding generation process is described in Algorithm 1 . Assume that the dual encoder is trained and parameters are fixed . Algorithm 1 Context-Aware Dual-Encoding ( CADE ) input : the whole graph G = ( V , E ) ; the feature matrix X ; the trained DualENC output : learned embeddings z ; 1 : Run random walks on G to gain a set of positive pair P ; 2 : Zv ← ∅ , ∀v ∈ V 3 : for ( v , vp ) ∈ P do 4 : zv , zvp = DualENC ( v , vp , G , X ) ; 5 : Zv ← Zv ∪ zv 6 : Zvp ← Zvp ∪ zvp 7 : end for 8 : for v ∈ V do 9 : zv = Mean ( Zv ) ; 10 : end for After training , positive pairs are collected by random walks on the whole dataset . The features of each positive pair are passed through a dual-encoder . Embeddings of nodes are generated so that the components of a pair are related and adjacent to each other . 3.2 DUAL-ENCODER WITH MULTI-SAMPLING . In this subsection , we explain the dual encoder . Algorithm 2 DualENC input : Training graph G ( V , E ) ; node attributes X ; global embedding bias matrix B ; sampling times K ; positive node pair ( v , vp ) ; output : adjacent embeddings zv and zvp ; 1 : For node v and vp , generate K embeddings , hv , hvp , using a base encoder SAGB 2 : for i , j ∈ { 1 , ... , K } do 3 : Si , j ← α ( hvi , hvpj ) 4 : end for 5 : softmax on flattened similarity matrix S : Si , j ← e Si , j∑K , K 0,0 e Si , j 6 : calculate attention av and avp : avi ← ∑K j=1 Si , j , avpj ← ∑K i=1 Si , j 7 : zv ← ∑K t=1 avkh L vk 8 : zvp ← ∑K t=1 avpkh L vpk In the hierarchical sampling and aggregating framework ( Hamilton et al . ( 2017b ) ) , it is challenging and vital to select relevant neighbor with the layer goes deeper and deeper . For example , given word ” mouse ” and its positive node ” PC ” , it is better to sample ” keyboard ” , instead of ” cat ” , as a neighbor node . However , to sample the satisfying node according to heuristic rules layer by layer is very time consuming , and it is difficult to learn attention over neighborhood for unsupervised embedding learning . As a matter of fact , these neighbor nodes are considered to be useful because they are more welcome to be sampled as input so as to produce more relevant output of the dual-encoder . Therefore , instead of physically sampling these neighbor nodes , in Step 2 to Step 6 in Algorithm 2 , we directly apply a bi-attention layer on the two sets of embedding outputs with different sampled neighborhood feature as input , so as to locate the most relevant representation match , as a more efficient approach to exploring the most useful neighborhood . We use the hierarchical sampling and aggregating framework as a base encoder in our experiments , but it can also be designed in many other ways . Given node v , and node vp as a positive node pair , after sampling and aggregating for K times , we have K different representations hvk/hvpk corresponding to K different sampled neighborhoods , their similarity matrix can be calculated by Si , j = α ( hvi , hvpj ) , i , j = 1 , ... , K ( 1 ) where α represents an dot-product operation . our goal is to find the closest neighborhood match between v and vp within K ×K possibilities , so we apply softmax on the flattened similarity matrix , and sum up by row ( column ) . In this way , we manage to gain different attention over K neighborhoods of v ( vp ) with respect to vp ( v ) , Sij ← exp ( Sij ) ∑K , K 0,0 exp ( Sij ) ( 2 ) avi = K∑ j=1 Sij ( 3 ) avpj = K∑ i=1 Sij ( 4 ) and sum up the lth layer representations with attention as the dual-encoder outputs , zv = K∑ k=1 avkhvk ( 5 ) zvp = K∑ k=1 avpkhvpk ( 6 ) To train our encoder before using it to generate final representations of nodes , we apply a typical pair reconstruction loss function with negative sampling Hamilton et al . ( 2017b ) : JG ( zv ) = −log ( σ ( zTv zvp ) ) −Q · Evn∼Pn ( v ) log ( σ ( −z T v zvn ) ) ( 7 ) where node vp co-occurs with v on fixed-length random walk ( Perozzi et al . ( 2014 ) ) , σ is the sigmoid function , Pn is a negative sampling distribution , Q defines the number of negative samples . Note that zv and zvp are dual representation to each other while zvn represents the direct encoder output of negative sample vn .

Summary: This paper extends GraphSAGE in several dimensions: 1) applying attention when aggregating neighbors (already used by GAT and many other approached); 2) Ensembling node embedding by applying DualENC multiple times on positive pairs selected by random walk (this is doing aggregation of neighborhood again); and 3) adding global bias. All this makes the proposed method an incremental extension of existing solutions.  There is no theoretical justification why these extensions should work.