Dual Graph Representation Learning

Graph representation learning embeds nodes in large graphs as low-dimensional vectors and benefit to many downstream applications . Most embedding frameworks , however , are inherently transductive and unable to generalize to unseen nodes or learn representations across different graphs . Inductive approaches , such as GraphSAGE , neglect different contexts of nodes and can not learn node embeddings dually . In this paper , we present a context-aware unsupervised dual encoding framework , CADE , to generate representation of nodes by combining real-time neighborhood structure with neighbor-attentioned representation , and preserving extra memory of known nodes . Experimently , we exhibit that our approach is effective by comparing to state-of-the-art methods . 1 INTRODUCTION . The study of real world graphs , such as social network analysis ( Hamilton et al . ( 2017a ) ) , molecule screening ( Duvenaud et al . ( 2015 ) ) , knowledge base reasoning ( Trivedi et al . ( 2017 ) ) , and biological protein-protein networks ( Zitnik & Leskovec ( 2017 ) ) , evolves with the development of computing technologies . Learning vector representations of graphs is effective for a variety of prediction and graph analysis tasks ( Grover & Leskovec ( 2016 ) ; Tang et al . ( 2015 ) ) . High-dimensional information about neighbors of nodes are represented by dense vectors , which can be fed to off-the-shelf approaches to tasks , such as node classification ( Wang et al . ( 2017 ) ; Bhagat et al . ( 2011 ) ) , link prediction ( Perozzi et al . ( 2014 ) ; Wei et al . ( 2017 ) ) , node clustering ( Nie et al . ( 2017 ) ; Ding et al . ( 2001 ) ) , recommender systems ( Ying et al . ( 2018a ) ) and visualization ( Maaten & Hinton ( 2008 ) ) . There are mainly two types of models for graph representation learning . Transductive approaches ( Perozzi et al . ( 2014 ) ; Grover & Leskovec ( 2016 ) ; Tang et al . ( 2015 ) ) are able to learn representations of existing nodes but unable to generalize to new nodes . However , in real-world evolving graphs such as social networks , new users will join and must be represented . Inductive approaches were proposed to address this issue . GraphSAGE ( Hamilton et al . ( 2017b ) ) , a hierarchical sampling and aggregating framework , successfully leverages feature information to generate embeddings of the new nodes . However , GraphSAGE has its own faults . Firstly , it samples all neighborhood nodes randomly and uniformly ; secondly , it treats the output of encoder as the final representation of node . Based on the hierarchical framework of GraphSAGE , GAT ( Velickovic et al . ( 2017 ) ) uses given class labels to guide attention over neighborhood so as to aggregate useful feature information . However , without knowledge of ground-truth class labels , it is difficult for unsupervised approaches to apply attention . To address this issue , we introduce a dual encoding framework for unsupervised inductive representation learning of graphs . Instead of learning self-attention over neighborhoods of nodes , we exploit the bi-attention between representations of two nodes that co-occur in a short random-walk ( which we call a positive pair ) . In Figure 1 , we illustrate how nodes are embedded into low-dimensional vectors , where each node v has an optimal embeddings ov . Yet the direct output of encoder zv of GraphSAGE could be located anywhere . Specifically , given feature input from both sides of a positive pair ( v , vp ) , a neural network is trained to encode the pair into K different embeddings zkv and z k vp through different sampled neighborhoods or different encoding functions . Then , a bi-attention layer is applied to generate the most adjacent matches zv|vp and zvp|v , which will be referred as dual-representations . By putting most attention on the pair of embeddings with smallest difference , dual representation of nodes with less deviation will be generated , which can be visualized as zv|· in Figure 1 . GraphSAGE naively assumes that unseen graph structure should be ( easily ) represented by known graphs data . We combine the ground truth structure and the learned dual-encoder to generate final representation . Unseen nodes can be represented based on their neighborhood structure . Current inductive approaches have no direct memory of the training nodes . We combine the idea of both transductive and inductive approaches via associating an additive global embedding bias to each node , which can be seen as a memorable global identification of each node in training sets . Our contributions can be summarized as follows : • we introduce a dual encoding framework to produce context-aware representation for nodes , and conduct experiments to demonstrate its efficiency and effectiveness ; • we apply bi-attention mechanism for graph representation dual learning , managing to learn dual representation of nodes more precisely ; • we combine the training of transductive global bias with inductive encoding process , as memory of nodes that are already used for training . 2 RELATED WORK . Following ( Cai et al . ( 2018 ) , Kinderkhedia ( 2019 ) and Goyal & Ferrara ( 2018 ) ) , there are mainly two types of approaches : 2.1 NETWORK EMBEDDING . For unsupervised embedding learning , DeepWalk ( Perozzi et al . ( 2014 ) ) and node2vec ( Grover & Leskovec ( 2016 ) ) are based on random-walks extending the Skip-Gram model ; LINE ( Tang et al . ( 2015 ) ) seeks to preserve first- and second-order proximity and trains the embedding via negative sampling ; SDNE ( Wang et al . ( 2016 ) ) jointly uses unsupervised components to preserve secondorder proximity and expolit first-order proximity in its supervised components ; TRIDNR ( Pan et al . ( 2016 ) ) , CENE ( Sun et al . ( 2016 ) ) , TADW ( Yang et al . ( 2015 ) ) , GraphSAGE ( Hamilton et al . ( 2017b ) ) utilize node attributes and potentially node labels . Convolutional neural networks are also applied to graph-structured data . For instance , GCN ( Kipf & Welling ( 2017 ) ) proposed an simplified graph convolutional network . These graph convolutional network based approaches are ( semi ) supervised . Recently , inductive graph embedding learning ( Hamilton et al . ( 2017b ) Velickovic et al . ( 2017 ) Bojchevski & Günnemann ( 2017 ) Derr et al . ( 2018 ) Gao et al . ( 2018 ) , Li et al . ( 2018 ) , Wang et al . ( 2018 ) and Ying et al . ( 2018b ) ) produce impressive performance across several large-scale benchmarks . 2.2 ATTENTION . Attention mechanism in neural processes have been extensively studied in neuroscience and computational neuroscience ( Itti et al . ( 1998 ) ; Desimone & Duncan ( 1995 ) ) and frequently applied in deep learning for speech recognition ( Chorowski et al . ( 2015 ) ) , translation ( Luong et al . ( 2015 ) ) , question answering ( Seo et al . ( 2016 ) ) and visual identification of objects ( Xu et al . ( 2015 ) ) . Inspired by ( Seo et al . ( 2016 ) and Abu-El-Haija et al . ( 2018 ) ) , we construct a bi-attention layer upon aggregators to capture useful parts of the neighborhood . 3 MODEL . Let G = { V , E , X } be an undirected graph , where a set of nodes V are connected by a set of edges E , and X ∈ R|V |×f is the attribute matrix of nodes . A global embedding bias matrix is denoted by B ∈ R|V |×d , where a row of B represents the d-dimensional global embedding bias of a node . The hierarchical layer number , the embedding output of the l-th layer and the final output embedding are denoted by L , hl and z , respectively . 3.1 CONTEXT-AWARE INDUCTIVE EMBEDDING ENCODING . The embedding generation process is described in Algorithm 1 . Assume that the dual encoder is trained and parameters are fixed . Algorithm 1 Context-Aware Dual-Encoding ( CADE ) input : the whole graph G = ( V , E ) ; the feature matrix X ; the trained DualENC output : learned embeddings z ; 1 : Run random walks on G to gain a set of positive pair P ; 2 : Zv ← ∅ , ∀v ∈ V 3 : for ( v , vp ) ∈ P do 4 : zv , zvp = DualENC ( v , vp , G , X ) ; 5 : Zv ← Zv ∪ zv 6 : Zvp ← Zvp ∪ zvp 7 : end for 8 : for v ∈ V do 9 : zv = Mean ( Zv ) ; 10 : end for After training , positive pairs are collected by random walks on the whole dataset . The features of each positive pair are passed through a dual-encoder . Embeddings of nodes are generated so that the components of a pair are related and adjacent to each other . 3.2 DUAL-ENCODER WITH MULTI-SAMPLING . In this subsection , we explain the dual encoder . Algorithm 2 DualENC input : Training graph G ( V , E ) ; node attributes X ; global embedding bias matrix B ; sampling times K ; positive node pair ( v , vp ) ; output : adjacent embeddings zv and zvp ; 1 : For node v and vp , generate K embeddings , hv , hvp , using a base encoder SAGB 2 : for i , j ∈ { 1 , ... , K } do 3 : Si , j ← α ( hvi , hvpj ) 4 : end for 5 : softmax on flattened similarity matrix S : Si , j ← e Si , j∑K , K 0,0 e Si , j 6 : calculate attention av and avp : avi ← ∑K j=1 Si , j , avpj ← ∑K i=1 Si , j 7 : zv ← ∑K t=1 avkh L vk 8 : zvp ← ∑K t=1 avpkh L vpk In the hierarchical sampling and aggregating framework ( Hamilton et al . ( 2017b ) ) , it is challenging and vital to select relevant neighbor with the layer goes deeper and deeper . For example , given word ” mouse ” and its positive node ” PC ” , it is better to sample ” keyboard ” , instead of ” cat ” , as a neighbor node . However , to sample the satisfying node according to heuristic rules layer by layer is very time consuming , and it is difficult to learn attention over neighborhood for unsupervised embedding learning . As a matter of fact , these neighbor nodes are considered to be useful because they are more welcome to be sampled as input so as to produce more relevant output of the dual-encoder . Therefore , instead of physically sampling these neighbor nodes , in Step 2 to Step 6 in Algorithm 2 , we directly apply a bi-attention layer on the two sets of embedding outputs with different sampled neighborhood feature as input , so as to locate the most relevant representation match , as a more efficient approach to exploring the most useful neighborhood . We use the hierarchical sampling and aggregating framework as a base encoder in our experiments , but it can also be designed in many other ways . Given node v , and node vp as a positive node pair , after sampling and aggregating for K times , we have K different representations hvk/hvpk corresponding to K different sampled neighborhoods , their similarity matrix can be calculated by Si , j = α ( hvi , hvpj ) , i , j = 1 , ... , K ( 1 ) where α represents an dot-product operation . our goal is to find the closest neighborhood match between v and vp within K ×K possibilities , so we apply softmax on the flattened similarity matrix , and sum up by row ( column ) . In this way , we manage to gain different attention over K neighborhoods of v ( vp ) with respect to vp ( v ) , Sij ← exp ( Sij ) ∑K , K 0,0 exp ( Sij ) ( 2 ) avi = K∑ j=1 Sij ( 3 ) avpj = K∑ i=1 Sij ( 4 ) and sum up the lth layer representations with attention as the dual-encoder outputs , zv = K∑ k=1 avkhvk ( 5 ) zvp = K∑ k=1 avpkhvpk ( 6 ) To train our encoder before using it to generate final representations of nodes , we apply a typical pair reconstruction loss function with negative sampling Hamilton et al . ( 2017b ) : JG ( zv ) = −log ( σ ( zTv zvp ) ) −Q · Evn∼Pn ( v ) log ( σ ( −z T v zvn ) ) ( 7 ) where node vp co-occurs with v on fixed-length random walk ( Perozzi et al . ( 2014 ) ) , σ is the sigmoid function , Pn is a negative sampling distribution , Q defines the number of negative samples . Note that zv and zvp are dual representation to each other while zvn represents the direct encoder output of negative sample vn .

Summary: This paper proposed a dual graph representation method to learn the representation of nodes in a graph. In particular, it learns the embedding of paired nodes simultaneously for multiple times, and use the mean values as the final representation. The experimental result demonstrates some improvement over existing methods. Overall, the idea is presented clearly and the writing is well structured. But the novelty is limited. Specifically,