BIGSAGE: unsupervised inductive representation learning of graph via bi-attended sampling and global-biased aggregating

1 INTRODUCTION . Graphs and networks , e.g. , social network analysis Hamilton et al . ( 2017a ) , molecule screening Duvenaud et al . ( 2015 ) , knowledge base reasoning Trivedi et al . ( 2017 ) , and biological proteinprotein networks ( Zitnik & Leskovec ( 2017 ) ) , emerge in many real-world applications . Learning low-dimensional vector embeddings of nodes in large graphs has been proved effective for a wide variety of prediction and graph analysis tasks ( Grover & Leskovec ( 2016 ) ; Tang et al . ( 2015 ) ) . The high-level idea of node embedding is to explore high-dimensional information about the neighborhood of a node with a dense vector embedding , which can be fed to off-the-shelf machine learning approaches to tasks such as node classification and link prediction ( Perozzi et al . ( 2014 ) ) . Whereas previous approaches ( Perozzi et al . ( 2014 ) ; Grover & Leskovec ( 2016 ) ; Tang et al . ( 2015 ) ) can transductively learn embeddings on graphs , without re-training they can not generalize to new nodes that are newly added to graphs . It is ubiquitous in real-world evolving networks , e.g. , new users joining in a social friendship circle such as facebook . To address the problem , Hamilton et al . ( 2017b ) propose an approach , namely GraphSAGE , to leverage node feature information ( e.g. , text attributes ) to efficiently generate node embeddings for previously unseen nodes . Despite the success of GraphSAGE , it randomly and uniformly samples neighbors of nodes , which suggests it is difficult to explore the most useful neighbor nodes . It could be helpful if we can take advantage of the most relevant neighbors and ignore irrelevant neighbors of the target node . Besides , GraphSAGE only focuses on training parameters of the hierarchical aggregator functions , but lose sight of preserving the memory of the training nodes , which means when training is finished , those nodes that have been trained over and over again would still be treated like unseen nodes , which causes a huge waste . To address the first issue , inspired by GAT ( Velickovic et al . ( 2017 ) ) , a supervised approach that assigns different weights to all neighbors of each node in each aggregating layer , we introduce a bi-attention architecture ( Seo et al . ( 2016 ) ) to perform selective neighbor sampling in unsupervised learning scenarios . In unsupervised representation learning , when encoding embeddings of a positive1 node pair before calculating their proximity loss ( Hamilton et al . ( 2017a ) ) , we assume that 1In random-walk based approaches , nodes that tend to cooccur in short random walks over the graph are usually referred as positive to each other . neighbor nodes positive to both of the pair should have larger chance to be selected , since they are statistically more relevant to the current positive pair than other neighbors . For example , when embedding words like ” mouse ” , in Figure 1 , it ’ s more reasonable to choose ” keyboard ” rather than ” cat ” as sampled neighbor while maximizing co-occrrence probability between ” mouse ” and ” PC ” , because ” keyboard ” also tends to co-occurr with ” PC ” , which means its imformation should be more relevant . We thus stack a bi-attention architecture ( Seo et al . ( 2016 ) ) on representations aggregated from both side in a positive node pair . In this way , we learn the most relevant representations for each positive node pair corresponding to their most relevant neighbors , and simply use a fixed-size uniform sampling which allows us to efficiently generate node embeddings in batches . To address the second issue , we combine the idea behind transductive approaches and inductive approaches , by intuitively applying an additive global embedding bias to each node ’ s aggregated embedding . The global embedding biases are trainable as well as parameters of aggregator functions and can be considered as a memorable global identification of each node in training sets . When the training is completed , we generate the embedding for each node by calculating an average of multiple embedding outputs corresponding to different sampled neighbors with respect to different positive nodes . In this way , nodes that tend to co-occur in short random-walks will have more similar embeddings based on our bi-attention mechanism . Based on the above-mentioned two techniques , we propose a novel approach , called BIGSAGE ( which stands for the BI-attention architeture , global BIas and the original framework GraphSAGE , ) to explore most relevant neighbors and preserve previously learnt knowledge of nodes by utilizing bi-attention architecture and introducing global bias , respectively . 2 RELATED WORK . 2.1 NETWORK EMBEDDING . In this paper , we focus on unsupervised and inductive node embedding learning for large and evolving network data . For unsupervised learning , many different approachs have been proposed . DeepWalk ( Perozzi et al . ( 2014 ) ) and node2vec ( Grover & Leskovec ( 2016 ) ) are two classic approaches that learn node embeddings based on random-walks using or extending the Skip-Gram model . Similarly , LINE ( Tang et al . ( 2015 ) ) seeks to preserve first- and second-order proximity and trains the embedding via negative smpling . SDNE ( Wang et al . ( 2016 ) ) jointly uses unsupervised components to preserve second-order proximity and expolit first-order proximity in its supervised components . Unlike the methods mentioned above , some approaches were proposed to takes use of not only network structure but also node attributes and potentially node labels . Such as TRIDNR ( Pan et al . ( 2016 ) ) , CENE ( Sun et al . ( 2016 ) ) , TADW ( Yang et al . ( 2015 ) ) . GraphSAGE ( Hamilton et al . ( 2017b ) ) , which this paper is motivated from , also requires rich attributes of nodes for sampling and aggregating into embeddings that preserve rich local neighborhood strutural infromation . Recently , in order to address the problem in large and dense networks of consequently encounted newly jointed nodes/edges , approaches such as Hamilton et al . ( 2017b ) Velickovic et al . ( 2017 ) Bojchevski & Günnemann ( 2017 ) were proposed as inductive ways of graph embedding learning and had produced impressive performance across several large-scale inductive benchmarks . In fact , we find that the key of inductive learning is to learn an embedding encoder that relies on only information from a single node itself and/or its local observable neighborhood , instead of the entire graph as in transductive setting . 2.2 ATTENTION . Attention mechanism in neural processes have been largely studied in Neuroscience and Computational Neuroscience ( Itti et al . ( 1998 ) ; Desimone & Duncan ( 1995 ) ) and since these few years frequently applied in Deep Learning for speech recognition ( Chorowski et al . ( 2015 ) ) , translation ( Luong et al . ( 2015 ) ) , question answering ( Seo et al . ( 2016 ) ) and visual identification of objects Xu et al . ( 2015 ) . The principle inside attention mechanism is that focusing on most pertinent parts of the input , rather than using all available information , a large part of which being irrelevant to compute the desirable output . In this paper , we are inspired by Seo et al . ( 2016 ) to construct a bi-attention layer upon aggregators in order to capture the useful part of the neighborhood . 3 MODEL . In this section , we propose a hierarchical bi-attended sampling and global-biased aggregating framework ( BIGSAGE ) . We start by presenting an overview of our framework : the training in Algorithm 1 and the embedding generation in Algorithm 2 . Followingly , section 3.2 gives the detailed implemention of our bi-attention architeture , section 3.3 demonstrates how we combine global bias within the framework . Given an undirected network G = { V , E , X } , in which a set of nodes V are connected by a set of edges E , and X ∈ R|V |×f is the attribute matrix of nodes . We denote the global embedding bias matrix as B ∈ R|V |×d , where each row of B represents the d-dimensional global embedding bias of each node . The hierarchical layer number is set as K , the embedding output of k-th layer is represented by hk , and the final output embedding z . 3.1 OVERVIEW . To learn representations in unsupervised setting , we apply the same graph-based loss function used in the origin GraphSAGE : JG ( zv ) = −log ( σ ( zTv zvp ) ) −Q · Evn∼Pn ( v ) log ( σ ( −z T v zvn ) ) where node vp co-occurs with v on fixed-length random walk ( Perozzi et al . ( 2014 ) ) , sigma is the sigmoid function , Pn is a negative sampling distribution , and Q defines the number of negative samples . Algorithm 1 shows the training of our framework . Algorithm 1 BIGSAGE : training input : Training graph Gtrain ( Vtrain , Etrain ) ; node attributes X ; global embedding bias matrix B ; sampling times T 1 : zero initialize ( B ) 2 : h0v ← xv , ∀v ∈ Vtrain 3 : run random-walk in Gtrain and do negative sampling to gain a set of triplets : { ( v , vp , vn ) } 4 : for ( v , vp , vn ) do 5 : Z1 ← ∅ , Z2 ← ∅ 6 : for t ∈ { 1 , ... , T } do 7 : Z1 ← Z1 ∪ { SAGB ( v ) } 8 : Z2 ← Z2 ∪ { SAGB ( vp ) } 9 : zv , zvp ← BIATT ( Z1 , Z2 ) 10 : zvn ← SAGB ( vn ) 11 : calculate the graph-based loss JG and update model parameters with SGD When generating embeddings with the learned parameters after optimization is done , GraphSAGE encode only one single random-partial-sampled neighborhoods for each node , likely leaving out information of those unselected part . To fully preseve the structural information around , we first rerun random walk on a full graph that inlcudes the seen and unseen nodes ; Then , through our bi-attention mechanism built upon the aggregating layers , we generate the most relevant embeddings of each node w.r.t its positive nodes ; Eventually , we take average of these generated embeddings as final embeddings of each node and use them in downstream machine learning tasks . The generation process is shown in Algorithm 2 : Algorithm 2 BIGSAGE : generation of embedding input : Testing graph Gtest ( Vtest , Etest ) ; node attributes Xtest ; learned model BIGSAGE output : Vector representations zv for all v ∈ Vtest 1 : run random-walks for each node in Gtest and gain a set of positive nodes pair : { ( v , vp ) } 2 : Zi ← ∅ , ∀i ∈ Vtest 3 : for i , j ∈ { ( v , vp ) } do 4 : use step 5-9 in Algorithm 1 to generate zi , zj 5 : Zi ← Zi ∪ { zi } 6 : Zj ← Zj ∪ { zj } return zv ← MEAN ( Zv ) , ∀v ∈ Vtest 3.2 CO-OCCURRENCE BI-ATTENDED ” SAMPLING ” . We now describe our bi-attention achitecture . Given node n , and node m as a positive node pair , after sampling T times using a uniform fixed-size sampler , we have hKnt = Agg ( h K−1 n , { hK−1j∈Nt ( n ) } ) hKmt = Agg ( h K−1 m , { hK−1j∈Nt ( m ) } ) t = 1 , ... , T With T different representations corresponding to T different sampled neighborhoods , their similarity matrix can be calculated by Si , j = α ( h K ni , h K mj ) , i , j = 1 , ... , T where α represents an dot-product operation . our goal is to find the most similar or relevant neighborhood match between n and m within T × T possibilities , so we need to apply softmax on the flattened similarity matrix , and sum up by collumn ( row ) to gain attention over T neighborhoods of n ( m ) : attn = reduce sum ( softmax ( S ) , 0 ) , attm = reduce sum ( softmax ( S ) , 1 ) , and apply attention to T Kth layer representations for the final encoded embeddings , hn = T∑ t=1 attnth K nt hm = T∑ t=1 attmth K mt The aggregation process with bi-attention architecture is illustrated by Figure 2 .

Summary: This paper studied learning unsupervised inductive node embeddings with an attention mechanism. For each positive edge, multiple different sets of neighborhoods are sampled for both the source and target nodes, and the similarity between the neighborhood are used as the attention functions. Experimental results prove the effectiveness of the proposed approach over GraphSAGE on a few networks.