GRAPH NEIGHBORHOOD ATTENTIVE POOLING

1 INTRODUCTION . NRL is a powerful technique to learn representation of a graph . Such a representation gracefully lends itself to a wide variety of network analysis tasks , such as link prediction , node clustering , node classification , recommendation , and so forth . In most studies , the learning is done in a context-free fashion . That is , the representation of a node characterizes just a single aspect of the node , for instance , the local or global neighborhood of a node . Recently , a complementary line of research has questioned the sufficiency of single representations and considered a context-sensitive approach . Given a node , this approach projects it to different points in a space depending on other contexts it is coupled with . A context node can be sampled from a neighborhood ( Tu et al. , 2017 ; Zhang et al. , 2018 ) , random walk ( Ying et al. , 2018 ) , and so on . In this study we sample from a node neighborhood ( nodes connected by an edge ) . Thus , in the learning process of our approach a source node ’ s representation changes depending on the target ( context ) node it is accompanied by . Studies have shown that context-sensitive approaches significantly outperform previous context-free SOTA methods in link-prediction task . A related notion ( Peters et al. , 2018 ; Devlin et al. , 2018 ) in NLP has significantly improved SOTA across several NLP tasks . In this paper we propose GAP ( Graph neighborhood attentive pooling ) , which is inspired by attentive pooling networks ( APN ) ( dos Santos et al. , 2016 ) , originally proposed for solving the problem of pair ranking in NLP . For instance , given a question q , and a set of answers A = { a1 , . . . , az } , an APN can be trained to rank the answers in A with respect to q by using a two-way attention mechanism . APN is based on the prevalent deep learning formula for SOTA NLP , that is , embed , encode , attend , predict ( Honnibal , 2018 ) . Given a question-answer pair ( q , ai ) , the APN model first projects the embedding of the pairs using two separate encoders , and the encoder can be a CNN or LSTM . The projection helps to capture n-gram context information and/or long-term dependencies in the input sequence . Next , a soft-alignment matrix is obtained by a mutual-attention mechanism that transforms these projections using a parameter matrix . Attention vectors are then computed through a column-wise and row-wise pooling operations on the alignment matrix . Finally , the weighted sum of each of the above projections by its respective attention vector is computed to obtain the representations of the question and answer . Each candidate answer ai is then ranked according to its similarity with the question q computed using the representations of q and ai . Recently , APN have been applied to context-sensitive NRL by Tu et al . ( 2017 ) , and the inputs are textual information attached with a pair of incident nodes of edges in a graph . Such information , however , has the added overhead of encoding textual information . Though we adopt APN in GAP , we capitalize on the graph neighborhood of nodes to avoid the need for textual documents without compromising the quality of the learned representations . Our hypothesis is that one can learn high-quality context-sensitive node representations just by mutually attending to the graph neighborhood of a node and its context node . To achieve this , we naturally assume that the order of nodes in the graph neighborhood of a node is arbitrary . Moreover , we exploit this assumption to simplify the APN model by removing the expensive encode phase . Akin to textual features in APN , GAP simply uses graph neighborhood of nodes . That is , for every node in the graph we define a graph neighborhood function to build a fixed size neighborhood sequence , which specifies the input of GAP . In the APN model , the encoder phase is usually required to capture high-level features such as n-grams , and long term and order dependencies in textual inputs . As we have no textual features and due to our assumption that there is no ordering in the graph neighborhood of nodes , we can effectively strip off the encoder . The encoder is the expensive part of APN as it involves a RNN or CNN , and hence GAP can be trained faster than APN . This simple yet empirically fruitful modification of the APN model enables GAP to achieve SOTA performance on link prediction and node clustering tasks using three real world datasets . Furthermore , we have empirically shown that GAP is more than 2 times faster than an APN like NRL algorithm based on text input . In addition , the simplification in GAP does not introduce new hyperparameters other than the usual ones , such as the learning rate and sequence length in APN . 2 APN ARCHITECTURE . For the sake of being self-contained , here we briefly describe the original APN architecture . We are given a pair of natural language texts ( x , y ) as input , where x = ( w1 , . . . wX ) and y = ( w1 , . . . , wY ) are a sequence of words of variable lengths , and each word wi is a sample from a vocabulary V , X = |x| , Y = |y| , and X and Y could be different . The APN ’ s forward execution is shown in Fig . 1 ( A ) and in the following we describe each component . Embed : First embedding matrices of x and y are constructed through a lookup operation on an embedding matrix E ∈ Rd×|V| of words , where d is the embedding dimension . That is , for both x and y , respectively , embedding matrices X = [ w1 , . . . , wX ] and Y = [ w1 , . . . , wY ] are constructed by concatenating embeddings wi of each word wi in x and y , the Embed box in Fig . 1 ( A ) . Encode : Each embedding matrix is then projected using a CNN or BI-LSTM encoder to capture inherent high-level features , the Encode box in Fig . 1 ( A ) . More formally , the embedded texts X ∈ Rd×X , Y ∈ Rd×Y are projected as Xenc = f ( X , Θ ) and Yenc = f ( Y , Θ ) where f is the encoder , CNN or BI-LSTM , Θ is the set of parameters of the encoder , and Xenc ∈ Rc×X and Yenc ∈ Rc×Y , where c is the number of filters or hidden features of the CNN and BI-LSTM , respectively . Attend : In the third step , a parameter matrix P ∈ Rc×c is introduced so as to learn a similarity or soft-alignment matrix A ∈ RX×Y between the sequence projections Xenc and Yenc as : A = tanh ( XTencPYenc ) Then unnormalized attention weight vectors x′ ∈ RX and y′ ∈ RY are obtained through a column-wise and row-wise max-pooling operations on A , respectively as x′i = max ( Ai , : ) , y ′ j = max ( A : ,j ) , where 0 ≤ i < X , 0 ≤ j < Y and Ai , : and A : ,j are the i-th and j-th row and column of A , respectively . Next , the attention vectors are normalized using softmax , x = softmax ( x′ ) and y = softmax ( y′ ) . Finally , the normalized attention vectors are used to compute the final representations as rx = Xenc · xT and ry = Yenc · yT . Predict : In the last step , the representations rx and ry will be used for ranking depending on the task on hand . For instance , in a question and answer setting , each candidate answer ’ s representation ry will be ranked based on its similarity score with the question ’ s representation rx . 3 GAP . GAP adopts the APN model for learning the representations of node pairs in a graph G = ( V , E ) with a set of n nodes V and m edges E. G can be a directed or undirected and weighted or unweighted graph . Without loss of generality we assume that G is an unweighted directed graph . We define a neighborhood function N : V → 2V , which maps each node u ∈ V to a set of nodes Nu ⊆ V . A simple way of materializing Nu is to consider the first-order neighbors of u , that is , Nu = [ v : ( u , v ) ∈ E ∨ ( v , u ) ∈ E ] . An important assumption that we have on Nu is that the ordering of the nodes in Nu is not important . GAP capitalizes on this assumption to simplify the APN model and achieve SOTA performance . Even though one can explore more sophisticated neighborhood functions , in this study we simply consider the first order neighborhood . Our goal is to learn node representations using the simplified APN based on the node neighborhood function N . Hence , akin to the input text pairs in APN , we consider a pair of neighborhood sequences Ns = ( u1 , . . . , uS ) and Nt = ( v1 , . . . , vT ) associated with a pair ( s , t ) ∈ E of nodes s and t , and S = |Ns| and T = |Nt| . Without loss of generality we consider S = T . Recall that we assume the order of nodes in Ns and Nt is arbitrary . Given a source node s , we seek to learn multiple context-sensitive embeddings of s with respect to a target node t it is paired with . In principle one can learn using all pairs of nodes , however that is not scalable , and hence we restrict learning between pairs in E. GAP ’ s forward execution model is shown in Fig 1 ( B ) , and learning starts by embedding Ns and Nt , respectively , as S = ( u1 , . . . , uS ) and T = ( v1 , . . . , vT ) . Since there is no order dependency between the nodes in Ns or Nt , besides being a neighbor of the respective node , we leave out the CNN or BI-LSTM based projections of S and T that could capture the dependencies . No encoder ! Thus , the next step of GAP is mutually attending on the embeddings , S and T , of the graph neighborhood of the node pairs ; the Attend box of 1 ( B ) . That is , we employ the trainable parameter matrix P ∈ Rd×d and compute the soft-alignment matrix , A , between the neighbors of s and t. A = tanh ( STPT ) ( 1 ) Here A is a soft-alignment matrix between every pair of nodes , ( u , v ) ∈ Ns × Nt . Therefore , for each axis of A , we proceed by pooling the maximum alignment score for each node to obtain the unnormalized attention vectors s′i = max ( Ai , : ) and t ′ j = max ( A : ,j ) . As a result of the pooling operations , each neighbor u ∈ Ns of the source node , s , selects a neighbor v ∈ Nt of the target node , t , with the maximum alignment or similarity score . A similar selection is done for v ∈ Nt . This enables the source and target neighborhood sequences of the pair to influence each other in order to learn a context-sensitive representation of s and t. The normalized attention vectors are then obtained by applying softmax as s = softmax ( s′ ) and t = softmax ( t′ ) . Ultimately , we compute the context-sensitive representations rs and rt of the source and target nodes s and t , respectively as rs = S · sT and rt = T · tT . Optimization : The objective of GAP is to maximize the likelihood of the graph ( edges ) by predicting edge similarities using the dot product of the source and target representations as rs · rt ; the Predict box of Fig 1 ( B ) . Hence , we employ a hard-margin loss given in Eq . 2 . L ( s , t ) = max ( 0 , 1− rs · rt + rs · r−t ) ( 2 ) where r−t is the representation of a negative target node t − , that is ( s , t− ) /∈ E. The goal is to learn , in an unsupervised fashion , a context-sensitive embedding of nodes that enable us to rank the positive edges ( s , t ) ∈ E higher than the negative pairs ( s , t− ) . Finally a word on the computational complexity of GAP that is proportional to the the number of edges , as we are considering each edge as an input pair .

Summary: This paper proposed a GAP (graph neighborhood attentive pooling) method to solve problems of node clustering and link prediction. The proposed idea is mostly inspired by the attentive pooling network approach (APN) that has been widely used in the NLP domains; indeed the detailed steps of the proposed method is almost a ``mirror” application of APN from question-answer pair ranking in NLP to the graph domains, and the only difference is that rather than using an RNN or LSTM to deal with the temporal orders in sentences (as in APN), the proposed method simply collect the first-order neighbors of the source node and target node and concatenate them without considering their orders. For the rest part, the authors used a hard margin loss function which is also the same as in APN.