Multi-scale Attributed Node Embedding

1 INTRODUCTION . Node embedding is a fundamental technique in network analysis that serves as a precursor to numerous downstream machine learning and optimisation tasks , e.g . community detection , network visualization and link prediction ( Perozzi et al. , 2014 ; Grover & Leskovec , 2016 ; Tang et al. , 2015 ) . Several recent network embedding methods , such as Deepwalk ( Perozzi et al. , 2014 ) , Node2Vec ( Grover & Leskovec , 2016 ) and Walklets ( Perozzi et al. , 2017 ) , achieve impressive performance by learning the network structure following an approach similar to Word2Vec Skip-gram ( Mikolov et al. , 2013b ) , originally designed for word embedding . In these works , sequences of neighboring nodes are generated from random walks over a network , and representations are distilled from extracted node-node proximity statistics that capture local neighbourhood information . When the nodes of a network have attributes ( or features ) , their embeddings can be used to capture information about the attributes in their local neighbourhood . For a social network , attributes might represent a person ’ s interests , habits , history or preferences . The pattern of node attributes are often similar in a neighborhood , and conversely , nodes with similar attributes are more likely to be connected . This property is known as homophily . Attributed network embedding methods ( Yang et al. , 2015 ; Huang et al. , 2017 ; Liao et al. , 2018 ) leverage this additional information to supplement that of node neighbourhood structure , benefiting many applications , e.g . recommender systems , node classification and link prediction ( Yang et al. , 2018 ; Yang & Yang , 2018 ; Zhang et al. , 2018 ) . The neighborhood of a node can be considered at different path lengths , or scales . In a social network , near neighbors may correspond to classmates , whereas nodes separated by greater scales may be in different cities or countries . Attributes of neighbors at different scales can be considered separately ( multi-scale ) or pooled in some way ( e.g . weighted average ) . Figure 1a shows how the attribute distribution over neighbourhoods at different scales can indicate nodes with similar network roles even if they are distant in the network , or even in different networks . Methods that take attributes of nearby nodes into account generalizes those that do not , e.g . Perozzi et al . ( 2017 ) , for which feature vectors can be considered standard basis vectors . Many embedding methods correspond to matrix factorization , indeed some attributed embedding methods ( e.g . Yang et al . ( 2018 ) ) explicitly factorize a matrix of link-attribute information . Embeddings learned using Skip-gram are known to factorize a matrix of pointwise mutual information ( PMI ) of co-occurrences between each word and local context words ( Levy & Goldberg , 2014 ) . Related network embedding methods ( Perozzi et al. , 2014 ; Grover & Leskovec , 2016 ; Tang et al. , 2015 ; Qiu et al. , 2018 ) also implicitly factorize PMI matrices based on the probability of encountering each ( context ) node on a random walk from each starting node ( Qiu et al. , 2018 ) . Our key contributions are : 1. to introduce the first Skip-gram style embedding algorithms that consider attribute distributions over local neighborhoods , both pooled ( AE ) and multi-scale ( MUSAE ) , and their counterparts that attribute distinct features to each node ( AE-EGO and MUSAE-EGO ) ; 2. to theoretically prove that their embeddings approximately factorize PMI matrices based on the product of an adjacency matrix power and node-feature matrix ; 3. to show that popular network embedding methods DeepWalk ( Perozzi et al. , 2014 ) and Walklets ( Perozzi et al. , 2017 ) are special cases of our AE and MUSAE ; 4. we show empirically that AE and MUSAE embeddings enable strong performance at regression , classification , and link prediction tasks for real-world networks ( e.g . Wikipedia and Facebook ) , are computationally scalable and enable transfer learning between networks . We provide reference implementations of AE and MUSAE , together with the datasets used for evaluation at https : //github.com/iclr2020/MUSAE . 2 RELATED WORK . Efficient unsupervised learning of node embeddings for large networks has seen unprecedented development in recent years . The current paradigm focuses on learning latent space representations of nodes such that those that share neighbors ( Perozzi et al. , 2014 ; Tang et al. , 2015 ; Grover & Leskovec , 2016 ; Perozzi et al. , 2017 ) , structural roles ( Ribeiro et al. , 2017 ; Ahmed et al. , 2018 ) or attributes are located close together in the embedding space . Our work falls under the last of these categories as our goal is to learn similar latent representations for nodes with similar sets of features in their neighborhoods , both on a pooled and multi-scale basis . Neighborhood preserving node embedding procedures place nodes with common first , second and higher order neighbors within close proximity in the embedding space . Recent works in the neighborhood preserving node embedding literature were inspired by the Skip-gram model ( Mikolov et al. , 2013a ; b ) , which generates word embeddings by implicitly factorizing a shifted pointwise mutual information ( PMI ) matrix ( Levy & Goldberg , 2014 ) obtained from a text corpus . This procedure inspired DeepWalk ( Perozzi et al. , 2014 ) , a method which generates truncated random walks over a graph to obtain a “ corpus ” from which the Skip-gram model generates neighborhood preserving node embeddings . In doing so , DeepWalk implicitly factorizes a PMI matrix , which can be shown , based on the underlying first-order Markov process , to correspond to the mean of a set of normalized adjacency matrix powers up to a given order ( Qiu et al. , 2018 ) . Such pooling of matrices can be suboptimal since neighbors over increasing path lengths ( or scales ) are treated equally or according to fixed weightings ( Mikolov et al. , 2013a ; Grover & Leskovec , 2016 ) ; whereas it has been found that an optimal weighting may be task or dataset specific ( Abu-El-Haija et al. , 2018 ) . In contrast , multi-scale node embedding methods such as LINE ( Tang et al. , 2015 ) , GraRep ( Cao et al. , 2015 ) and Walklets ( Perozzi et al. , 2017 ) separately learn lower-dimensional node embedding components from each adjacency matrix power and concatenate them to form the full node representation . Such un-pooled representations , comprising distinct but less information at each scale , are found to give higher performance in a number of downstream settings , without increasing the overall number of free parameters ( Perozzi et al. , 2017 ) . Attributed node embedding procedures refine ideas from neighborhood based node embeddings to also incorporate node attributes ( equivalently , features or labels ) ( Yang et al. , 2015 ; Liao et al. , 2018 ; Huang et al. , 2017 ; Yang et al. , 2018 ; Yang & Yang , 2018 ) . Similarities between both a node ’ s neighborhood structure and features contribute to determining pairwise proximity in the node embedding space . These models follow quite different strategies to obtain such representations . The most elemental procedure , TADW ( Yang et al. , 2015 ) , decomposes a convex combination of normalized adjacency matrix powers into a matrix product that includes the feature matrix . Several other models , such as SINE ( Zhang et al. , 2018 ) and ASNE ( Liao et al. , 2018 ) , implicitly factorize a matrix formed by concatenating the feature and adjacency matrices . Other approaches such as TENE ( Yang & Yang , 2018 ) , formulate the attributed node embedding task as a joint non-negative matrix factorization problem in which node representations obtained from sub-tasks are used to regularize one another . AANE ( Huang et al. , 2017 ) uses a similar network structure based regularization approach , in which a node feature similarity matrix is decomposed using the alternating direction method of multipliers . The method most similar to our own is BANE ( Yang et al. , 2018 ) , in which the product of a normalized adjacency matrix power and a feature matrix is explicitly factorized to obtain attributed node embeddings . Many other methods exist , but do not consider the attributes of higher order neighborhoods ( Yang et al. , 2015 ; Liao et al. , 2018 ; Huang et al. , 2017 ; Zhang et al. , 2018 ; Yang & Yang , 2018 ) . The relationship between our pooled ( AE ) and multi-scale ( MUSAE ) attributed node embedding methods mirrors that between graph convolutional neural networks ( GCNNs ) and multi-scale GCNNs . Widely used graph convolutional layers , such as GCN ( Kipf & Welling , 2017 ) , GraphSage ( Hamilton et al. , 2017 ) , GAT ( Veličković et al. , 2018 ) , APPNP ( Klicpera et al. , 2019 ) , SGCONV ( Wu et al. , 2019 ) and ClusterGCN ( Chiang et al. , 2019 ) , create latent node representations that pool node attributes from arbitrary order neighborhoods , which are then inseparable and unrecoverable . In contrast , MixHop ( Abu-El-Haija et al. , 2019 ) learns latent features for each proximity . 3 ATTRIBUTED EMBEDDING MODELS . We now define algorithms to learn node embeddings using the attributes of nearby nodes , that allows both node and attribute embeddings to be learned jointly . The aim is to learn similar embeddings for nodes that occur in neighbourhoods of similar attributes ; and similar embeddings for attributes that often occur in similar neighbourhoods of nodes . Let G = ( V , L ) be an undirected graph of interest where V and L are the sets of vertices and edges ( or links ) respectively ; and let F be the set of all possible node features ( i.e . attributes ) . We define Fv ⊆ F as the subset of features belonging to each node v ∈V . An embedding of nodes is a mapping g : V→Rd that assigns a d-dimensional representation g ( v ) ( or simply gv ) to each node v and is fully described by a matrix G ∈ R|V|×d . Similarly , an embedding of the features ( to the same latent space ) is a mapping h : F → Rd with embeddings denoted h ( f ) ( or simply hf ) , and is fully described by a matrix H∈R|F|×d . 3.1 ATTRIBUTED EMBEDDING . The Attributed Embedding ( AE ) procedure is described by Algorithm 1 . We sample n nodes w1 , from which to start attributed random walks on G , with probability proportional to their degree ( Line 2 ) . From each starting node , a node sequence of length l is sampled over G ( Line 3 ) , where sampling follows a first order random walk . For a given window size t , we iterate over each of the first l−t nodes of the sequence termed source nodes wj ( Line 4 ) . For each source node , we consider the following t nodes as target nodes ( Line 5 ) . For each target node wj+r , we add the tuple ( wj , f ) to the corpus D for each target feature f ∈Fwj+r ( Lines 6 and 7 ) . We also consider features of the source node f ∈Fwj , adding each ( wj+r , f ) tuple to D ( Lines 9 and 10 ) . Running Skip-gram on D with b negative samples ( Line 15 ) generates the d-dimensional node and feature embeddings . Data : G = ( V , L ) – Graph to be embedded . { Fv } V – Set of node feature sets . n – Number of sequence samples . l – Length of sequences . t – Context size . d – Embedding dimension . b – Number of negative samples . Result : Node embedding g and feature embedding h. 1 for i in 1 : n do 2 Pick w1 ∈ V according to P ( w1 ) ∼ deg ( w1 ) /vol ( G ) . 3 ( w1 , w2 , . . . , wl ) ← Sample Nodes ( G , w1 , l ) 4 for j in 1 : l − t do 5 for r in 1 : t do 6 for f in Fwj+r do 7 Add tuple ( wj , f ) to multiset D. 8 end 9 for f in Fwj do 10 Add tuple ( wj+r , f ) to multiset D. 11 end 12 end 13 end 14 end 15 Run SGNS on D with b negative samples and d dimensions . 16 Output gv , ∀v ∈ V , and hf , ∀f ∈ F = ∪VFv . Algorithm 1 : AE sampling and training procedure Data : G = ( V , L ) – Graph to be embedded . { Fv } V – Set of node feature sets . n – Number of sequence samples . l – Length of sequences . t – Context size . d – Embedding dimension . b – Number of negative samples . Result : Node embeddings gr and feature embeddings hr for r = 1 , . . . , t. 1 for i in 1 : n do 2 Pick w1 ∈ V according to P ( w1 ) ∼ deg ( w1 ) /vol ( G ) . 3 ( w1 , w2 , . . . , wl ) ← Sample Nodes ( G , w1 , l ) 4 for j in 1 : l − t do 5 for r in 1 : t do 6 for f in Fwj+r do 7 Add the tuple ( wj , f ) to multiset D→ r . 8 end 9 for f in Fwj do 10 Add the tuple ( wj+r , f ) to multiset D← r . 11 end 12 end 13 end 14 end 15 for r in 1 : t do 16 Create Dr by unification of D→ r and D← r . 17 Run SGNS on Dr with b negative samples and dt dimensions . 18 Output grv , ∀v ∈ V , and hrf , ∀f ∈ F = ∪VFv . 19 end Algorithm 2 : MUSAE sampling and training procedure

Summary: This paper introduces Skip-gram style embedding algorithms that consider attribute distributions over local neighborhoods. Algorithm 1 and 2 shows that in fact they propagate randomly selected node features to neighbors. The reviewer doesn’t think this random-walk way for selecting node feature is appropriate.  Node features describe node content. The features of neighboring nodes may complement each other. However, there is no benefit to select random features and then propagate, given that there already many approaches smartly combining node content in neighborhood.