Edge Partition Modulated Graph Convolutional Networks

1 INTRODUCTION . The graph data structure is widely adopted to describe interconnected objects , such as users in a social network and atoms held up by chemical bonds to form molecules . The edges in real-world graphs usually indicate strong and logically reasonable relations between nodes . Graph convolutional networks ( GCNs ) ( Defferrard et al. , 2016 ; Kipf & Welling , 2017 ; Veličković et al. , 2018 ; Klicpera et al. , 2019a ; b ) capture this property of graph data by introducing neighborhood aggregation into regular perceptron layers . While this approach typically enhances the feature dependencies among strongly-related adjacent nodes , the absence of relational inference makes the aggregation mechanism indifferent to the multiple types of latent inter-node relations , ignoring which limits the ultimate potential of the model performance . Extensive efforts have been made to seek solution for performance enhancement in dealing with graphs with heterogeneous relations ( Schlichtkrull et al. , 2018 ; Vashishth et al. , 2020 ; Nickel et al. , 2015 ) , where the common ground these methods converges to is to learn multiple graphs , each one specified to one type of relation . The separation of relations-specific graphs from one graph is easy when these relations are explicitly annotated by edge labels , but is non-trivial when the relations are non-observable . To address this issue , some GCNs introduce techniques such as multi-head ( Veličković et al. , 2018 ; Gao & Ji , 2019 ) or multi-hop ( Abu-El-Haija et al. , 2019 ) aggregation that increase the model capacity for modeling multi-relations , however , the aggregation weights in these models are either short of interpretability or too dependent on human knowledge to adapt to the underlying relations . Another line of research aims to disentangle relation-specific node representations by task-driven graph factorization ( Ma et al. , 2019 ; Yang et al. , 2020 ) , where the relational disentanglement is performed relying on label supervision , hence their performances would be negatively impacted by the scarcity of observed labels . In contrast to the aforementioned methods , we introduce edge partition modulated graph convolutional networks ( EPM-GCNs ) , a generative framework that models the generation of edges and labels from latent relations . To begin with , we represent the latent relations with co-membership of K overlapping latent communities . For instance , in a social network where people are linked by online friendship , one person may be simultaneously affiliated with multiple social groups ( e.g. , graduating from the same school A , or working for the same company B , etc. ) . Then some similaritybased relations ( e.g. , schoolmates of A , colleagues in B ) would naturally exist among people that are affiliated with the same social group . Under this modeling assumption , the strength of each relation between a pair of nodes is measured by the multiplication of the following latent quantities : ( i ) the average interaction rate among co-members in the corresponding community and ( ii ) how strong the two ending nodes are affiliated with this community . The edges are further modeled by the logical OR of binary latent edges independently generated through the Bernoulli-Poisson link function given the strengths of each relation . This part of the generative model explains the shrinkage of heterogeneous relations into binary latent edges . We next build up the generation process of labels . Since one node simultaneously engages in multiple relations , we generate its representation as a composition of relation-specific sub-representations . These sub-representations are learned by three steps : the first step partitions the edges by normalizing the strengths of K community-based relations , and defines K independent GCNs with each set of weighted edges ; the second step obtains the desirable sub-representations by running the K GCNs in parallel . The labels of nodes or graphs could be predicted with the node representations generated in this way . This generative model allows the community-based relations to be inferred from both labels and edges , which not only systematically solves the relation non-observable issue , but also incorporates edges as an alternative source of information provided for the posterior inference of latent relations . Since the observed edges dominate labels in quantity , our method would not heavily depend on label supervision , hence may suffer less from the scarcity of labels . We summarize the major contributions of EPM-GCNs as follows : • We develop a novel generative model that generates labels from relation-specific subrepresentations , which are at first learned through independent neighborhood aggregations with weighted edges that are partitioned by the strengths of latent relations ; • We integrate the label generation process with the generation of edges , which enables the observed edges to compensate for the scarcity of observed labels by providing extra information for latent communities/relations ; • The proposed EPM-GCNs are end-to-end trainable via variational inference ; • We analyze the working mechanism of EPM-GCNs and evaluate the models over various real-world network datasets . Empirical results show that the EPM-GCNs achieve state-ofthe-art performance in most of the node and graph-level classification tasks . 2 EDGE PARTITION MODULATED GCNS . In this section , we present EPM-GCNs that learn the representation of node or graph from relationspecific sub-representations . As shown in Figure 1 , the overall architecture of EPM-GCNs consists of a generative network ( parameterized with θ ) and an inference network ( parameterized with ω ) . Section 2.1 outlines the data generation process and corresponding supportive modules in the generative network , Section 2.2 elaborates the module in the inference network as well as the inferencelearning details . 2.1 GENERATIVE TASK-LEARNING WITH LATENT COMMUNITIES . For the generative supervised graph analytic tasks under discussion , the information of a graph G with N nodes could be summarized by the triplet ( X , A , y ) , where X is the design matrix whose rows represent the features of individual nodes , A is the binary adjacency matrix of G , with each “ 1 ” entry indicating the presence of an edge between the corresponding node pair , and y ∈ { yo , yu } is the set of labels , with subscripts o and u denoting the observed and unobserved parts of labels , respectively . The goal of our latent factor model is to describe the generation process of observed edges and labels . Based on that , we are enabled to predict the holdout labels from their posterior . The joint generation of edges A and labels y is modeled by stacking up the generation process of edges and the generation of labels given the edges . 2.1.1 THE GENERATION OF EDGES . We adopt the generation process developed in Zhou ( 2015 ) to model the formation of edges A : Ai , j ∼ Ber ( pij ) , pij : = 1− e− ∑K k=1 γkΦi , kΦj , k , ( 1 ) which has an alternative representation under the Bernoulli-Poisson link as Ai , j = 1Mi , j≥1 , Mi , j ∼ Poisson ( ∑K k=1 γkΦi , kΦj , k ) . In this expression , Φ is a positive node-community affiliation matrix , whose entry at the ith row and kth column could be interpreted as the strength that node i is affiliated with community k , γk ( k ∈ [ K ] ) is a positive community activation level indicator that measures the member interaction frequencies featured by community k , and γkΦi , kΦj , k quantifies the interaction rate between nodes i and j through community k. In our practice , we treat γ = [ γ1 , γ2 , · · · , γk ] ′ as a part of the trainable parameters θ . In this paper , we focus on another equivalent representation that partitions each edge into the logical disjunctions of K latent binary edges , expressed as Ai , j = ∨Kk=1Ai , j , k , Ai , j , k ∼ Ber ( pijk ) , pijk : = 1− e−γkΦi , kΦj , k ∀i , j ∈ [ N ] 1 where ∨ is the logical disjunction ( i.e. , logical OR ) operator and pijk denotes the probability that the interaction of nodes i and j in community k results in an edge . The connection between nodes i and j would establish as long as their interaction in one community forms up an edge , in other words , nodes i and j would be disconnected only if their interactions in ALL communities fail to generate edges . Link function 1 gives the conditional probability pθ ( A |Φ ) . Note that the node interaction over each community uniquely corresponds to one type of inter-node relation , the generation of A reflects the aggregation of these heterogeneous relations , suggesting that one would need to partition the edges to separate these relations . 2.1.2 THE GENERATION OF LABELS : . In consonance with the graph generative model that establishes the graph from latent communities , we develop the generation process of labels from all relation-specific representations . To achieve that , we first need to separate each relation from their totality , i.e. , factorize the edges with the edge partitioner module : Edge PARTitioner ( EPART ) . The edge partitioner takes edges A and community-affiliation score matrix Φ as inputs and returns K positive-weighted edges : { A ( 1 ) , A ( 2 ) , · · · , A ( K ) } , 1We use the [ N ] notation for the integer set { 1 , 2 , · · · , N } throughout the text . s.t . ∑K k=1 A ( k ) = A . The partition function follows the form A ( k ) i , j = e ( γkΦi , kΦj , k ) /τ∑ k′ e ( γk′Φi , k′Φj , k′ ) /τ = ( 11−pijk ) 1 τ∑ k′ ( 1 1−pijk′ ) 1 τ , k = 1 , 2 , · · · , K , ∀i , j ∈ [ N ] . ( 2 ) Here τ denotes the “ temperature ” that controls the sharpness of weights , its value is specified in Table 6 in the appendix . The edge weights produced from edge partitioner represent the proportion that each community-based relation takes in contributing to edge formation . When τ is set sufficiently small , A ( 1 ) , A ( 2 ) , · · · , A ( K ) would approximate binary factors graphs of G , we hence refer to them as “ relational graph factors ” . Replacing γkΦi , kΦj , k with Φi , kdiag ( γk ) Φ ′ j , k generalizes Equation 2 to metacommunity-based edge partition , in this expression , the total number of communities is greater than K , and Φi , k denotes the kth segment in node i ’ s community-affiliation encoding . The generalized edge partitioner enriches the choices of Φ ’ s dimensionalities , which enhances the flexibility in model implementation . Subsequent to the edge partitioner follows the relational GCN bank : Relational GCN BANK ( RGCN BANK ) . The relational GCN bank is made up of K independent GCN components , coupling with the corresponding relational graph factors . It takes X∗ : = X ‖Φ as input2 , and outputs g ( 1 ) θ ( X ∗ ) , g ( 2 ) θ ( X ∗ ) , · · · , g ( K ) θ ( X∗ ) ; g ( k ) θ ( · ) : = GCNθ ( · , A ( k ) ) , k ∈ [ K ] . The function of this design is to learn relation-wise separable node representations . We now qualitatively justify the fulfillment of this requirement . Without loss of generality , let us treat the relational graph factors as approximately binary , this means the each observed edge is annotated with the relation that makes the greatest contribution in its generation . Let us denote the relations as r1 , r2 , · · · , rk which are uniquely featured with communities c1 , c2 , · · · , cK . Assuming node u connects to its neighbors v1 and v2 by relations r1 and r2 , respectively . Without edge partition , features of v1 would be propagated to v2 in two runs of neighborhood aggregation even if they connects to u with different community identities . This issue is moderated by the relational GCN bank , as the aggregation path ( v1 , u ) would only be sent to the GCN component for relation r1 whereas the path ( u , v2 ) would only exist in the GCN component corresponds to relation r2 . As a result , in each GCN component , nodes only share features through edges explained by the same relation , we hence consider the outputs of relational GCN bank as relation-specific . Finally , we learn the representations of nodes from relation-specific sub-representations . This step involves the representation composer module : REPresentation COMPoser ( REPCOMP ) . Let H ( k ) : = g ( k ) θ ( X ∗ ) denote the node representations learned from the kth relation , k ∈ [ K ] , and function f ( · ) denote the representation composer , whose functionality is to project a composite of relation-specific node representations to one representation matrix , i.e. , HV = f ( H ( 1 ) , H ( 2 ) , · · · , H ( K ) ) : = GCNθ ( ‖Kk=1H ( k ) , A ) . We could further obtain hG , the vector representation of graph G , by sending the node level representations HV to a graph pooling layer , i.e. , hG : = GRAPHPOOL ( HV ) . Taking softmax on the feature dimension of the representation composer ’ s output gives the uncertainty distribution of labels , from which we are able to make predictions on the category of the object ( node or graph ) . Cascading the edge partitioner , the community-GCN bank with the representation composer yields the probability pθ ( y |A , Φ , X ) .

Summary: This paper develops Edge-Partition Modulated Graph Convolutional Networks, a GNN architecture that explicitly models latent relations between nodes in a graph. The key idea is that nodes may be organized as overlapping "communities", where each community represents a particular type of relation. For example, in a citation network of papers, where the task is to predict the category of each paper, all papers from a research group could form one latent community. If one could infer this latent community information, then this could help predict unknown labels.  To accomplish this task, the model learns a community affiliation matrix whose entries express how strongly each node is affiliated to a particular community. Once we have this matrix, the adjacency matrix is partitioned as a sum of K adjacency matrices, where K is the number of communities. The model then runs K different GCN models using the same node features but K different adjacency matrices, and the outputs of all these GCN models are concatenated. Finally, we apply a GCN model using these composite node features and the __original__ adjacency matrix, to get a final classification for each node. These GCN models together comprise the "generative network".  The different weights are learned using an intricate training procedure. First, the model learns a good initial community affiliation matrix, by maximizing the posterior probability of the community matrix given the adjacency matrix. This stage is entirely unsupervised, because it uses only the observed edges and not the edge labels.  In the next stage, the model iteratively runs inference (where the community affiliation matrix is learned by optimizing the ELBO using label supervision) and learning (where the community matrix is held fixed but the generative GCN models are trained). Inference and learning are repeated till convergence.  Finally, a community matrix is sampled from the learned posterior and this is used along with the generative network to predict unknown labels.  The authors evaluate their model on a variety of node classification and graph classification benchmarks. They also visualize some embeddings for better interpretability, and perform some ablation studies.