Edge Partition Modulated Graph Convolutional Networks

1 INTRODUCTION . The graph data structure is widely adopted to describe interconnected objects , such as users in a social network and atoms held up by chemical bonds to form molecules . The edges in real-world graphs usually indicate strong and logically reasonable relations between nodes . Graph convolutional networks ( GCNs ) ( Defferrard et al. , 2016 ; Kipf & Welling , 2017 ; Veličković et al. , 2018 ; Klicpera et al. , 2019a ; b ) capture this property of graph data by introducing neighborhood aggregation into regular perceptron layers . While this approach typically enhances the feature dependencies among strongly-related adjacent nodes , the absence of relational inference makes the aggregation mechanism indifferent to the multiple types of latent inter-node relations , ignoring which limits the ultimate potential of the model performance . Extensive efforts have been made to seek solution for performance enhancement in dealing with graphs with heterogeneous relations ( Schlichtkrull et al. , 2018 ; Vashishth et al. , 2020 ; Nickel et al. , 2015 ) , where the common ground these methods converges to is to learn multiple graphs , each one specified to one type of relation . The separation of relations-specific graphs from one graph is easy when these relations are explicitly annotated by edge labels , but is non-trivial when the relations are non-observable . To address this issue , some GCNs introduce techniques such as multi-head ( Veličković et al. , 2018 ; Gao & Ji , 2019 ) or multi-hop ( Abu-El-Haija et al. , 2019 ) aggregation that increase the model capacity for modeling multi-relations , however , the aggregation weights in these models are either short of interpretability or too dependent on human knowledge to adapt to the underlying relations . Another line of research aims to disentangle relation-specific node representations by task-driven graph factorization ( Ma et al. , 2019 ; Yang et al. , 2020 ) , where the relational disentanglement is performed relying on label supervision , hence their performances would be negatively impacted by the scarcity of observed labels . In contrast to the aforementioned methods , we introduce edge partition modulated graph convolutional networks ( EPM-GCNs ) , a generative framework that models the generation of edges and labels from latent relations . To begin with , we represent the latent relations with co-membership of K overlapping latent communities . For instance , in a social network where people are linked by online friendship , one person may be simultaneously affiliated with multiple social groups ( e.g. , graduating from the same school A , or working for the same company B , etc. ) . Then some similaritybased relations ( e.g. , schoolmates of A , colleagues in B ) would naturally exist among people that are affiliated with the same social group . Under this modeling assumption , the strength of each relation between a pair of nodes is measured by the multiplication of the following latent quantities : ( i ) the average interaction rate among co-members in the corresponding community and ( ii ) how strong the two ending nodes are affiliated with this community . The edges are further modeled by the logical OR of binary latent edges independently generated through the Bernoulli-Poisson link function given the strengths of each relation . This part of the generative model explains the shrinkage of heterogeneous relations into binary latent edges . We next build up the generation process of labels . Since one node simultaneously engages in multiple relations , we generate its representation as a composition of relation-specific sub-representations . These sub-representations are learned by three steps : the first step partitions the edges by normalizing the strengths of K community-based relations , and defines K independent GCNs with each set of weighted edges ; the second step obtains the desirable sub-representations by running the K GCNs in parallel . The labels of nodes or graphs could be predicted with the node representations generated in this way . This generative model allows the community-based relations to be inferred from both labels and edges , which not only systematically solves the relation non-observable issue , but also incorporates edges as an alternative source of information provided for the posterior inference of latent relations . Since the observed edges dominate labels in quantity , our method would not heavily depend on label supervision , hence may suffer less from the scarcity of labels . We summarize the major contributions of EPM-GCNs as follows : • We develop a novel generative model that generates labels from relation-specific subrepresentations , which are at first learned through independent neighborhood aggregations with weighted edges that are partitioned by the strengths of latent relations ; • We integrate the label generation process with the generation of edges , which enables the observed edges to compensate for the scarcity of observed labels by providing extra information for latent communities/relations ; • The proposed EPM-GCNs are end-to-end trainable via variational inference ; • We analyze the working mechanism of EPM-GCNs and evaluate the models over various real-world network datasets . Empirical results show that the EPM-GCNs achieve state-ofthe-art performance in most of the node and graph-level classification tasks . 2 EDGE PARTITION MODULATED GCNS . In this section , we present EPM-GCNs that learn the representation of node or graph from relationspecific sub-representations . As shown in Figure 1 , the overall architecture of EPM-GCNs consists of a generative network ( parameterized with θ ) and an inference network ( parameterized with ω ) . Section 2.1 outlines the data generation process and corresponding supportive modules in the generative network , Section 2.2 elaborates the module in the inference network as well as the inferencelearning details . 2.1 GENERATIVE TASK-LEARNING WITH LATENT COMMUNITIES . For the generative supervised graph analytic tasks under discussion , the information of a graph G with N nodes could be summarized by the triplet ( X , A , y ) , where X is the design matrix whose rows represent the features of individual nodes , A is the binary adjacency matrix of G , with each “ 1 ” entry indicating the presence of an edge between the corresponding node pair , and y ∈ { yo , yu } is the set of labels , with subscripts o and u denoting the observed and unobserved parts of labels , respectively . The goal of our latent factor model is to describe the generation process of observed edges and labels . Based on that , we are enabled to predict the holdout labels from their posterior . The joint generation of edges A and labels y is modeled by stacking up the generation process of edges and the generation of labels given the edges . 2.1.1 THE GENERATION OF EDGES . We adopt the generation process developed in Zhou ( 2015 ) to model the formation of edges A : Ai , j ∼ Ber ( pij ) , pij : = 1− e− ∑K k=1 γkΦi , kΦj , k , ( 1 ) which has an alternative representation under the Bernoulli-Poisson link as Ai , j = 1Mi , j≥1 , Mi , j ∼ Poisson ( ∑K k=1 γkΦi , kΦj , k ) . In this expression , Φ is a positive node-community affiliation matrix , whose entry at the ith row and kth column could be interpreted as the strength that node i is affiliated with community k , γk ( k ∈ [ K ] ) is a positive community activation level indicator that measures the member interaction frequencies featured by community k , and γkΦi , kΦj , k quantifies the interaction rate between nodes i and j through community k. In our practice , we treat γ = [ γ1 , γ2 , · · · , γk ] ′ as a part of the trainable parameters θ . In this paper , we focus on another equivalent representation that partitions each edge into the logical disjunctions of K latent binary edges , expressed as Ai , j = ∨Kk=1Ai , j , k , Ai , j , k ∼ Ber ( pijk ) , pijk : = 1− e−γkΦi , kΦj , k ∀i , j ∈ [ N ] 1 where ∨ is the logical disjunction ( i.e. , logical OR ) operator and pijk denotes the probability that the interaction of nodes i and j in community k results in an edge . The connection between nodes i and j would establish as long as their interaction in one community forms up an edge , in other words , nodes i and j would be disconnected only if their interactions in ALL communities fail to generate edges . Link function 1 gives the conditional probability pθ ( A |Φ ) . Note that the node interaction over each community uniquely corresponds to one type of inter-node relation , the generation of A reflects the aggregation of these heterogeneous relations , suggesting that one would need to partition the edges to separate these relations . 2.1.2 THE GENERATION OF LABELS : . In consonance with the graph generative model that establishes the graph from latent communities , we develop the generation process of labels from all relation-specific representations . To achieve that , we first need to separate each relation from their totality , i.e. , factorize the edges with the edge partitioner module : Edge PARTitioner ( EPART ) . The edge partitioner takes edges A and community-affiliation score matrix Φ as inputs and returns K positive-weighted edges : { A ( 1 ) , A ( 2 ) , · · · , A ( K ) } , 1We use the [ N ] notation for the integer set { 1 , 2 , · · · , N } throughout the text . s.t . ∑K k=1 A ( k ) = A . The partition function follows the form A ( k ) i , j = e ( γkΦi , kΦj , k ) /τ∑ k′ e ( γk′Φi , k′Φj , k′ ) /τ = ( 11−pijk ) 1 τ∑ k′ ( 1 1−pijk′ ) 1 τ , k = 1 , 2 , · · · , K , ∀i , j ∈ [ N ] . ( 2 ) Here τ denotes the “ temperature ” that controls the sharpness of weights , its value is specified in Table 6 in the appendix . The edge weights produced from edge partitioner represent the proportion that each community-based relation takes in contributing to edge formation . When τ is set sufficiently small , A ( 1 ) , A ( 2 ) , · · · , A ( K ) would approximate binary factors graphs of G , we hence refer to them as “ relational graph factors ” . Replacing γkΦi , kΦj , k with Φi , kdiag ( γk ) Φ ′ j , k generalizes Equation 2 to metacommunity-based edge partition , in this expression , the total number of communities is greater than K , and Φi , k denotes the kth segment in node i ’ s community-affiliation encoding . The generalized edge partitioner enriches the choices of Φ ’ s dimensionalities , which enhances the flexibility in model implementation . Subsequent to the edge partitioner follows the relational GCN bank : Relational GCN BANK ( RGCN BANK ) . The relational GCN bank is made up of K independent GCN components , coupling with the corresponding relational graph factors . It takes X∗ : = X ‖Φ as input2 , and outputs g ( 1 ) θ ( X ∗ ) , g ( 2 ) θ ( X ∗ ) , · · · , g ( K ) θ ( X∗ ) ; g ( k ) θ ( · ) : = GCNθ ( · , A ( k ) ) , k ∈ [ K ] . The function of this design is to learn relation-wise separable node representations . We now qualitatively justify the fulfillment of this requirement . Without loss of generality , let us treat the relational graph factors as approximately binary , this means the each observed edge is annotated with the relation that makes the greatest contribution in its generation . Let us denote the relations as r1 , r2 , · · · , rk which are uniquely featured with communities c1 , c2 , · · · , cK . Assuming node u connects to its neighbors v1 and v2 by relations r1 and r2 , respectively . Without edge partition , features of v1 would be propagated to v2 in two runs of neighborhood aggregation even if they connects to u with different community identities . This issue is moderated by the relational GCN bank , as the aggregation path ( v1 , u ) would only be sent to the GCN component for relation r1 whereas the path ( u , v2 ) would only exist in the GCN component corresponds to relation r2 . As a result , in each GCN component , nodes only share features through edges explained by the same relation , we hence consider the outputs of relational GCN bank as relation-specific . Finally , we learn the representations of nodes from relation-specific sub-representations . This step involves the representation composer module : REPresentation COMPoser ( REPCOMP ) . Let H ( k ) : = g ( k ) θ ( X ∗ ) denote the node representations learned from the kth relation , k ∈ [ K ] , and function f ( · ) denote the representation composer , whose functionality is to project a composite of relation-specific node representations to one representation matrix , i.e. , HV = f ( H ( 1 ) , H ( 2 ) , · · · , H ( K ) ) : = GCNθ ( ‖Kk=1H ( k ) , A ) . We could further obtain hG , the vector representation of graph G , by sending the node level representations HV to a graph pooling layer , i.e. , hG : = GRAPHPOOL ( HV ) . Taking softmax on the feature dimension of the representation composer ’ s output gives the uncertainty distribution of labels , from which we are able to make predictions on the category of the object ( node or graph ) . Cascading the edge partitioner , the community-GCN bank with the representation composer yields the probability pθ ( y |A , Φ , X ) .

Summary: This paper investigates how an edge is formed by different latent inter-node relations and extends the community-based edge formulation mechanism to graph neural networks. A variational inference framework has been proposed to jointly learn how to partition the edges into different communities and combine relation-specific GCNs for the end classification tasks. Experiments on several real-world graph datasets demonstrate the effectiveness of the proposed method in both node-level and graph-level representation learning problems.