VEM-GCN: Topology Optimization with Variational EM for Graph Convolutional Networks

1 INTRODUCTION . Complex graph-structured data are ubiquitous in the real world , ranging from social networks to chemical molecules . Inspired by the remarkable performance of convolutional neural networks ( CNNs ) in processing data with regular grid structures ( e.g. , images ) , a myriad of studies on GCNs have emerged to execute “ convolution ” in the graph domain ( Niepert et al. , 2016 ; Kipf & Welling , 2017 ; Gilmer et al. , 2017 ; Hamilton et al. , 2017 ; Monti et al. , 2017 ; Gao et al. , 2018 ) . Many of these approaches follow a neighborhood aggregation mechanism ( a.k.a. , message passing scheme ) that updates the representation of each node by iteratively aggregating the transformed messages sent from its neighboring nodes . Commencing with the pioneering works ( Kipf & Welling , 2017 ; Gilmer et al. , 2017 ) , numerous strategies have been developed to improve the vanilla message passing scheme such as introducing self-attention mechanism ( Veličković et al. , 2018 ; Zhang et al. , 2020 ) , incorporating local structural information ( Zhang et al. , 2020 ; Jin et al. , 2019 ; Ye et al. , 2020 ) , and leveraging the link attributes ( Gong & Cheng , 2019 ; Li et al. , 2019 ; Jiang et al. , 2019 ) . Despite significant success in many fundamental tasks of graph-based machine learning , message passing-based GCNs almost all process the observed graph structure as ground truth and might suffer from the over-smoothing problem ( Li et al. , 2018 ) , which would seriously affect the node classification performance . Given the observed noisy graph topology ( i.e. , excessive inter-class edges are linked while many intra-class edges are missing ) , when multiple message passing layers are stacked to enlarge the receptive field ( the maximum hop of neighborhoods ) , features of neighboring nodes in different classes would be dominant in message passing . Thus , node representations would be corrupted by the harmful noise and affect the discrimination of graph nodes . The over-smoothing phenomenon in GCNs has already been studied from different aspects . Li et al . ( 2018 ) first interpreted over-smoothing from the perspective of Laplacian smoothing , while Xu et al . ( 2018 ) and Klicpera et al . ( 2019a ) associated it with the limit distribution of random walk . Furthermore , Chen et al . ( 2020a ) developed quantitative metrics to measure the over-smoothness from the topological view . They argued that the key factor leading to over-smoothing is the noise passing between nodes of different categories and the classification performance of GCNs is positively correlated with the proportion of intra-class node pairs in all edges . In this paper , we propose VEM-GCN , a novel architecture to address the over-smoothing problem with topology optimization for uncertain graphs . Considering that a “ clearer ” graph with more intra-class edges and fewer inter-class edges would improve the node classification performance of GCNs ( Yang et al. , 2019 ; Chen et al. , 2020a ) , VEM-GCN approaches a latent adjacency matrix parameterized by the assortative-constrained stochastic block model ( SBM ) where nodes share the same label are linked and inter-class edges should be cut off . To jointly refine the latent graph structure and learn desirable node representations for classification , variational EM algorithm ( Neal & Hinton , 1998 ) is adopted to optimize the evidence lower bound ( ELBO ) of the likelihood function . In the inference procedure ( E-step ) , graph topology is optimized by approximating the posterior probability distribution of the latent adjacency matrix with a neural network learned from node embeddings . In the learning procedure ( M-step ) , a conventional GCN is trained to maximize the log-likelihood of the observed node labels based on the learned latent graph structure . The E-step and M-step optimize the graph topology and improve the classification of unlabeled nodes in an alternating fashion . The proposed VEM-GCN architecture is flexible and general . In the E-step , the neural network can support arbitrary desirable node embeddings generated by algorithms such as node2vec ( Grover & Leskovec , 2016 ) , struc2vec ( Ribeiro et al. , 2017 ) , and GCNs , or the raw node attributes . The GCN in the M-step can also be substituted with arbitrary graph models . Furthermore , recent strategies for relieving the over-smoothing issue , i.e. , AdaEdge ( Chen et al. , 2020a ) and DropEdge ( Rong et al. , 2020 ) , are shown to be the specific cases of VEM-GCN under certain conditions . For empirical evaluation , we conduct extensive experiments on seven benchmarks for node classification , including four citation networks , two Amazon co-purchase graphs , and one Microsoft Academic graph . Experimental results demonstrate the effectiveness of the proposed VEM-GCN architecture in optimizing graph topology and mitigating the over-smoothing problem for GCNs . 2 BACKGROUND AND RELATED WORKS . Problem Setting . This paper focuses on the task of graph-based transductive node classification . A simple attributed graph is defined as a tuple Gobs = ( V , Aobs , X ) , where V = { vi } Ni=1 is the node set , Aobs = [ aobsij ] ∈ { 0 , 1 } N×N is the observed adjacency matrix , and X ∈ RN×f represents the collection of attributes with each row corresponding to the features of an individual node . Given the labels Yl = [ yic ] ∈ { 0 , 1 } |Vl|×C for a subset of graph nodes Vl ⊂ V assigned to C classes , the task is to infer the classes Yu = [ yjc ] ∈ { 0 , 1 } |Vu|×C for the unlabeled nodes Vu = V\Vl based on Gobs . Graph Convolutional Networks ( GCNs ) . The core of most GCNs is message passing scheme , where each node updates its representation by iteratively aggregating features from its neighborhoods . Denote with W ( l ) the learnable weights in the l-th layer , N ( i ) the set of neighboring node indices for node vi , and σ ( · ) the nonlinear activation function . A basic message passing layer takes the following form : h ( l+1 ) i = σ ( ∑ j∈N ( i ) ∪ { i } α ( l ) ij W ( l ) h ( l ) j ) . ( 1 ) Here , h ( l ) j is the input features of node vj in the l-th layer , W ( l ) h ( l ) j is the corresponding transformed message , and α ( l ) ij is the aggregation weight for the message passing from node vj to node vi . Existing GCNs mainly differ in the mechanism for computing α ( l ) ij ( Kipf & Welling , 2017 ; Veličković et al. , 2018 ; Ye et al. , 2020 ; Hamilton et al. , 2017 ; Zhang et al. , 2020 ) . Stochastic Block Model ( SBM ) . SBM ( Holland et al. , 1983 ) is a generative model for producing graphs with community structures . It parameterizes the edge probability between each node pair by āij |yi , yj ∼ { Bernoulli ( p0 ) , if yi = yj Bernoulli ( p1 ) , if yi 6= yj , ( 2 ) where āij is an indicator variable for the edge linking nodes vi and vj , yi and yj denote their corresponding communities ( classes ) , p0 and p1 are termed community link strength and cross- community link probability , respectively . The case where p0 > p1 is called an assortative model , while the case p0 < p1 is called disassortative . In this paper , we leverage an assortative-constrained SBM ( Gribel et al. , 2020 ) with p0 = 1 and p1 = 0 to model the latent graph for a clear topology . Over-smoothing . Real-world graphs often possess high sparsity and are corrupted by certain noise that leads to inter-class misconnection and missing intra-class edges . Over-smoothing is mainly caused by the indistinguishable features of nodes in different classes produced by the message passing along inter-class edges . Various strategies have been developed to alleviate this problem . JK-Net ( Xu et al. , 2018 ) utilizes skip connection for adaptive feature aggregation and DNA ( Fey , 2019 ) further makes improvements based on the attention mechanism . PPNP and APPNP ( Klicpera et al. , 2019a ) modify the message passing scheme by personalized PageRank ( PPR ) to avoid reaching the limit distribution of random walk . CGNN ( Xhonneux et al. , 2020 ) addresses over-smoothing in a similar manner as PPR . Zhao & Akoglu ( 2020 ) introduced a graph layer normalization scheme termed PairNorm to maintain the total pairwise distance between nodes unchanged across layers . GCNII ( Chen et al. , 2020b ) extends GCN with Initial residual and Identity mapping . However , these methods can not fundamentally address the over-smoothing issue , as they all view the observed graph as ground truth and the features of nodes in different classes would still be over-mixed along the inter-class edges . AdaEdge ( Chen et al. , 2020a ) constantly refines the graph topology by adjusting the edges in a self-training-like fashion . However , AdaEdge only adjusts the edges linking nodes classified with high confidence , which leads to limited improvement or degradation in classification performance due to the incorrect operations for misclassified nodes . DropEdge ( Rong et al. , 2020 ) randomly removes a certain fraction of edges to reduce message passing . Despite enhanced robustness , DropEdge does not essentially optimize the graph topology . BBGDC ( Hasanzadeh et al. , 2020 ) generalizes Dropout ( Srivastava et al. , 2014 ) and DropEdge by adaptive connection sampling . Uncertain Graphs and Topology Optimization . Learning with uncertain graphs is another related research area , where the observed graph structure is supposed to be derived from noisy data rather than ground truth . Bayesian approaches are typical methods that introduce uncertainty to network analysis . Zhang et al . ( 2019 ) developed BGCN that considers the observed graph as a sample from a parametric family of random graphs and makes maximum a posteriori ( MAP ) estimate of the graph parameters . Tiao et al . ( 2019 ) also viewed graph edges as Bernoulli random variables and used variational inference to optimize the posterior distribution of the adjacency matrix by approximating the pre-defined graph priors . Some other Bayesian methods have also been developed to combine GCNs with probabilistic models ( Ng et al. , 2018 ; Ma et al. , 2019 ) . However , without explicit optimization for the graph structure , they only improve the robustness under certain conditions such as incomplete edges , active learning , and adversarial attacks . For explicit topology optimization , Franceschi et al . ( 2019 ) presented LDS to parameterize edges as independent Bernoulli random variables and learn discrete structures for GCNs by solving a bilevel programming . However , LDS requires an extra validation set for training and suffers from limited scalability . TO-GCN ( Yang et al. , 2019 ) only adds the intra-class edges derived from the labeled nodes , which causes topology imbalance between Vu and Vl . GDC ( Klicpera et al. , 2019b ) refines the adjacency matrix with graph diffusion to consider the links between high-order neighborhoods . However , the added edges might still be noisy to hamper the classification . GRCN ( Yu et al. , 2020 ) modifies the original adjacency matrix by adding a residual matrix with each element measuring the similarity between two corresponding node embeddings , and IDGL ( Chen et al. , 2020c ) iteratively learns the graph structure in a similar manner . Pro-GNN ( Jin et al. , 2020 ) introduces low rank and sparsity constraints to recover a clean graph in defending adversarial attacks . NeuralSparse ( Zheng et al. , 2020 ) uses the Gumbel Softmax trick ( Jang et al. , 2017 ) to sample k neighbors from the original neighborhoods for each node but does not consider recovering missing intra-class edges . Different from the aforementioned methods , VEM-GCN aims at relieving the over-smoothing issue . We introduce a learned latent graph based on the assortative-constrained SBM to explicitly enhance intra-class connection and suppress inter-class interaction with the variational EM algorithm .

Summary: This paper proposes a method to alleviate the over-smoothing problem of GNNs. The key idea is to generate a latent graph structure via leveraging stochastic block model to approximate the observed graph structure and label information. The learned latent graph is expected to have a clear community structure with dense intra-class edges and sparse inter-class edges, so that labels of unlabeled nodes are better predicted based on the latent structure. The whole framework is well designed as an MLE problem, with EBLO solved by an alternate EM style algorithm. Both E-step and M-step are assumed to enhance each other's performance, but this point is not clearly validated in the experiments. Also, it is good to see some discussions on the relationship between the proposed framework and dropedge and adaedge methods. Overall, the idea makes sense in terms of joint topology optimization (via SBM) and node classification. The methodology is designed well as an MLE problem. The paper writes well and the experimental results demonstrate effectiveness to some extent.