Unifying Graph Convolutional Networks as Matrix Factorization

1 INTRODUCTION . In recent years , works on graph convolutional networks ( GCN ) ( Kipf & Welling , 2017 ) have achieved great success in many graph-based tasks , e.g. , semi-supervised node classification ( Kipf & Welling , 2017 ) , link prediction ( Zhang & Chen , 2018 ) and recommendation systems ( Ying et al. , 2018 ) . GCN defines a graph convolution operation , which generates the embedding of each node by aggregating the representations of its neighbors . Given a graph , GCN performs the graph convolution operation layer by layer to obtain the final node representations , which will be passed to neural networks to support various tasks . To perform GCN on large scale graphs in constrained memory or distributed computing environments , different sampling methods have been proposed , such as neighbor sampling ( Hamilton et al. , 2017 ) and importance sampling ( Chen et al. , 2018b ) . Instead of sampling , Cluster-GCN ( Chiang et al. , 2019 ) proposes an approach to convert computation on a huge matrix to computing on a set of small matrices . However , these methods still suffer from performance loss when conducting distributed computing . To take use of various contextual information on edges in a graph , Relational GCN ( RGCN ) ( Schlichtkrull et al. , 2018 ) extends neighbor aggregation by using edge types in link prediction . Besides the edge types , Edge-enhanced Graph Neural Networks ( EGNNs ) ( Gong & Cheng , 2019 ) takes more contextual features into consideration . However , in general , GCN still has the efficiency problem when facing complex forms of contextual information . Besides GCN , graph embedding methods ( Perozzi et al. , 2014 ; Tang et al. , 2015b ; a ; Grover & Leskovec , 2016 ) are also widely applied . In general , these methods rely on first-order and secondorder proximity to embed very large information networks into low-dimensional vector spaces . The first-order proximity in a graph is the local pairwise proximity between two vertices , and the secondorder proximity between a pair of vertices in a graph is the similarity between their neighborhood structures . As for GCN , previous work shows that the graph convolution operation is actually a special form of Laplacian smoothing ( Li et al. , 2018 ) . Thus , as the converging of the model , the smoothing process can keep the final representation of a node more and more similar to those of its neighbors . Therefore , GCN is consistent with graph embedding methods in capturing the structural information . According to previous work ( Qiu et al. , 2018 ) , graph embedding methods have been successfully unified as matrix factorization ( MF ) . Thus , we believe that there might be some connections between GCN and MF . Meanwhile , comparing with GCN , MF-based methods are extremely flexible and suitable for distributed computing ( Gemulla et al. , 2011 ; Zhuang et al. , 2013 ; Yu et al. , 2014 ) . MF-based methods are also easy and efficient to be extended to tasks with complex forms of contextual information on graph edges ( Rendle et al. , 2011 ; Rendle , 2012 ; Jamali & Lakshmanan , 2013 ; Shi et al. , 2014 ; Liu et al. , 2015 ) . Thus , if we can unify the GCN model as a special form of MF , large scale and complex real-world applications will benefit from this . In this paper , we theoretically reveal the connections between GCN and MF , and unify GCN as matrix factorization with co-training and unitization in section 2 . Here , the co-training process means co-training with the classification task of labeled nodes as in ( Weston et al. , 2012 ; Yang et al. , 2016 ) , and the unitization indicates conducting vector unitization on node representations . Then , under the guidance of our theoretical analysis , we formally propose an alternative model to GCN named Co-training and Unitized Matrix Factorization ( CUMF ) in section 3 . Extensive experiments are conducted on several real-world graphs , and show co-training and unitization are two essential components of CUMF . Under centralized computing settings , CUMF achieves similar or superior performances comparing with GCN . These observations strongly verify the correctness of our theoretical analysis . Moreover , GCN performs poor on dense graphs , while CUMF has great performances . This is may caused by the over-smoothing of graph convolution on dense graphs , while CUMF can balance the smoothing of neighbours and the classification of labeled nodes through the co-training process . Experiments under distributed computing settings are also conducted , and distributed CUMF significantly outperforms the state-of-the-art distributed GCN method , i.e. , cluster-GCN ( Chiang et al. , 2019 ) . Thus , CUMF is extremely friendly to large scale real-world graphs . Meanwhile , lots of works have been done to model contextual information in MF-based methods ( Rendle et al. , 2011 ; Rendle , 2012 ; Jamali & Lakshmanan , 2013 ; Shi et al. , 2014 ; Liu et al. , 2015 ) , which have shown great effectiveness , efficiency and flexibility . 2 GCN AS CO-TRAINING AND UNITIZED MF . In this section , we plan to theoretically unify GCN as a specific form of matrix factorization . First , we need to start from the analysis of how node representations are learned in GCN . According to the definition in previous work ( Kipf & Welling , 2017 ) , we can formulate each layer of GCN as H ( l+1 ) = σ ( ∼ D − 12 ∼ A ∼ D − 12 H ( l ) W ( l ) ) , ( 1 ) where ∼ A = A + IN is the adjacency matrix of the graph G with added self-connections , IN is the identity matrix forN nodes in graphG , ∼ D is a diagonal degree matrix with ∼ Di , i = ∑ j ∼ Ai , j , H ( l ) is the representation of each node at layer l , W ( l ) is a layer-specific trainable weight matrix , and σ ( · ) denotes an activation function ( such as ReLU ( · ) = max ( 0 , · ) ) . For a node classification task , we can obtain a classification loss lclass = CrossEntropy ( Y , softmax ( H ( −1 ) ) ) , ( 2 ) where Y is the ground truth labels for the classification task , H ( −1 ) is the representation of each node at the final layer of GCN . Via optimizing Eq . ( 2 ) , the cross-entropy error of the node classification task can be minimized , and the GCN model can be learned . As in the implementation in previous work ( Kipf & Welling , 2017 ; Veličković et al. , 2017 ; Wu et al. , 2019 ) , there is no activation function on the last layer of GCN , and the final representations can be formulated as H ( −1 ) = ∼ D − 12 ∼ A ∼ D − 12 H ( −2 ) W ( −2 ) . ( 3 ) In ( Li et al. , 2018 ) , GCN has been proven to be a special form of Laplacian smoothing . The final representations in Eq . ( 3 ) tends to converge to H ( −1 ) = ∼ D − 12 ∼ A ∼ D − 12 H ( −1 ) , ( 4 ) which is an approximate solution of the final representations in GCN . Specifically , for each node i in graph G , the approximate solution of the corresponding final representation is h ( −1 ) i = ∑ j 1√ ( di +1 ) ( dj +1 ) ∼ Ai , j h ( −1 ) j , ( 5 ) from which we have h ( −1 ) i = ∑ j 1√ ( di +1 ) ( dj +1 ) Ai , j h ( −1 ) j + 1 di +1 h ( −1 ) i . ( 6 ) Then , the representation of node i can be obtained as h ( −1 ) i = ∑ j 1 di √ di +1 dj +1 Ai , j h ( −1 ) j , ( 7 ) where di is the degree of node i . According to above analysis , to train an approximate GCN model with one loss , which simultaneously models the structure of graph convolution and the node classification task , we can minimize the following loss function l = α lclass + ( 1− α ) lstructure , ( 8 ) where α a hyper-parameter to control the balance between two losses , and the structure loss lstructure refers to lstructure = ∑ i∈I dis ( h ( −1 ) i , ∑ j 1 di √ di +1 dj +1 Ai , j h ( −1 ) j ) , ( 9 ) where I denotes the set of all the nodes in graph G , and dis ( · , · ) is a distance measurement . Here , we apply the commonly used cosine similarity and obtain lstructure = ∑ i∈I cos ( h ( −1 ) i , ∑ j 1 di √ di +1 dj +1 Ai , j h ( −1 ) j ) , ( 10 ) which is equivalent to lstructure = − ∑ i∈I h ( −1 ) i ( ∑ j 1 di √ di +1 dj +1 Ai , j h ( −1 ) j ) > ∥∥∥h ( −1 ) i ∥∥∥∥∥∥∑j 1di√ di +1dj +1 Ai , j h ( −1 ) j ∥∥∥ . ( 11 ) To verify whether cosine similarity is consistent with the convergence of GCN during the training procedure , we conduct empirical experiments and train GCN models on the Cora dataset and the Pubmed dataset . Figure 1 demonstrates the average cosine similarity between nodes in the graph and their neighbors , as well as the convergence curves estimated by accuracy during the training of GCN on the two datasets . It is obvious that , the curves on the same dataset share similar tendency . That is to say , as we train a GCN model , the cosine similarity between nodes in the graph and their neighbors is being optimized . This strongly proves that cosine similarity is consistent with the convergence of the GCN model . Then , to simplify the form of Eq . ( 11 ) , we conduct vector unitization in the learned representations H ( −1 ) , and thus each representation h ( −1 ) i has similar l2-norm . Accordingly , through unitization , Eq . ( 11 ) is equivalent to lstructure = − ∑ i∈I h ( −1 ) i ( ∑ j 1 di √ di +1 dj +1 Ai , j h ( −1 ) j ) > , ( 12 ) which leads to lstructure = − ∑ i∈I ∑ j∈Ci 1 di √ di +1 dj +1 Ai , j vi v > j , ( 13 ) where Ci denotes all the nodes that node i is connected to , and vi = h ( −1 ) i for simplicity . Moreover , for better optimization , we can incorporate negative log likelihood and minimize the following loss function equivalently to Eq . ( 13 ) lstructure = − ∑ i∈I ∑ j∈Ci 1 di √ di +1 dj +1 Ai , j log ( λ ( vi v > j ) ) , ( 14 ) where λ ( · ) = sigmoid ( · ) . So far , the structure loss lstructure is equivalent to factorizing all the positive edges in the graph G weighted by βi , j = d −1 i ( di +1 ) 1/2 ( dj +1 ) −1/2 . Usually , in graph embedding methods ( Perozzi et al. , 2014 ; Tang et al. , 2015b ; a ; Grover & Leskovec , 2016 ) , negative edges sampling is used , for better convergence . Thus , we can randomly sample negative edges for each edge in graph G. Following previous work in unifying word embedding ( Levy & Goldberg , 2014 ) and graph embedding ( Qiu et al. , 2018 ) as implicit matrix factorization , we can rewrite Eq . ( 14 ) as lstructure = − ∑ i∈I ∑ j∈Ci ( βi , j Ai , j log ( λ ( vi v > j ) ) + kEj′∼PG [ βi , j′ log ( λ ( − vi v > j′ ) ) ] ) , ( 15 ) and lstructure = − ∑ i∈I ∑ j∈Ci βi , j Ai , j log ( λ ( vi v > j ) ) − k ∑ i∈I di Ej′∼PG [ βi , j′ log ( λ ( − vi v > j′ ) ) ] , ( 16 ) where k is the number of negative samples for each edge , and PG denotes the distribution that generates negative samples in graph G. For each node j , PG ( j ) = dj / |G| , where |G| is the number of edges in graph G. Then , we can explicitly express the expectation term as Ej′∼PG [ βi , j′ log ( λ ( − vi v > j′ ) ) ] = ∑ j′∈I βi , j′ dj′ |G| log ( λ ( − vi v > j′ ) ) , ( 17 ) from which we have Ej′∼PG [ βi , j′ log ( λ ( − vi v > j′ ) ) ] = βi , j dj |G| log ( λ ( − vi v > j ) ) + ∑ j′∈I\ { j } βi , j′ dj′ |G| log ( λ ( − vi v > j′ ) ) . ( 18 ) Via combining Eq . ( 16 ) and Eq . ( 18 ) , we can obtain the local structure loss lstructure for a specific edges ( i , j ) lstructure ( i , j ) = −βi , j Ai , j log ( λ ( vi v > j ) ) − k βi , j di dj |G| log ( λ ( − vi v > j ) ) . ( 19 ) To optimize the objective , we need to calculate the partial derivative of lstructure with respect to vi v T j ∂ lstructure ∂ ( vi v > j ) = −βi , j Ai , j λ ( − vi v > j ) + k βi , j di dj|G| λ ( vi v > j ) . ( 20 ) After setting Eq . ( 20 ) to zero , we can obtain e2 vi v > j − ( |G|Ai , j k di dj − 1 ) evi v > j −|G|Ai , j k di dj = 0 , ( 21 ) which has two solutions , evi v > j = −1 and evi v > j = |G|Ai , j k di dj , ( 22 ) which leads to vi v > j = log ( |G|Ai , j k di dj ) . ( 23 ) Accordingly , the GCN model can be unified as the following matrix factorization VV > = log ( |G|D−1 AD−1 ) − log ( k ) , ( 24 ) co-training with the classification loss lclass , where node representations in V are unitized . D is a diagonal degree matrix with Di , i = di . Moreover , according to the conclusion in ( Qiu et al. , 2018 ) , the unified matrix of GCN in Eq . ( 24 ) is as the same as the unified matrix of LINE with the second order proximity ( Tang et al. , 2015b ) , except there are two representation matrices in LINE . More specifically , the matrix factorization in Eq . ( 24 ) is as the same as LINE with the first order proximity , which is implicit matrix factorization . Our theoretical analysis on GCN is summarized as : Conclusion 2.1 Given a connected graph G with adjacency matrix A , graph convolutional networks can be unified as implicit matrix factorization with ( 1 ) co-training with labeled node classification ; ( 2 ) unitization of node representations .

Summary: In this paper, the authors propose a new method for semi-supervised node classification by drawing connection between GCN and MF. The authors borrow the idea of convergence of GCN as Laplacian Smoothing. With this observation, the authors propose a joint loss with two components: classification loss and structure loss for the similarity between embedding of neighboring nodes. The authors train the parameters via optimizing the two losses alternatively. Experiments are carried out on seven networks with comparison to baselines.