Attributed Graph Learning with 2-D Graph Convolution

1 INTRODUCTION . In an attributed graph , each node is associated with a feature vector , and nodes are connected by edges encoding their relations . Commonly seen attributed graphs include citation networks where each node is a document represented by a bag-of-words feature vector and edges are citation links , web graphs where each webpage is also represented as a vector of words and edges are hyperlinks , social networks where each user is represented by a user profile vector and edges indicate user friendship , and protein-protein interaction networks where each protein is represented by a list of protein signatures and edges encode interactions between proteins . Learning on attributed graphs including node classification and clustering finds many important applications in real-world networks . Since the connectivity patterns and node contents of an attributed graph usually contain different information , it often requires joint modelling both aspects of information to achieve good learning performance . A major class of methods ( Yang et al. , 2015 ; Pan et al. , 2016 ; Huang et al. , 2017 ) is devoted to learning efficient node representations of an attributed graph via nonnegative matrix factorization or random walk statistics and then perform downstream learning tasks with the learned representations . A number of semi-supervised classification methods ( Belkin et al. , 2006 ; Weston et al. , 2008 ; Yang et al. , 2016 ) classify nodes in an attributed graph by training a supervised classifier on node features with some kind of graph regularizer . Recently , a series of works based on graph convolutional neural networks including ChebyNet ( Defferrard et al. , 2016 ) , graph convolutional networks ( GCN ) ( Kipf & Welling , 2017 ) , GraphSAGE ( Hamilton et al. , 2017 ) and graph attention networks ( GAT ) ( Velickovic et al. , 2018 ) have been shown to achieve state-of-the-art performance in node classification and clustering ( Kipf & Welling , 2016 ; Wang et al. , 2017 ) on attributed graphs . The key component of these models is one-dimensional ( 1-D ) graph convolution , a function that naturally combines graph structures and node contents by aggregating a node ’ s features with its neighbours ’ . As shown in ( Li et al. , 2019 ) , the graph convolutional operator used in GCN and many follow-up works acts as a low-pass graph filter that smooths a node ’ s features with its neighbours ’ . Under the assumption that nearby nodes tend to be in the same class , it can produce similar feature representations for nodes in the same class , thereby making them easier to be classified or clustered . While this works well for attributed graphs with clear cluster structures , real-world networks could be highly noisy and sparse . For example , in a web graph such as Wikipedia , a hyperlink between two webpages does not necessarily indicate that they belong to the same category , so mixing their features could be harmful for learning . Furthermore , it has been shown that many real-world networks are scale-free ( Albert & Barabási , 2002 ) , which means there exist many low-degree nodes . Since these nodes may have very few or even no links to other nodes , it would be difficult and even impossible for them to learn similar feature representations as other same-class nodes , which is an intrinsic limitation of the 1-D graph convolution that is commonly adopted in existing models . To address these limitations , we propose to explore relational information on a different dimension – the relations between feature attributes , in addition to node relations . Likewise , a relation between two feature attributes should reflect some kind of similarity between them . The assumption is that attributes that tend to indicate same classes should have strong relations . For example , in a citation network , node attributes are words , and documents of AI category usually contain words such as “ learning ” , “ robotics ” , “ machine ” , “ neural ” , etc . These indicative words for AI category should have much stronger relations among themselves than with other non-indicative words . These informative relations can be used to construct an attribute affinity graph to smooth node features in a similar way as the node relations do , only in a different dimension . Importantly , attribute relations can complement node relations in node representation learning . For instance , consider a document that has no links to others and hence it is impossible to do feature smoothing with node relations . But with attribute relations , it can still learn similar feature representations as other same-class nodes . In this paper , we make the following contributions . • Methodology : We propose to use 2-D graph convolution to jointly model node relations and attribute relations for learning node representations of attributed graphs . Further , we develop a computationally efficient dimensionwise separable 2-D graph convolution ( DSGC ) , which is equivalent to performing 1-D graph convolution alternately on the node dimension and the attribute dimension respectively . • Theoretical Insight : We show that regular 1-D graph convolution on the node dimension can reduce intra-class variance of node features , which explains the success of many existing methods . Further , we show that with a properly constructed attribute affinity graph , graph convolution on the attribute dimension can also reduce intra-class variance of node features . Jointly , our analysis provides a theoretical justification of DSGC . • Empirical Study : We implement DSGC for node classification and clustering on attributed graphs , and compare it with state-of-the-art methods on a citation network , a web graph , and an email network . The results demonstrate the superiority of DSGC over regular 1-D graph convolution on spare and noisy real-world attributed networks . We also show that DSGC can be plugged into existing models to substantially improve their performance . 2 RELATED WORKS . Structural Graph Learning . Methods for structural graph learning only utilize graph structures ( node relations ) . A common approach is to learn smooth low-dimensional embeddings of nodes using Markov random walks ( Szummer & Jaakkola , 2002 ; Perozzi et al. , 2014 ; Grover & Leskovec , 2016 ) , Laplacian eigenmaps ( Belkin & Niyogi , 2004 ) , spectral kernels ( Chapelle et al. , 2003 ; Zhang & Ando , 2006 ) , autoencoders ( Wang et al. , 2016 ; Cao et al. , 2016 ; Ye et al. , 2018 ) and generative adversarial nets ( Dai et al. , 2018 ; Wang et al. , 2018 ) . Another direction is graph-based semi-supervised classification , which includes methods based on low-density graph partition ( Blum & Chawla , 2001 ; Blum et al. , 2004 ; Joachims , 2003 ) and the popular label propagation method and its variants ( Zhu et al. , 2003 ; Zhou et al. , 2004 ; Bengio et al. , 2006 ; Hein & Maier , 2007 ; Wu et al. , 2012 ) . Attributed Graph Learning . Methods for attributed graph learning take into account both graph structures and node features . A major class of methods learns node representations ( Yang et al. , 2015 ; Pan et al. , 2016 ; Huang et al. , 2017 ) or clustering nodes ( Xia et al. , 2014 ; Zhou et al. , 2010 ; Li et al. , 2018 ) by applying nonnegative matrix factorization , random walk statistics , or Laplacian eigenmaps on both graph structures and node features . Some node clustering methods use a Bayesian model ( Xu et al. , 2012 ; Bojchevski & Günnemann , 2018 ) or design a distance measure that trades off structural and feature information ( Zhou et al. , 2009 ; Cheng et al. , 2011 ) . Statistical relational learning methods model node relations and features with probabilistic graphical models , e.g. , rela- tional Markov networks ( Taskar et al. , 2002 ) . Another category of related work is the graph-based semi-supervised node classification methods that exploit both graph structures and node features . For example , iterative classification algorithm ( Sen et al. , 2008 ) iteratively classifies an unlabeled node using its neighbours ’ labels and features . Manifold regularization ( Belkin et al. , 2006 ) , deep semi-supervised embedding ( Weston et al. , 2008 ) , and Planetoid ( Yang et al. , 2016 ) classify node features by training a supervised classifier with a Laplacian or embedding-based regularizer . Graph Neural Networks . Another line of research on attributed graph learning is based on graph neural networks ( Scarselli et al. , 2009 ; Li et al. , 2016 ) . Inspired by the success of convolutional neural networks ( CNN ) on Euclidean data , recent works ( Bruna et al. , 2014 ; Henaff et al. , 2015 ; Duvenaud et al. , 2015 ; Atwood & Towsley , 2016 ) proposed to use 1-D graph convolution for attributed graph learning . To avoid the expensive eigen-decomposition , ChebyNet ( Defferrard et al. , 2016 ) uses a polynomial filter represented by k-th order polynomials of graph Laplacian via Chebyshev expansion . Graph convolutional networks ( GCN ) ( Kipf & Welling , 2017 ) further simplifies ChebyNet by designing an efficient layer-wise propagation rule via a first-order approximation of spectral graph convolution . GCN achieved outstanding results in semi-supervised classification and inspired many follow-up works including MoNet ( Monti et al. , 2017 ) , GraphSAGE ( Hamilton et al. , 2017 ) , graph attention networks ( Velickovic et al. , 2018 ) , gated attention networks ( Zhang et al. , 2018a ) , FastGCN ( Chen et al. , 2018b ) , dual graph convolutional neural network ( Zhuang & Ma , 2018 ) , stochastic GCN ( Chen et al. , 2018a ) , attributed network representation learning ( Zhang et al. , 2018c ) , LanczosNet ( Liao et al. , 2019 ) , deep graph infomax ( Velickovic et al. , 2019 ) , graph Markov neural networks ( Qu et al. , 2019 ) , DisenGCN ( Ma et al. , 2019a ) , MixHop ( Abu-El-Haija et al. , 2019 ) , etc . A recent interesting research direction is on learning graph structures for graph neural networks , e.g. , Bayesian GCN ( Zhang et al. , 2018b ) and LDS ( Franceschi et al. , 2019 ) . Remarkably , LDS can automatically learn a graph for data samples even when it is not available . Some attributed graph clustering methods based on 1-D graph convolution also showed promising performance , including graph autoencoder and graph variational autoencoder ( Kipf & Welling , 2016 ) , marginalized graph autoencoder ( Wang et al. , 2017 ) , adversarially regularized graph autoencoder and adversarially regularized variational graph autoencoder ( Pan et al. , 2018 ) , and adaptive graph convolution ( Zhang et al. , 2019 ) . More comprehensive reviews of graph neural networks can be found in ( Cai et al. , 2018 ; Zhang et al. , 2018d ; Zhou et al. , 2018 ) . 3 2-D GRAPH CONVOLUTION . In this section , we present 2-D graph convolution for attributed graph learning . A comprehensive introduction of multi-dimensional graph Fourier transform can be found in ( Kurokawa et al. , 2017 ) . Different from Kurokawa et al . ( 2017 ) , here we propose a localized 2-D graph convolution to circumvent the computationally intensive graph Fourier transform . Furthermore , we propose an even simpler dimensionwise separable 2-D graph convolution to efficiently model both node relations and attribute relations along the two dimensions of the feature matrix of an attributed graph .

Summary: The paper proposes a new 2-D graph convolution method to aggregate information using both the node relation graph and the attribute graph generated using, e.g., PMI and KNN. The motivation does make sense that using 1-D convolution along the node dimension might not enough for learning representation for those low-degree nodes. Then, the attribute relation might be used to further smooth the node representation. The assumption could be that documents in a class are likely to consist of similar (related) words. To achieve this, the information aggregation along the node and the attribute dimension is implemented via a product of three matrices, the node graph convolutional filter computed from node affinity matrix, the attribute graph convolutional filter computed the attribute affinity matrix given by PMI or KNN, and the node-attribute matrix, which can be one main contribution.