Robust Graph Representation Learning via Neural Sparsification

1 INTRODUCTION . Representation learning has been in the center of many machine learning tasks on graphs , such as name disambiguation in citation networks ( Zhang et al. , 2018c ) , spam detection in social networks ( Akoglu et al. , 2015 ) , recommendations in online marketing ( Ying et al. , 2018a ) , and many others ( Hamilton et al. , 2017 ; Li et al. , 2018 ) . As a class of models that can simultaneously utilize non-structural ( e.g. , node and edge features ) and structural information in graphs , Graph Neural Networks ( GNNs ) ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Li et al. , 2016 ) construct effective representations for downstream tasks by iteratively aggregating neighborhood information ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ) . Such methods have demonstrated state-of-the-art performance in classification and prediction tasks on graph data ( Veličković et al. , 2018 ; Chen et al. , 2018 ; Xu et al. , 2019 ; Veličković et al. , 2019 ) . Meanwhile , graphs from real-life applications are usually large with complex local neighborhood , where each node has rich features and dozens or even hundreds of neighbors . As shown in Figure 1 ( a ) , this subgraph from Transaction dataset ( detailed in Section 5.1 ) consists of 38 nodes ( i.e. , promising organizations and other organizations ) with average node degree 15 and node feature dimension 120 . The GNNs are expected to grasp useful patterns from neighboring nodes ; however , as representative patterns are diluted by overwhelming information in local neighborhood , graph learning algorithms could be misled by neighborhood aggregation . Such complexity in input graphs poses non-trivial overfitting risk to existing GNN based learning techniques . While it is straightforward yet expensive ( sometimes even impractical ) to address this overfitting problem by increasing the number of labeled samples , we investigate a cheaper alternative of reducing input graph complexity by graph sparsification in this work . Graph sparsification ( Liu et al. , 2018 ; Zhang & Patone , 2017 ) aims to find smaller subgraphs from input large graphs that best preserve desired properties . Existing sparsification methods could lead to suboptimal performance for downstream prediction tasks : ( 1 ) these methods are unsupervised such that the resulting sparsified graphs may not favor downstream tasks ; and ( 2 ) they only consider structural information for sparsification decision , while non-structural information in graphs , such as node/edge features , could have non-trivial impact to the quality of sparsification . Recently , there have been GNN models attempting to sample subgraphs from predefined distributions ( Leskovec & Faloutsos , 2006 ; Adhikari et al. , 2018 ; Hamilton et al. , 2017 ; Chen et al. , 2018 ) . As the predefined distributions could be irrelevant to subsequent tasks , the sparsified graphs may miss important information for downstream tasks , leading to suboptimal prediction performance . Present work . We propose Neural Sparsification ( NeuralSparse ) , a general framework that simultaneously learns graph sparsification and graph representation by feedback signals from downstream tasks . The NeuralSparse consists of two major components : sparsification network and GNN . For the sparsification network , we utilize a deep neural network to parameterize the sparsification process : how to select edges from one-hop neighborhood given a fixed budget . In the training phase , the network learns to optimize a sparsification strategy that favors downstream tasks . In the testing phase , the network sparsifies input graphs following the learned strategy , instead of sampling subgraphs from a predefined distribution . Unlike conventional sparsification techniques , our technique takes both structural and non-structural information as input and optimizes the sparsification strategy by feedback from downstream tasks , instead of using ( possibly irrelevant ) heuristics . For the GNN component , the NeuralSparse feeds the sparsified graphs to a GNN and learns a graph representation for subsequent prediction tasks . Under the framework of NeuralSparse , we are able to leverage the standard stochastic gradient descent and backpropagation techniques to simultaneously optimize graph sparsification and representation . As shown in Figure 1 ( b ) , the graph sparsified by the NeuralSparse has lower complexity with average node degree around 5 . As a result ( illustrated in Figure 1 ( c ) ) , the testing classification accuracy on the sparsified graph is improved by 15 % , compared with its counterpart in the original input graph , while conventional techniques could not offer competitive sparsification for the classification task . Experimental results on both public and private datasets show that the NeuralSparse is able to consistently provide improved performance for existing GNNs on node classification tasks , bringing up to 7 % improvement . 2 RELATED WORK . Our work is related to two lines of research : graph sparsification and graph representation learning . Graph sparsification . The goal of graph sparsification is to find small subgraphs from input large graphs that best preserve desired properties . Existing techniques are mainly unsupervised and deal with simple graphs without node/edge features for preserving predefined graph metrics ( Hübler et al. , 2008 ) , information propagation traces ( Mathioudakis et al. , 2011 ) , graph spectrum ( Calandriello et al. , 2018 ; Chakeri et al. , 2016 ; Adhikari et al. , 2018 ) , node degree distribution ( Eden et al. , 2018 ; Voudigari et al. , 2016 ) , node distance distribution ( Leskovec & Faloutsos , 2006 ) , or clustering coefficient ( Maiya & Berger-Wolf , 2010 ) . Importance based edge sampling has also been studied in a scenario where we could predefine edge importance ( Zhao , 2015 ; Chen et al. , 2018 ) . Unlike existing methods that mainly work with simple graphs without node/edge features in an unsupervised manner , our method takes node/edge features as parts of input and optimizes graph sparsification by supervision signals from errors made in downstream tasks . Graph representation learning . Graph neural networks ( GNNs ) are the most popular techniques that enable vector representation learning for large graphs with complex node/edge features . All existing GNNs share a common spirit : extracting local structural features by neighborhood aggregation . Scarselli et al . ( 2009 ) explore how to extract multi-hop features by iterative neighborhood aggregation . Inspired by the success of convolutional neural networks , multiple studies ( Defferrard et al. , 2016 ; Bruna et al. , 2014 ) investigate how to learn convolutional filters in the graph spectral domain under transductive settings ( Zhang et al. , 2018b ; Zhuang & Ma , 2018 ) . To enable inductive learning , convolutional filters in the graph domain are proposed ( Simonovsky & Komodakis , 2017 ; Niepert et al. , 2016 ; Kipf & Welling , 2017 ; Veličković et al. , 2018 ; Xu et al. , 2018 ) , and a few studies ( Hamilton et al. , 2017 ; Lee et al. , 2018 ) explore how to differentiate neighborhood filtering by sequential models . In addition , multiple recent works ( Ying et al. , 2018b ; Xu et al. , 2019 ; Abu-ElHaija et al. , 2019 ) investigate the expressive power of GNNs . Recently , ( Franceschi et al. , 2019 ) study how to sample high-quality subgraphs from a space of all possible graphs of a complete graph so that the sampled graphs enhance the prediction power in downstream learning tasks . In particular , the proposed method only focus on transductive tasks . Our work contributes from a unique angle : by reducing the noise from input graphs , our technique can further boost testing performance of existing GNNs . 3 PROPOSED METHOD : NEURALSPARSE . In this section , we introduce the core idea of our method . We start with the notations that are frequently used in this paper . We then describe the theoretical justification behind NeuralSparse and our architecture to tackle the supervised node classification problem . Notations . In this paper , we represent an input graph of n nodes as G = ( V , E , A ) : ( 1 ) V ∈ Rn×dn includes node features with dimensionality dn ; ( 2 ) E ∈ Rn×n is a binary matrix where E ( u , v ) = 1 if there is an edge between node u and node v ; ( 3 ) A ∈ Rn×n×de encodes input edge features of dimensionality de . In addition , we use Y to denote the prediction target in downstream tasks ( e.g. , Y ∈ Rn×dl if we are dealing with a node classification problem with dl classes ) . Theoretical justification . From the perspective of statistical learning , the key of a defined prediction task is to learn P ( Y | G ) , where Y is the prediction target andG is an input graph . Instead of directly working with original graphs , we would like to leverage sparsified subgraphs to mitigate overfitting risks . In other words , we are interested in the following variant , P ( Y | G ) ≈ ∑ g∈SG P ( Y | g ) P ( g | G ) , ( 1 ) where g is a sparsified subgraph , and SG is a class of sparsified subgraphs of G. In general , because of the combinatorial complexity in graphs , it is intractable to enumerate all possible g as well as estimate the exact values of P ( Y | g ) and P ( g | G ) . Therefore , we approximate the distributions by tractable functions , ∑ g∈SG P ( Y | g ) P ( g | G ) ≈ ∑ g∈SG Qθ ( Y | g ) Qφ ( g | G ) ( 2 ) where Qθ and Qφ are approximation functions for P ( Y | g ) and P ( g | G ) parameterized by θ and φ , respectively . Moreover , to make the above graph sparsification process differentiable , we employ reparameterization tricks ( Jang et al. , 2017 ) to make Qφ ( g | G ) directly generate differentiable samples , such that ∑ g∈SG Qθ ( Y | g ) Qφ ( g | G ) ∝ ∑ g′∼Qφ ( g|G ) Qθ ( Y | g′ ) ( 3 ) where g′ ∼ Qφ ( g | G ) means g′ is a random sample drawn from Qφ ( g | G ) . To this end , the key is how to find appropriate approximation functions Qφ ( g | G ) and Qθ ( Y | g ) . Architecture . In this paper , we propose Neural Sparsification ( NeuralSparse ) to implement the theoretical framework discussed in Equation 3 . As shown in Figure 2 , NeuralSparse consists of two major components : sparsification network and GNNs . • The sparsification network is a multi-layer neural network that implements Qφ ( g | G ) : Taking G as input , it generates a random sparsified subgraph of G drawn from a learned distribution . • GNNs implement Qθ ( Y | g ) that takes a sparsified subgraph as input , extracts node representations , and makes predictions for downstream tasks . Algorithm 1 Training algorithm for NeuralSparse Input : graph G = ( V , E , A ) , integer l , and training labels Y . 1 : while stop criterion is not met do 2 : Generate sparsified subgraphs { g1 , g2 , · · · , gl } by sparsification network ( Section 4 ) ; 3 : Produce prediction { Ŷ1 , Ŷ2 , · · · , Ŷl } by feeding { g1 , g2 , · · · , gl } into GNNs ; 4 : Calculate loss function J ; 5 : Update φ and θ by descending J 6 : end while As the sparsified subgraph samples are differentiable , the two components can be jointly trained using gradient descent based backpropagation techniques from a supervised loss function , as illustrated in Algorithm 1 . While the GNNs have been widely investigated in recent works ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ) , we focus on the practical implementation for sparsification network in the remaining of this paper .

Summary: The authors argue that existing GCN-based approaches may pose non-trival overfitting risk during the training phase, especially when high-dimensional features and high-degree entities are observed in the graphs. To address the issue, the authors integrate graph sparsification with conventional graph neural nets. Experimental results show the efficacy of the proposed model in a series of benchmark datasets. In general, the paper is easy-to-follow and well-organized. My main concern is there lack some insightful discussion regarding the problem motivation and the proposed algorithm. In particular,