Boost then Convolve: Gradient Boosting Meets Graph Neural Networks

1 INTRODUCTION . Graph neural networks ( GNNs ) have shown great success in learning on graph-structured data with various applications in molecular design ( Stokes et al. , 2020 ) , computer vision ( Casas et al. , 2019 ) , combinatorial optimization ( Mazyavkina et al. , 2020 ) , and recommender systems ( Sun et al. , 2020 ) . The main driving force for progress is the existence of canonical GNN architecture that efficiently encodes the original input data into expressive representations , thereby achieving high-quality results on new datasets and tasks . Recent research has mostly focused on GNNs with sparse data representing either homogeneous node embeddings ( e.g. , one-hot encoded graph statistics ) or bag-of-words representations . Yet tabular data with detailed information and rich semantics among nodes in the graph are more natural for many situations and abundant in real-world AI ( Xiao et al. , 2019 ) . For example , in a social network , each person has socio-demographic characteristics ( e.g. , age , gender , date of graduation ) , which largely vary in data type , scale , and missing values . GNNs for graphs with tabular data remain unexplored , with gradient boosted decision trees ( GBDTs ) largely dominating in applications with such heterogeneous data ( Bentéjac et al. , 2020 ) . GBDTs are so successful for tabular data because they possess certain properties : ( i ) they efficiently learn decision space with hyperplane-like boundaries that are common in tabular data ; ( ii ) they are well-suited for working with variables of high cardinality , features with missing values , and of different scale ; ( iii ) they provide qualitative interpretation for decision trees ( e.g. , by computing decrease in node impurity for every feature ) or for ensembles via post-hoc analysis stage ( Kaur et al. , 2020 ) ; ( iv ) in practical applications , they mostly converge faster even for large amounts of data . In contrast , a crucial feature of GNNs is that they take into account both the neighborhood information of the nodes and the node features to make a prediction , unlike GBDTs that require additional preprocessing analysis to provide the algorithm with graph summary ( e.g. , through unsupervised graph embeddings ( Hu et al. , 2020a ) ) . Moreover , it has been shown theoretically that message-passing GNNs can compute any function on its graph input that is computable by a Turing machine , i.e. , GNN is known to be the only learning architecture that possesses universality properties on graphs ( approximation ( Keriven & Peyré , 2019 ; Maron et al. , 2019 ) and computability ( Loukas , 2020 ) ) . Furthermore , gradient-based learning of neural networks can have numerous advantages over the treebased approach : ( i ) relational inductive bias imposed in GNNs alleviates the need to manually engineer features that capture the topology of the network ( Battaglia et al. , 2018 ) ; ( ii ) the end-to-end nature of training neural networks allows multi-stage ( Fey et al. , 2019 ) or multi-component ( Wang et al. , 2020 ) integration of GNNs in application-dependent solutions ; ( iii ) pretraining representations with graph networks enriches transfer learning for many valuable tasks such as unsupervised domain adaptation ( Wu et al. , 2020 ) , self-supervised learning ( Hu et al. , 2020b ) , and active learning regimes ( Satorras & Estrach , 2018 ) . Undoubtedly , there are major benefits in both GBDT and GNN methods . Is it possible to get advantages of both worlds ? All previous approaches ( Arik & Pfister , 2020 ; Popov et al. , 2019 ; Badirli et al. , 2020 ) that attempt to combine gradient boosting and neural networks are computationally heavy , do not consider graph-structured data , and suffer from the lack of relational bias imposed in GNN architectures , see Appendix A for a more detailed comparison with related literature . To the best of our knowledge , the current work is the first to explore using GBDT models for graph-structured data . In this paper , we propose a novel learning architecture for graphs with tabular data , BGNN , that combines GBDT ’ s learning on tabular node features with GNN that refines the predictions utilizing the graph ’ s topology . This allows BGNN to inherit the advantages of gradient boosting methods ( heterogeneous learning and interpretability ) and graph networks ( representation learning and end-toend training ) . Overall , our contributions are the following : ( 1 ) We design a novel generic architecture that combines GBDT and GNN into a unique pipeline . To the best of our knowledge , this is the first work that systematically studies the application of GBDT to graph-structured data . ( 2 ) We overcome the challenge of end-to-end training of GBDT by iteratively adding new trees that fit the gradient updates of GNN . This allows us to backpropagate the error signal from the topology of the network to GBDT . ( 3 ) We perform an extensive evaluation of our approach against strong baselines in node prediction tasks . Our results consistently demonstrate significant performance improvements on heterogeneous node regression and node classification tasks over a variety of real-world graphs with tabular data . ( 4 ) We show that our approach is also more efficient than the state-of-the-art GNN models due to much faster loss convergence during training . Furthermore , learned representations exhibit discernible structure in the latent space , which further demonstrates the expressivity of our approach . 2 BACKGROUND . Let G = ( V , E ) be a graph with nodes having features and target labels . In node prediction tasks ( classification or regression ) , some target labels are known , and the goal is to predict the remaining ones . Throughout the text , by lowercase variables xv ( v ∈ V ) or x we denote features of individual nodes , and X represents the matrix of all features for v ∈ V . Individual target labels are denoted by yv , while Y is the vector of known labels . Graph Neural Networks ( GNNs ) use both the network ’ s connectivity and the node features to learn latent representations for all nodes v ∈ V . Many popular GNNs use a neighborhood aggregation approach , also called the message-passing mechanism , where the representation of a node v is updated by applying a non-linear aggregation function of v ’ s neighbors representation ( Fey & Lenssen , 2019 ) . Formally , GNN is a differentiable , permutation-invariant function gθ : ( G , X ) 7→ Ŷ , where Ŷ is the vector of predicted labels . Similar to traditional neural networks , GNNs are composed of multiple layers , each representing a non-linear message-passing function : xtv = COMBINE t ( xt−1v , AGGREGATE t ( { ( xt−1w , x t−1 v ) : ( w , v ) ∈ E } ) ) , ( 1 ) where xtv is the representation of node v at layer t , and COMBINE t and AGGREGATEt are ( parametric ) functions that aggregate representations from the local neighborhood of a node . Then , the GNN mapping gθ includes multiple layers of aggregation ( 1 ) . Parameters of GNN model θ are optimized with gradient descent by minimizing an empirical loss function LGNN ( Y , gθ ( G , X ) ) . Gradient Boosted Decision Trees ( GBDT ) is a well-known and widely used algorithm that is defined on non-graph tabular data ( Friedman , 2001 ) and is particularly successful for tasks containing heterogeneous features and noisy data . The core idea of gradient boosting is to construct a strong model by iteratively adding weak ones ( usually decision trees ) . Formally , at each iteration t of the gradient boosting algorithm , the model f ( x ) is updated in an additive manner : f t ( x ) = f t−1 ( x ) + ht ( x ) , ( 2 ) where f t−1 is a model constructed at the previous iteration , ht is a weak learner that is chosen from some family of functions H , and is a learning rate . The weak learner ht ∈ H is chosen to approximate the negative gradient of a loss function L w.r.t . the current model ’ s predictions : ht = argmin h∈H ∑ i ( −∂L ( f t−1 ( xi ) , yi ) ∂f t−1 ( xi ) − h ( xi ) ) 2 . ( 3 ) The gradient w.r.t . the current predictions indicates how one should change these predictions to improve the loss function . Informally , gradient boosting can be thought of as performing gradient descent in functional space . The set of weak learnersH is usually formed by shallow decision trees . Decision trees are built by a recursive partition of the feature space into disjoint regions called leaves . This partition is usually constructed greedily to minimize the loss function ( 3 ) . Each leaf Rj of the tree is assigned to a value aj , which estimates the response y in the corresponding region . In our case , aj is equal to the average negative gradient value in the leaf Rj . To sum up , we can write h ( x ) = ∑ j aj1 { x∈Rj } . 3 GBDT MEETS GNN . GBDT : 𝑓 Node features 𝑿 GNN : 𝑔𝜃 𝑓 ( 𝑿 ) Pass to GNN 𝐺 , 𝑿′ Loss 𝐿 𝑔𝜃 𝐺 , 𝑿 ′ , 𝑌 New tree : 𝑓𝑖 𝜕𝐿 𝑔𝜃 𝐺 , 𝑿 ′ , 𝑌 𝜕𝑿′ Compute update 1 2 3 4 Gradient boosting approach is successful for learning on tabular data ; however , there are challenges of applying GBDT on graph-structured data : ( i ) how to propagate relational signal , in addition to node features , to otherwise inherently tabular model ; and ( ii ) how to train it together with GNN in an end-to-end fashion . Indeed , optimizations of GBDT and GNN follow different approaches : the parameters of GNN are optimized via gradient descent , while GBDT is constructed iteratively , and the decision trees remain fixed after being built ( decision trees are based on hard splits of the feature space , which makes them non-differentiable ) . A straightforward approach would be to train the GBDT model only on the node features and then use the obtained predictions , jointly with the original input , as new node features for GNN . In this case , the graph-insensitive predictions of GBDT will further be refined by a graph neural network . This approach ( which we call Res-GNN ) can already boost the performance of GNN for some tasks . However , in this case , the GBDT model completely ignores the graph structure and may miss descriptive features of the graph , providing inaccurate input data to GNN . In contrast , we propose end-to-end training of GBDT and GNN called BGNN ( for Boost-GNN ) . As before , we first apply GBDT and then GNN , but now we optimize both of them , taking into account the quality of final predictions . The training of BGNN is shown in Figure 1 . Recall that one can not tune already built decision trees due to their discrete structure , so we iteratively update the GBDT model by adding new trees that approximate the GNN loss function . In Algorithm 1 , we present the training of BGNN that combines GBDT and GNN for any node-level prediction problem such as semi-supervised node regression or classification . In the first iteration , we build a GBDT model f1 ( x ) with k decision trees by minimizing the loss function LGBDT ( f1 ( x ) , y ) ( e.g. , RMSE for regression or cross-entropy for classification ) averaged over the train nodes , following the equations ( 2 ) - ( 3 ) . Using all predictions f1 ( X ) , we update the node features to X′ that we pass to GNN . Possible update functions that we experiment with include concatenation with the original node features and their replacement by f1 ( X ) . Next , we train a graph neural network gθ on a graph G with node features X′ by minimizing LGNN ( gθ ( G , X′ ) , Y ) with l steps of gradient descent.1 Importantly , we optimize both the parameters θ of GNN and the node features X′ . Then , we use the difference between the optimized node features X′new and the input node features X ′ = f1 ( X ) as the target for the next decision trees built by GBDT . If l = 1 , the difference X′new −X′ exactly equals the negative gradient of the loss function w.r.t . the input features X′ multiplied by the learning rate η : X′new = X ′ − η ∂LGNN ( gθ ( G , X ′ ) , Y ) ∂X′ . In the second iteration , we train a new GBDT model f2 with the original input features X but new target labels : X′new −X′ . Intuitively , f2 fits the direction that would improve GNN prediction based on the first predictions f1 ( X ) . In other words , GBDT approximates the gradient steps made by GNN for the node features X′ . This is a regression problem , so here LGBDT is the RMSE loss . After f2 is trained , we combine the predictions f ( X ) = f1 ( X ) + f2 ( X ) and pass the obtained values X′ to GNN as node features . GNN model gθ again does l steps of backpropagation and passes the new difference X′new −X′ as a target to the next iteration of GBDT . In total , the model is trained for N epochs and outputs a GBDT model f : X 7→ Y and GNN model gθ : ( G , X ) 7→ Y , which can be used for downstream tasks . Intuitively , BGNN model consists of two consecutive blocks , GBDT and GNN , which are trained end-to-end , and therefore can be interpreted from two angles : GBDT as an embedding layer for GNN or GNN as a parametric loss function for GBDT . In the former case , GBDT transforms the original input features X to new node features X′ , which are then passed to GNN . In the latter case , one can see BGNN as a standard gradient boosted training where GNN acts as a complex loss function that depends on the graph topology .

Summary: The paper proposes a GNN model by incorporating gradient boosting. In the proposed BGNN, the input feature on the graph is learned by the gradient boosting model. The processed feature then becomes a new feature for a GNN model following the gradient boosting. Several experiments demonstrate the improvement of the performance for a tabular feature and graph-structured datasets. The running time of Res-GNN/BGNN is shown to have a significant reduction as compared to the plain GNN methods.