Convergent Boosted Smoothing for Modeling GraphData with Tabular Node Features

1 INTRODUCTION . Tabular data consists of observations stored as rows of a table , where multiple numeric/categorical features are recorded for each observation , one per column . Models for tabular data must learn to output accurate predictions solely from ( potentially high-dimensional or sparse ) sets of heterogeneous feature values . For learning from tabular data , ensembles of decision trees frequently rank on the top of model leaderboards , as they have proven to be highly performant when trained via multi-round boosting algorithms that progressively encourage the learner to focus more on “ difficult ” examples predicted inaccurately in earlier rounds ( Bansal , 2018 ; Elsayed et al. , 2021 ; Fakoor et al. , 2020 ; Huang et al. , 2020b ; Ke et al. , 2017 ; Prokhorenkova et al. , 2018 ) . Boosted trees thus form the cornerstone of supervised learning when rows of a table are independent and identically distributed ( iid ) . However , the iid assumption is severely violated for graph structured data ( Chami et al. , 2020 ) , in which nodes store features/labels that are highly correlated with those of their neighbors . Many methods have been proposed to account for the graph structure during learning , with graph neural networks ( GNN ) becoming immensely popular in recent years ( Scarselli et al. , 2008 ; Wang et al. , 2019 ; Zhou et al. , 2020a ) . Despite their success , modern GNNs suffer from various issues ( Alon & Yahav , 2021 ; Oono & Suzuki , 2019 ) : they are complex both in their implementation and the amount of required computation ( Faber et al. , 2021 ; Huang et al. , 2020a ; Shchur et al. , 2018 ) , and limited theoretical guarantees exist regarding their performance or even the convergence of their training ( Garg et al. , 2020 ; Li & Cheng , 2021 ; Zhou et al. , 2020b ) . Furthermore , the sample complexity of sophisticated GNN models and their ability to handle rich node features may be worse than simpler models like those used for tabular data ( Faber et al. , 2021 ; Huang et al. , 2020a ; Shchur et al. , 2018 ) . Graph datasets store tabular feature vectors , along with additional edge information . Despite the close relationship between graph and tabular data , there has been little work on how to adapt powerful boosting methods – our sharpest tool for dissecting tabular data – to account for edges . Here we consider how to best leverage tabular models for non-iid node classification/regression tasks in graphs . We focus on settings where each node is associated with tabular ( numeric/categorical ) features x and labels y , and the goal is to predict the labels of certain nodes . Straightforward application of tabular modeling ( in which nodes are treated as iid examples ) will produce a learned mapping where training and inference are uninformed by how nodes are arranged in the graph . To account for the graph structure , we introduce simple graph propagation operations into the definition of a modified , non-iid boosting loss function , such that edge information can be exploited to improve model accuracy relative to classical alternatives ( Friedman , 2001 ) . We refer to the resulting algorithm as EBBS which stands for efficient bilevel boosted smoothing , whereby the end-to-end training of a bilevel loss is such that values of the boosted base model f ebb and flow across the input graph producing a smoothed predictor f̃ . And unlike an existing adaptation of boosting for graph data ( Ivanov & Prokhorenkova , 2021 ) , our approach is simpler ( no GNN or other auxiliary model required ) , produces more accurate predictions on benchmark datasets , and enjoys reliable convergence guarantees . 2 BACKGROUND . Consider a graph G with a set of vertices , i.e . nodes , V , { v1 , v2 , . . . , vn } whose connectivity is described by the edge set E , { ( vi , vj ) ∈ V × V } , where the tuple of nodes ( vi , vj ) implies a relationship among vi and vj . The edge set is represented by the adjacency matrixA = ( aij ) n×n ∈ Rn×n , whose entry aij is nonzero if ( vi , vj ) ∈ E . In node prediction tasks , each node vi is associated with a feature vector xi ∈ Rd0 , as well as a label yi ∈ Rc that might hold discrete or continuous values ( corresponding to classification or regression tasks ) . In certain cases we have access to the labels at only a subset of nodes { yi } i∈L , with L ⊂ V . Given { yi } i∈L , the connectivity of the graph E , and the feature values of all nodes { xi } i∈V , our task is to predict the labels of the unlabeled nodes { yi } i∈U , with U = V \ L. In the most straightforward application , a tabular model may simply be fit to dataset { ( xi , yi ) } i∈L , disregarding that the observations are not iid . For prediction , the learned model is then independently applied to each xi for i ∈ U . Of course this naive approach may fare poorly as it fails to leverage E , but we note that it empirically performs better than one might expect on certain graph datasets ( where perhaps each xi contains sufficiently rich information to infer the corresponding yi , or nodes connected in E only exhibit weak correlation in their features or labels ) ( Faber et al. , 2021 ; Huang et al. , 2020a ; Ivanov & Prokhorenkova , 2021 ) . 2.1 GRADIENT BOOSTED DECISION TREES ( GBDT ) . GBDT represents a popular model for iid ( non-graph ) tabular data ( Friedman , 2001 ) , whereby remarkably accurate predictions are produced by an ensemble of weak learners . Formally , at each iteration t of boosting , the current ensemble model f ( t ) ( x ) is updated in an additive manner via f ( t+1 ) ( x ) = f ( t ) ( x ) + η ( t ) h ( t ) ( x ) , where h ( t ) ( x ) is a weaker learner selected from some candidate function spaceH ( typically decision trees ) , and η ( t ) is the learning rate calculated with the aid of line search . The weak learner h ( t ) ∈ H is chosen to approximate the pseudo-residuals given by the negative gradient of some loss function ` w.r.t the current model ’ s predictions . This involves solving h ( t ) = arg min h∈H m∑ i=1 ∥∥∥∥−∂ ` ( yi , f ( t ) ( xi ) ) ∂f ( t ) ( xi ) − h ( xi ) ∥∥∥∥2 2 , ( 1 ) where { xi , yi } mi=1 is a set of m training points . Decision trees are constructed by a recursive partition of feature space into Jt disjoint regions R1t , ... , RJtt to minimize the loss function ( 1 ) and predict a constant value bjt in each region Rjt . The output of h ( t ) ( x ) can be written as the sum : h ( t ) ( x ) = ∑Jt j=1 bjt1Rjt ( x ) , where 1 is the indicator notation . Many efficient GBDT implementations are available today ( Ke et al. , 2017 ; Prokhorenkova et al. , 2018 ; Wen et al. , 2019 ) , and these models can easily be trained for nonstandard prediction tasks via custom loss functions ( Elsayed et al. , 2021 ; Li et al. , 2007 ; Velthoen et al. , 2021 ) . 2.2 PRIOR ATTEMPTS TO COMBINE BOOSTING WITH GRAPH NEURAL NETWORKS . The AdaGCN model from Sun et al . ( 2019 ) proposes a GNN architecture motivated by the structure of AdaBoost iterations ; however , this approach is not actually designed to handle tabular data . More related to our work is the boosted graph neural network ( BGNN ) approach from Ivanov & Prokhorenkova ( 2021 ) , which proposes a novel architecture that jointly trains GBDT and GNN models . Within this framework , GBDT extracts predictive information from node features and the GNN accounts for the graph structure , achieving a significant performance increase on various graph datasets with tabular features . In the first iteration , BGNN builds a GBDT model f ( 1 ) ( x ) over the training node features that are treated essentially as iid tabular data . Using all GBDT predictions , BGNN then concatenates these with the original node features and subsequently uses these as inputs to a GNN model . Next , the GNN is trained with l steps of gradient descent to optimize network parameters as well as the input node features themselves by backpropagating gradients entirely through the GNN into its input space . BGNN uses the difference between the optimized node features and input node features as a new target value for the next decision tree in the subsequent boosting round . After GBDT has been updated with the addition of the new tree , it makes predictions that are used to augment the node features for subsequent use in the GNN , and the process repeats until some stopping criteria is met . While BGNN has thus far demonstrated promising empirical results , there is no guarantee of convergence or even cost function descent . Our EBBS model addresses this issue through a fully integrated alternative described in Section 3 , which enjoys convergence guarantees provided in Section 4 . Besides its favorable analytical properties , EBBS naturally supports a suite of graph-based regularizers promoting different properties described in Section 3.4 , empirically achieves higher accuracy , and is less prone to overfitting without careful hyperparameter-tuning since the graph is accounted for via a simple regularizer instead of an additional parameterized GNN model . Appendix B provides further discussion about the connection between EBBS and BGNN , highlighting key technical differences . 3 END-TO-END INTEGRATION OF GRAPH PROPAGATION AND BOOSTING . In this section we describe our method for combining GBDT with graph-aware propagation layers . We first describe a general family of propagation layers that mimic gradient descent steps along a principled graph-regularized loss , followed by the derivation of a bilevel optimization algorithm that exploits these layers . In terms of notation , we will henceforth usemi to reference the i-th row of an arbitrary matrixM . 3.1 GRAPH-AWARE PROPAGATION LAYERS INSPIRED BY GRADIENT DESCENT . Recently there has been a surge of interest in GNN architectures with layers defined with respect to the minimization of a principled class of graph-regularized energy functions ( Klicpera et al. , 2018 ; Ma et al. , 2020 ; Pan et al. , 2021 ; Yang et al. , 2021 ; Zhang et al. , 2020 ; Zhu et al. , 2021 ) . In this context , the basic idea is to associate each descent step along an optimization trajectory ( e.g. , a gradient descent step , power iteration , or related ) , with a GNN layer , such that in aggregate , the forward GNN pass can be viewed as minimization of the original energy . Hence GNN training can benefit from the inductive bias afforded by energy function minimizers ( or close approximations thereof ) whose specific form can be controlled by trainable parameters . The most common instantiation of this framework , with roots in Zhou et al . ( 2004 ) , begins with the energy ` Z ( Z ) , ‖Z − f ( X ; θ ) ‖2F + λtr [ Z > LZ ] , ( 2 ) where λ is a trade-off parameter , Z ∈ Rn×d is a learnable embedding of d-dimensional features across n nodes , and f ( X ; θ ) denotes a base model ( parameterized by θ ) that computes an initial target embedding based on the d0-dimensional node featuresX ∈ Rn×d0 . We also define the graph Laplacian of G as L ∈ Rn×n , meaning L = D −A , whereD represents the degree matrix . Intuitively , solutions of ( 2 ) reflect a balance between proximity to f ( X ; θ ) and minimal quadratic differences across graph edges as enforced by the trace term tr [ Z > LZ ] = ∑ { i , j } ∈E ‖zi − zj‖ 2 2 ; more general regularization factors are considered in Section 3.4 . On the positive side , ( 2 ) can be solved in closed-form via f̃∗ ( X ; θ ) , arg min Z ` Z ( Z ) = P ∗f ( X ; θ ) , with P ∗ , ( I + λL ) −1 . ( 3 ) However , for large graphs the requisite inverse is not practically-feasible to compute , and instead iterative approximations are preferable . To this end , we may initialize as Z ( 0 ) = f ( X ; θ ) , and then proceed to iteratively descend in the direction of the negative gradient . Given that ∂ ` Z ( Z ) ∂Z = 2λLZ + 2Z − 2f ( X ; θ ) , ( 4 ) the k-th iteration of gradient descent becomes Z ( k ) = Z ( k−1 ) − α [ ( λL+ I ) Z ( k−1 ) − f ( X ; θ ) ] , ( 5 ) where α2 serves as the effective step size . Given that L is generally sparse , computation of ( 5 ) can leverage efficient sparse matrix multiplications , and we may also introduce modifications such as Jacobi preconditioning to speed convergence ( Axelsson , 1996 ; Yang et al. , 2021 ) . Furthermore , based on well-known properties of gradient descent , if k is sufficiently large and α is small enough , then f̃∗ ( X ; θ ) ≈ f̃ ( k ) ( X ; θ ) , P ( k ) [ f ( X ; θ ) ] , ( 6 ) where the operator P ( k ) ( · ) computes k gradient steps via ( 5 ) . The structure of these propagation steps , as well as related variants based on normalized modifications of gradient descent , equate to principled GNN layers , such as those used by GCN ( Kipf & Welling , 2016 ) , APPNP ( Klicpera et al. , 2018 ) , and many others , which can be trained within a broader bilevel optimization framework as described next .

Summary: The authors propose a new method for integrating graph-based models with boosting. This is done using the typical method involving residuals and weak-learners, but adding a step where information is propagated in the graph. The approach is also simple, as no GNNs or other auxiliary models are required. It is also shown how the meta-loss introduced by the authors provides convergence given some moderate assumptions. According to the experiments reported, the proposed model is better than the current state of the art in the considered domain.