# XGBoost-Enhanced Graph Neural Networks: A New Architecture for Heterogeneous Tabular Data
## 3 Models and Methods
### Extreme Gradient Boosting (XGBoost)  
XGBoost is an efficient gradient boosting decision tree algorithm based on GBDT (gradient boosting decision tree), which is based on the model of a boosting algorithm. Boosting is the process of accumulating the weak learners generated at each time and weighting them to the total model to form a strong learner, which can be used for regression and classification problems. The basic idea of XGBoost is the same as GBDT but optimized in these aspects, including second-order derivatives to make the loss function more accurate, regular terms to avoid tree overfitting, block storage that can be computed in parallel, and so on. Firstly, the objective function is defined as Equation (4); Equation (5) is the Taylor expansion; Equation (6) is the complexity of a tree; and Equation (7) is the definition of a tree.  
\[\text{obj}(\theta)=\sum\
olimits_{i}^{n}l(y_{i},y_{i})+\sum\
olimits_{k=1}^{ K}\Omega(f_{k}), \tag{4}\]  
\[\sum\
olimits_{i=1}^{n}\biggl{[}g_{i}f_{t}(x_{i})+\frac{1}{2}h_{i}f_{t}^{2}(x_ {i})\biggr{]}+\Omega(f_{t}), \tag{5}\]  
\[\Omega(f_{t})=\gamma T+\frac{1}{2}\lambda\sum\
olimits_{j=1}^{T}w_{j}^{2}, \tag{6}\]  
\[f_{t}(x)=w_{q(x)}w\in R^{T},q:R^{d}\rightarrow\{1,2,\cdots,T\}, \tag{7}\]  
where \(K\) denotes the number of trees, and \(f\) denotes the function space \(F\) in which a function representing an abstract structure like a tree. \(l\) is our loss function, and \(\Omega\) is the penalty term. \(g_{i}\) and \(h_{i}\) are the first- and second-order derivatives of our loss function with respect \(\cdot\cdot\cdot(t-1)\) to \(yi\), the first- and second-order derivatives of \(w\), which denotes the weight vector. \(\gamma T\) is the number of leaves and \(\frac{1}{2}\lambda\sum\
olimits_{j=1}^{T}w_{j}^{2}\) is the square of the L2 module of W, and \(q\) denotes the leaf node mapping relation corresponding to each data sample. At the same time, we define \(R^{d}\rightarrow\{1,2,\cdots,T\}\) to denote the set of nodes that a given sample maps to. Since multiple samples will fall into a single node, we change the scope of the definition of the objective function from n samples to T nodes.  
Then, the above objective function is processed to obtain Equation (8); if the gain \(>0\), it means that the cut object is smaller, and the model is better.  
\[\text{Gain}\ =\frac{1}{2}\bigg{[}\frac{G_{L}^{2}}{H_{L}+\lambda}+\frac{G_{R}^{ 2}}{H_{R}+\lambda}-\frac{(G_{L}+G_{R})^{2}}{H_{L}+H_{R}+\lambda}\bigg{]}-\gamma, \tag{8}\]  
Although XGBoost can use pre-sorting and approximation algorithms to reduce the computation when finding the optimal cut-points of leaf minima, the time overhead is still large because it needs to traverse the whole dataset. Moreover, the space complexity of pre-sorting is high, as it not only needs to store the feature values but also the indexes of the gradient statistics of the samples corresponding to the features, resulting in an exponential increase in memory consumption. The introduction of graph neural networks (GNNs) can solve these problems to some extent, especially in feature engineering and complex feature relationship processing.