# XGBoost-Enhanced Graph Neural Networks: A New Architecture for Heterogeneous Tabular Data
## 3 Models and Methods  
Although the gradient enhancement-based approach is successful for learning tabular data, it has technical difficulties in graph-structured data applications. For example, how do you effectively integrate the relational information between data points into traditional tabular data models? How can we jointly train XGBoost and GNN? For the first problem, our scheme is to transform the tabular data into a graph structure, in which each data point serves as a node and the relationships between nodes serve as edges. If the data points have time-series relationships, we can define edges based on specific relationship signals, such as measuring the similarity between two data points by distance or similarity and constructing edges based on temporal order. For the second problem, our approach involves conducting comparative experiments, specifically Res-GNN, in which we directly train the node features on the XGBoost model. We then use these predictions to create new node features in the GNN, incorporating the initial inputs. However, the XGBoost model completely ignores the graph structure and misses some of the graph features, resulting in a failure to provide correct input information. XGNN, on the other hand, trains both XGBoost and GNN simultaneously, iteratively updating the XGBoost model by adding new trees that approximate the GNN loss function, as depicted in Figure 1.  
Algorithm 1 outlines of the XGNN training process. ```
1. Input: Graph \(G\), node features X, targets \(Y\)
2. Initialize XGBoost targets y = Y
3. for epoch i = 1 to 3 do
4. # Train mrees of XGBoosts with Equations (2) and (3)
5. \(f^{i}\gets arg\,min\,\textsc{XGBoost}(f^{i}(X),\hat{Y})\)
6. \(f\gets f^{i}+f\)
7. # Train lsteps of GNN
8. \(\mathsf{X}^{\prime}\gets f(\mathsf{X})\)
9. \(\mathsf{X}^{\prime}\leftarrow arg\,min\,\textsc{L}_{\text{GNN}}(g_{\theta}(G, \mathsf{X}^{\prime}),Y)\)
10. # Update targets for next iteration of XGBoosts
11. \(y\leftarrow\mathsf{X}^{\prime}\gets f(\mathsf{X})\)
12. end for
13. Output: Models XGBoost\(f\) and GNN \(g_{\theta}\)
```  
**Algorithm 1** Training of XGNN  
Figure 1: Training of XGNN. In Algorithm 1, we suggest a new XGNN model that combines the best features of XGBoost and graph neural networks (GNNs). Its goal is to quickly solve prediction problems at the node level, such as semi-supervised node regression and classification tasks. The first inputs are graph \(G\), node features X, and target \(Y\). In the first iteration, by minimizing the loss function \(L_{\text{XGBoost}}(f^{i}(\text{X}),\hat{Y})\), we construct the XGBoost model, which contains m decision trees \(f^{1}(\text{x})\). RMSE, or cross entropy of classification, is averaged over the training set using Equations (1) and (2), where \(f^{t-1}\) the model was constructed after the previous iteration, and \(g^{t}\) is the weak learner. \[f^{t}(\text{x})=f^{t-1}(\text{x})+\epsilon g^{t}(\text{x}), \tag{1}\]  
\[g^{t}=\underset{h\in\mathcal{H}}{\text{argmin}}\sum\
olimits_{i}(-\frac{ \partial L(f^{t-1}(\text{x}_{i}),y_{i})}{\partial f^{t-1}(\text{x}_{i})}-g( \text{x}_{i}))^{2}\,, \tag{2}\]  
Next, we update the node feature \(\text{X}^{\prime}\) based on the prediction of \(f^{1}\) (x) and feed it to the GNN. We minimize the graph neural network based on graph \(G\) by reducing its loss function of the \(g_{\theta}\). The node features of the GNN are subjected to \(l\) round gradient descent optimization, thus optimizing the parameters of the GNN \(\theta\) and node features \(\text{X}^{\prime}\). Using Equation (3), we define the optimized node feature \(\text{X}^{\prime}_{\text{new}}\) difference is the difference with the original input feature \(\text{X}^{\prime}=f^{1}(\text{X})\) difference, which serves as the objective of XGBoost new tree construction, where \(\eta\) the learning rate is determined.