# XGBoost-Enhanced Graph Neural Networks: A New Architecture for Heterogeneous Tabular Data
## 1 Introduction  
As AI becomes more prevalent in many real-world applications, tabular data storage is becoming more common. A table format organizes tabular data. It is also commonly known as structured data. Tabular data are a commonly used data type in various fields, including medicine [1], finance [2], online advertising [3], and recommender systems [4]. They usually consist of rows and columns, where each row represents a data instance, and each column represents a feature or attribute. Columns in tabular data can contain different types of data, including numeric, categorical, textual, date-time, etc. Tables can mix these different data types. In recent years, with the rapid development of deep learning in text, image, and audio, people have started to show great interest in its application to tabular data. However, the effectiveness of deep learning when working with tabular data is largely dependent on the homogeneity of the input data, as well as the fact that the structure used to organize the information provides insight into the data. Deep learning performs better on tabular data only when these conditions are met. Integrated tree-based models, such as GBDT [5], XGBoost [6], CatBoost [7], and LightGBM [8], have achieved state-of-the-art SOTA on tabular data, excelling in competitive prediction accuracy and fast training speed. However, tree-based methods also have limitations in some specific scenarios, such as in the case of continuous learning or reinforcement learning, when the tabular data are only a part of the model input or when the tabular data also include information such as images, text, or audio. In these cases, it is necessary to consider other methods, such as graph neuralnetworks (GNNs), which do not depend on the order of nodes but also take into account both the neighborhood information of the nodes and the node features for prediction. The following key attributes and features of XGBoost contribute to its success when handling tabular data: (1) Automatic handling of missing values. When splitting nodes, XGBoost seeks the optimal method to fill in the missing values, ensuring the model's performance remains unaffected. (2) Feature importance scoring. It helps users understand which features contribute most to model prediction. This is useful for feature selection and model interpretation. (3) Regularization. Additionally, the model uses a gradient boosting framework and L1 and L2 regularization to effectively avoid overfitting and improve its generalization abilities. (4) Gradient enhancement framework. We utilize an optimized tree structure to facilitate efficient model training and prediction, thereby enhancing overall performance. In contrast, a key feature of GNN is its ability to directly process graph-structured data, effectively capturing and utilizing the relationships between nodes and edges in graph-structured data. XGBoost primarily handles tabular data, unlike GNNs. A GNN requires converting textual data into graph structures, such as word co-occurrence graphs, dependent syntactic graphs, or semantic graphs, rather than using raw tabular data directly. Although this preprocessing differs from XGBoost's feature engineering, it is also a critical step for the algorithm's success. It goes without saying that both the XGBoost and GNN methods have their own benefits. Is it possible to combine the benefits of both models? The current work is the first to use the XGBoost model for graph-structured data on text categorization and prediction tasks, to the best of our knowledge. In this paper, we propose a new architecture, XGNN--a joint training of XGBoost and GNN models. It is possible to achieve end-to-end optimization by using XGBoost's heterogeneous feature processing and GNN's graph structure processing. This makes the model work better overall. Here is a summary of our contributions:  
1. We introduce XGNN, a graph neural network model for tabular data, to jointly train the XGBoost and GNN models.