SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ensemble trees with neighbor aggregation have superior performance. As highlighted in Figure 1, XGB-Graph and RF-Graph consistently surpass other compared models across all metrics using default hyperparameters. The performance gap becomes particularly significant in the fully-supervised setting, i.e., XGB-Graph surpasses BWGNN—the best GNN model in this setting—by an absolute average improvement of  $2.0\%$  on AUROC,  $12.9\%$  on AUPRC, and  $9.8\%$  on  $\mathrm{Rec}@\mathrm{K}$ . In the semi-supervised context, RF-Graph presents an absolute average improvement of  $2.8\%$  on AUROC,  $8.0\%$  on AUPRC, and  $3.1\%$  on  $\mathrm{Rec}@\mathrm{K}$ , as compared to GHRN, the best GNN model in this setting. It is important to highlight that the improvement in AUPRC and  $\mathrm{Rec}@\mathrm{K}$  is more pronounced than that in AUROC due to the imbalanced issue, suggesting that RF-Graph and XGB-Graph are more proficient in predicting top- $k$  high-confidence anomalies. Further, as shown in the bottom of Figure 1, tree ensembles with neighbor aggregation not only outperform GNNs in terms of efficiency but also exhibit lower memory consumption. Although the performance gap of different models narrow after hyperparameter tuning as in Table 4, RF-Graph and XGB-Graph still prevail among 6 out of 10 datasets. Accordingly, our observations show the superior effectiveness and efficiency of RF-Graph and XGB-Graph across diverse GAD datasets and scenarios.
Most standard GNNs prove unsuitable for GAD. As shown in Figure 1, it becomes clear that the majority of standard GNNs encounter difficulties when dealing with GAD tasks. To illustrate, the performance of GCN and GIN is on par with that of MLP—a method that does not take graph structure information into account. This indicates that standard GNNs often struggle to effectively handle structure camouflage or feature heterophily problems induced by anomalies. In Table 4, while hyperparameter tuning does improve the results of all standard GNNs, they remain subpar compared to methods in other categories. An exception is GraphSAGE, which displays an average absolute improvement of  $10.4\%$  on average AUPRC when optimal hyperparameters are used, making it competitive with specialized GNNs. BGNN, while impressive on specific datasets such as T-Social, exhibits poor performance on other datasets. This inconsistency may result from the inherent instability of its joint training scheme, especially when compared to the more stable two-step approach in XGB-Graph and RF-Graph.
Specialized GNNs require hyperparameter tuning to achieve satisfactory performance. Generally, specialized GNNs outperform standard GNNs in Figure 1, indicating that GNNs tailored for GAD can indeed enhance anomaly detection capabilities. However, the performance of these
Figure 1: Comparison of the anomaly detection performance, wall-clock time (on all datasets), and peak CPU/GPU memory utilization (on DGraph-Fin) among all models with default hyperparameters. Top three lines are in semi-supervised settings and the others are in fully-supervised settings. The color of the box plot represents the average score for each metric, while the central line within the box indicates the median score.
specialized GNNs strongly depends on hyperparameter tuning. As indicated in the last column of Table 4, all these methods witness a performance improvement after a hyperparameter search. For instance, when optimized hyperparameters are employed, BWGNN can surpass RF-Graph and XGB-Graph on the Reddit dataset. This demonstrates that under certain conditions, some specialized GNNs can deliver commendable performance. However, as illustrated at the bottom of Figure 1, these GNNs often demand more training time and memory. The inherent limitations of hyperparameter
Table 4: Comparison of the AUPRC score of each model with optimal hyperparameters through random search. Best results are highlighted in bold. In the last two columns, Ave. signifies the average score across the first 9 datasets without T-Social, while Imp. denotes the absolute increase in this average score when compared to the default hyperparameters. OOT means the model could not complete training within a day. Results for other metrics can be found in Appendix D and Table 13.
Model	Reddit	Weibo	Amazon	Yelp.	T-Fin.	Ellip.	Tolo.	Quest.	DGraph.	T-Social	Ave.	Imp.
MLP	5.91	84.88	87.34	47.68	74.21	43.77	38.29	15.34	2.69	9.69	44.46	2.67
KNN	6.12	81.12	84.41	54.39	74.97	60.98	35.30	15.37	1.67	36.32	46.04	9.13
SVM	6.88	84.91	85.80	41.01	78.10	20.98	37.90	15.37	2.65	OOT	41.51	4.53
RF	4.63	93.52	91.18	77.77	81.99	78.42	38.64	14.37	2.57	41.56	53.68	0.81
XGBoost	5.56	94.49	91.88	84.00	82.64	76.93	40.05	16.24	2.75	16.60	54.95	0.73
XGBOD	8.27	95.70	92.15	79.46	82.32	74.86	40.65	16.08	1.95	OOT	54.61	1.62
NA	9.70	94.09	91.56	63.93	88.78	29.14	51.06	14.32	4.13	79.21	49.64	3.38
GCN	4.63	94.64	45.65	20.88	78.22	25.37	40.57	14.06	3.80	76.35	36.42	1.54
SGC	6.04	91.16	42.69	19.87	68.68	17.82	39.59	10.53	2.49	16.28	33.21	5.66
GIN	6.41	91.67	84.61	33.63	78.35	26.21	40.36	13.68	3.47	60.79	42.04	2.57
GraphSAGE	5.56	94.02	82.45	46.64	84.71	57.82	51.41	17.50	3.77	75.32	49.32	10.44
GAT	7.20	92.91	87.94	43.62	82.72	27.53	45.25	15.51	3.85	32.07	45.17	2.80
GT	7.68	89.85	84.90	44.60	83.14	25.90	45.71	17.08	3.83	36.14	44.74	5.42
PNA	7.75	96.04	35.24	29.95	76.67	27.81	41.74	11.38	3.22	21.24	36.64	4.93
BGNN	6.87	95.99	67.92	29.71	83.72	62.03	45.35	9.43	4.24	99.09	45.03	0.95
GAS	4.43	96.76	81.43	35.11	85.95	29.80	47.21	15.48	3.65	62.36	44.42	6.62
DCI	7.74	91.77	85.17	39.88	63.68	27.39	37.73	14.59	3.31	12.97	41.25	1.01
PCGNN	7.73	89.07	89.33	44.51	83.31	42.66	44.85	15.59	3.42	80.29	46.72	4.69
BernNet	7.82	92.38	84.89	51.92	89.17	38.25	43.69	17.25	3.27	44.30	47.63	2.90
AMNet	7.87	94.99	88.36	46.86	88.87	25.18	40.74	15.63	2.81	37.70	45.70	2.49
GAT-sep	7.19	93.40	84.72	45.49	84.01	26.35	46.66	17.90	3.84	33.39	45.50	2.98
BWGNN	8.32	94.01	91.48	61.53	89.38	29.31	49.58	18.57	3.97	78.93	49.57	2.12
GHRN	4.66	95.27	89.52	55.42	87.60	43.90	47.45	18.31	3.80	86.78	49.55	1.77
RF-Graph	5.13	96.95	90.53	83.92	89.23	78.86	52.34	14.44	2.15	97.63	57.06	1.21
XGB-Graph	5.29	97.06	93.33	91.11	90.12	77.78	53.92	18.19	3.79	97.34	58.95	1.34
search also pose significant challenges, especially in real-world applications where there might be a scarcity of annotated labels or computational resources. Given these constraints, tree ensembles with neighborhood aggregation might still be the preferred choice.
NA is a versatile technique adaptable to any model in GADBench. We apply it to XGBoost and observe a remarkable enhancement in the semi-supervised setting, where the average AUPRC across 10 datasets increases from  $37.5\%$  to  $38.9\%$ . However, the boost is not significant in the fully-supervised setting. These findings highlight NA's potential as an effective strategy to address challenges associated with label scarcity. For additional results related to the application of NA on other models, please refer to Appendix D.
Finally, we observe that all methods perform poorly on the DGraph-Fin dataset. This can be attributed to the highly imbalanced and sparse graph structure, with an average degree of 1.16. Furthermore, we find that node features in this dataset are highly indistinguishable, as nearly all anomalous nodes share identical features with normal nodes. Indeed, the AUROC scores of all models on this dataset align with those reported in the original paper, as demonstrated in Appendix D.
4.2 Specialized Experiments in Heterogeneous and Inductive Settings
Table 5: Comparison of heterogeneous and homogeneous GAD methods on Amazon and Yelp datasets under both fully-supervised and semi-supervised settings. Results are averaged over 10 runs.
Model	Amazon (Semi-Supervised)	Amazon (Fully-Supervised)	Yelp (Semi-Supervised)	Yelp (Fully-Supervised)
AUROC	AUPRC	Rec@K	AUROC	AUPRC	Rec@K	AUROC	AUPRC	Rec@K	AUROC	AUPRC	Rec@K
GAT	92.44	81.57	77.07	96.66	86.67	83.10	65.56	25.03	28.08	79.50	43.41	43.65
BWGNN	91.83	81.68	77.71	97.95	89.09	85.00	64.30	23.66	26.44	84.89	55.06	52.18
RGCN	84.17	41.07	45.57	92.03	67.97	65.49	72.20	26.46	28.54	78.34	34.57	34.98
HGT	79.75	38.13	45.07	89.64	71.46	70.22	72.83	28.49	31.59	89.62	62.63	57.75
CARE-GNN	86.00	58.95	59.12	90.84	72.64	67.72	91.19	68.69	65.35	95.23	81.06	74.85
H2Detector	71.00	29.27	32.61	78.66	39.95	44.35	67.28	22.23	24.08	89.07	59.40	57.54
XGB-Graph	94.68	84.38	78.17	98.69	92.61	85.87	64.03	24.84	26.81	96.22	87.03	78.76
Table 6: Performance comparison on the Elliptic and DGraph-Fin datasets under inductive and transductive settings, with all results being averaged over 10 runs.
Model	Elliptic (Inductive)	Elliptic (Transductive)	DGraph-Fin (Inductive)	DGraph-Fin (Transductive)
AUROC	AUPRC	Rec@K	AUROC	AUPRC	Rec@K	AUROC	AUPRC	Rec@K	AUROC	AUPRC	Rec@K
GCN	75.79	14.97	16.73	92.40	73.87	69.99	73.99	3.35	5.61	75.85	3.99	7.05
GraphSAGE	79.51	19.64	20.59	82.85	34.76	45.95	72.66	3.06	5.43	75.63	3.76	6.97
BWGNN	82.29	22.49	28.26	96.12	86.58	81.14	73.85	3.24	5.83	76.26	4.01	7.52
GHRN	84.74	25.42	28.54	96.05	86.57	81.11	76.20	4.03	7.48	76.14	3.99	7.54
XGB-Graph	90.36	76.20	70.64	96.80	89.58	84.59	71.23	2.81	5.33	74.64	3.66	6.75
Dealing with heterogeneous graphs. We evaluated the performance of four GAD methods that consider heterogeneity—RGCN, HGT, CARE-GNN, and H2-FDetector—using the Amazon and Yelp datasets, each comprising three different types of edges as detailed in Appendix B. For comparison, we also tested three other methods—GAT, BWGNN, and XGB-Graph—that treat all edge types equivalently. As shown in table 5, considering heterogeneity does not enhance performance on the Amazon dataset, but it leads to improvements on the Yelp dataset. Specifically, CARE-GNN outperforms all other methods under label-scarce conditions.
Performance in the inductive setting. Our primary experiments focus on the transductive setting, characterized by the assumption that all nodes are visible in the training process. To offer a holistic evaluation, we also conduct experiments in the inductive setting using DGraph-Fin and Elliptic datasets which have temporal features. In this setting, features and structures associated with test nodes are not accessible during the training phase. As presented in table 6, the model performance is generally impacted in the inductive setting of two datasets. Specifically, XGB-Graph outperforms other models across all metrics on Elliptic, while GHRN stands out as the most robust model on DGraph-Fin.
The impact of different number of neighbor aggregation layers. Figure 2 illustrates the performance change in XGB-Graph and RF-Graph with varying numbers of neighbor aggregation layers. Observably, the performance on most datasets improves when the number of neighbor aggregation layers increases from 0 to 2, confirming the effectiveness of the neighbor aggregation process. However, further increments in the number of layers do not contribute to any significant improvement in the model performance. Consequently, in most instances, two layers are adequate for XGB-Graph and RF-Graph, and utilizing more layers could lead to unnecessary computational overhead and memory usage.
Figure 2: The impact of different number of neighbor aggregation layers on the performance of XGB-Graph and RF-Graph.
4.3 Why and When Do Tree Ensembles with Neighbor Aggregation Outperform GNNs?
An initial study on decision boundaries. Inspired by a recent benchmark about ensemble trees and neural networks on the tabular data [28], we explore the possible reasons for the superior performance of ensemble trees with neighborhood aggregation. Specifically, our primary investigation focuses on the models' decision boundaries.
In the left panel of Figure 3, we visualize the decision boundaries of GIN and RF-Graph on Amazon dataset. For detailed implementations, please see Appendix E. It is observed that the normal and anomalous nodes are closely intertwined, making them hard to separate. Unfortunately, GIN tends to produce simple and smooth decision boundaries, leading to frequent misclassification of normal nodes in the right bottom corners. Differently, RF-Graph can produce more intricate decision boundaries, demonstrating greater proficiency in distinguishing anomalous data. In the right panel of Figure 3, we visualize the decision boundaries of BWGNN and XGB-Graph on Weibo dataset. As can be seen, the anomalous nodes are grouped into several dispersed clusters. With this dispersed distributions, BWGNN is hard to achieve accurate classification due to simple and continuous decision boundaries. In contrast, XGB-Graph successfully classifies anomalies within each cluster.
Figure 3: Decision boundaries comparison of different approaches. Blue points represent anomalies while red points are normal nodes. Similarly, the blue/red regions correspond to model predictions for anomalous/normal classes.
In summary, anomaly instances tend to form multiple dispersed clusters and are coupled with normal instances, which fall in the categories of the inductive bias of RF-Graph and XGB-Graph that favor complex and disjoint decision boundaries. In contrast, as GNNs typically employ MLP as the final layer, they tend to generate simple and continuous decision boundaries, which makes GNNs sub-optimal on some challenging GAD datasets.
The impact of dataset feature types on model performance. As indicated in Table 3, out of the 10 datasets in GADBench, 3 datasets purely use text embeddings as node features, while in the remaining 7 datasets, node features contain miscellaneous information such as the combination of numerical, categorical, and temporal features. Notably, for datasets that rely on text-based features—namely Reddit, Weibo, and Questions—GNNs showcase competitive performance in comparison to other methods including tree ensembles. This could be attributed to the nature of text embeddings: they often represent low-dimensional manifolds in a high-dimensional feature space, where dimensions tend to be highly correlated. A GNN can process all these dimensions simultaneously, whereas an individual decision tree might only consider a limited subset of feature columns. Conversely, in the other 7 datasets with diverse feature types that have low correlation (for instance, gender and age information), tree ensembles with neighbor aggregation typically exhibit superior performance.
In conclusion, in common GAD scenarios such as fraud detection, node features mainly originate from user profiles, which may encompass varied feature types. Thus, tree ensembles with neighbor aggregation often emerge as the preferred choice. However, for specific tasks like fake news and rumor detection, where text data is pivotal, GNNs still present a compelling option.
5 Conclusion and Future Plan
In this paper, we introduce GADBench, the first comprehensive benchmark for supervised anomalous node detection on static attributed graphs. Our evaluation of 29 models on 10 real-world datasets shows that tree ensembles with simple neighborhood aggregation generally outperform other models, including GNNs specifically designed for the GAD task. The rationale behind this finding is initially examined from the standpoints of decision boundary and node feature type. Our results challenge the prevailing belief about the superiority of GNNs in GAD and underline the importance of a fair and comprehensive comparison in accurately understanding the capabilities of various models. By making GADBench open-source, we aim to foster further research and refinement of GAD algorithms, as well as their more informed evaluations and comparisons.
We regard GADBench as a long-term evolving project and are dedicated to its continuous development. Our roadmap for the future includes expanding its scope to include a broader spectrum of GAD scenarios, incorporating more cutting-edge models, and integrating newer datasets. At present, we primarily focus on treating datasets as static graphs to ensure compatibility with most baselines. We have only embarked on preliminary studies concerning heterogeneous and inductive settings. Looking ahead, we envision extending our evaluations to more complex types of graphs and anomalies. Our ultimate goal is to transform GADBench into a more robust, scalable GAD toolbox, with advanced features like automated model selection [95].
Acknowledgement This research was supported by NSFC Grant No. 62206067, Tencent AI Lab Rhino-Bird Focused Research Program and Guangzhou-HKUST(GZ) Joint Funding Scheme 2023A03J0673.
[99] Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, 33, 2020.
[100] Arthur Zimek, Ricardo JGB Campello, and Jörg Sander. Ensembles for unsupervised outlier detection: challenges and research questions a position paper. Acm Sigkdd Explorations Newsletter, 15(1):11-22, 2014.
A Detailed Description of Models in GADBench
Classic Methods
- MLP (Multi-Layer Perceptron [67]): A type of feedforward neural network composed of multiple layers of interconnected artificial neurons that can learn and make predictions by adjusting the weights and biases of the connections between the neurons.
- KNN ( $k$ -Nearest Neighbors [18]): A non-parametric classification algorithm that assigns labels to data points based on the labels of their  $k$  nearest neighbors within the feature space.
- SVM (Support Vector Machine [15]): A supervised learning algorithm that classifies data by identifying an optimal hyperplane that maximally separates different classes in a high-dimensional feature space.
- RF (Random Forest [12]): An ensemble learning algorithm that utilizes bagging to train a collection of individual decision tree models on subsets of the original dataset. The predictions from these individual models are then combined through averaging or voting, resulting in a robust and accurate prediction model.
- XGBoost (eXtreme Gradient Boosting [16]): A gradient boosting decision tree framework that uses a gradient descent algorithm to minimize the loss function and iteratively adds new trees into the model to correct the errors made by the previous trees.
- XGBOD (Extreme Boosting Based Outlier Detection [93]): An enhanced XGBoost model that integrates unsupervised outlier detectors to generate abnormality scores as additional features for the original data. These generated features are concatenated to the original feature set and used in an XGBoost classifier for outlier detection.
- Neighborhood Averaging [88]: A method that post-processes the outlier scores provided by any existing outlier detector by averaging it with the scores of its neighbors in the feature space. In GADBench, we integrated this technique into BWGNN as a case study for in-depth analysis.
Standard GNN Architectures
- GCN (Graph Convolutional Network [39]): A method that utilizes convolution operation on the graph to propagate information from a node to its neighboring nodes, enabling the network to learn a representation for each node based on its local neighborhood.
- ChebNet ( Chebyshev Spectral Convolution Network [19]): A variant of GCN that applies Chebyshev polynomials to approximate the spectral graph convolution operator. This approach allows the model to capture both local and global graph structures, making it scalable for larger graphs.
- GIN (Graph Isomorphism Network [85]): A type of GNN that learns to capture the structure of a graph while respecting graph isomorphism. This means it generates identical embeddings for graphs that are structurally identical, regardless of permutations in their node labels.
- GraphSAGE (Graph Sample and AggregateE [30]):) A general inductive learning framework that generates node embeddings by sampling and aggregating features from a node's local neighborhood.
- GAT (Graph Attention Networks [76]): A GNN framework that incorporates the attention mechanism. It assigns varying levels of importance to different nodes during the neighborhood information aggregation process, allowing the model to focus on the most informative parts.
- GT (Graph Transformer [73]): An adaptation of the neural network architecture that applies the principles of the Transformer model to graph-structured data. It uses masks in the self-attention process to leverage the graph structure and enhance model efficiency
- PNA (Principle Neighbor Aggregation)[17]: A novel graph neural network architecture combining multiple aggregators with degree-scalers in the neighborhood aggregation process.
- BGNN (Gradient Boosting Meets Graph Neural Networks) [37]: An end-to-end framework that trains GBDT and GNN jointly by allowing new trees to fit the gradient updates of GNN.
- RGCN (Relational Graph Convolution Network) [70]: A specialized framework derived from Graph Convolutional Networks, tailored for handling relational data.
- HGT (Heterogeneous Graph Transformer) [35]: A GNN framework to model Web-scale heterogeneous graphs by designing node- and edge-type dependent parameters for characterizing heterogeneous attention over each edge, thus facilitating the maintenance of dedicated representations for different types of nodes and edges.
GNNs Specialized for Graph Anomaly Detection
- GAS (GCN-based Anti-Spam [44]): A highly scalable method for detecting spam reviews. It extends GCN to handle heterogeneous and heterophilic graphs and adapts to the graph structure of specific GAD applications using the KNN algorithm.
- DCI (Deep Cluster Infomax[82]): A self-supervised learning scheme that decouples node representation learning from classification for anomaly detection. It mitigates inconsistencies between node behavior patterns and label semantics, and captures intrinsic graph properties in concentrated feature spaces by clustering the entire graph into multiple parts.
- PC-GNN (Pick and Choose Graph Neural Network [50]): A framework designed for imbalanced GNN learning in fraud detection. It uses a label-balanced sampler to select nodes and edges for training, resulting in a balanced label distribution in the induced sub-graph. Furthermore, it employs a learnable parameterized distance function to select neighbors, filtering out redundant links and adding beneficial ones for fraud prediction.
- BernNet [32]: A GNN variant that offers a robust scheme for designing and learning arbitrary graph spectral filters. It uses an order-K Bernstein polynomial approximation to estimate any filter over the normalized Laplacian spectrum of a graph.
- GAT-sep [99]: A GNN designed to enhance learning from graph structures under high heterophily. It combines key designs such as ego- and neighbor-embedding separation, higher-order neighborhoods, and intermediate representation combinations.
- AMNet (Adaptive Multi-frequency Graph Neural Network [13]): A method designed to capture both low-frequency and high-frequency signals through stacking multiple BernNets, and adaptively combine signals of different frequencies.
- BWGNN (Beta Wavelet Graph Neural Network [74]): A method proposed to tackle the 'right-shift' phenomenon of graph anomalies, i.e., the spectral energy distribution concentrates less on low frequencies and more on high frequencies. It employs the Beta kernel to address higher frequency anomalies through multiple flexible, spatial/spectral-localized, and band-pass filters
- GHRN (Graph Heterophily Reduction Network [26]): A method that addresses the heterophily issue in the spectral domain of graph anomaly detection. The approach prune inter-class edges to emphasize and delineate the graph's high-frequency components.
- CARE-GNN (CAmouflage-RESistant GNN) [22]: A GNN-based fraud detector designed for multi-relation graphs, which is equipped with three modules that enhance its performance against camouflaged fraudsters.
- H2-FDetector (Fraud Detector with Homophilic and Heterophilic Interactions) [72] A heterogeneous GNN framework for fraud detection that facilitates the propagation of analogous information through homophilic connections and varied information via heterophilic connection.
B Detailed Description of Datasets in GADBench