SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jianheng Tang $^{1,2*}$ , Fengrui Hua $^{1}$ , Ziqi Gao $^{1,2}$ , Peilin Zhao $^{3}$ , Jia Li $^{1,2\dagger}$
$^{1}$ Hong Kong University of Science and Technology (Guangzhou)
$^{2}$ Hong Kong University of Science and Technology,  $^{3}$ Tencent AI Lab
{jtagbf,zgaoat}@connect.ust.hk, huafengrui@outlook.com,
masonzhao@tencent.com, jialee@ust.hk
With a long history of traditional Graph Anomaly Detection (GAD) algorithms and recently popular Graph Neural Networks (GNNs), it is still not clear (1) how they perform under a standard comprehensive setting, (2) whether GNNs can outperform traditional algorithms such as tree ensembles, and (3) how about their efficiency on large-scale graphs. In response, we introduce GADBench—a benchmark tool dedicated to supervised anomalous node detection in static graphs. GADBench facilitates a detailed comparison across 29 distinct models on ten real-world GAD datasets, encompassing thousands to millions ( $\sim$ 6M) nodes. Our main finding is that tree ensembles with simple neighborhood aggregation can outperform the latest GNNs tailored for the GAD task. We shed light on the current progress of GAD, setting a robust groundwork for subsequent investigations in this domain. GADBench is open-sourced at https://github.com/squareRoot3/GADBench.
Graph Anomaly Detection (GAD) is the process of identifying uncommon graph objects, such as nodes, edges, or substructures, that significantly deviate from the majority of reference objects within a graph database [4, 55]. With a notable history spanning over two decades, GAD has proven its effectiveness across a variety of applications. These include, but are not limited to, the prevention of financial fraud [36] and money-laundering [83], the prediction of network intrusions [57, 64] and device failures [46], the identification of spam reviews [56] and fake news [8]. Unlike outlier detection in tabular data, GAD considers the inter-dependencies among a group of objects, which can often yield additional insights for identifying fraudulent patterns. Meanwhile, GAD also presents unique challenges in terms of modeling efficiency and necessitates strategies to address issues such as label imbalance [50], feature heterophily [26], and relation camouflage [22].
The definitions of GAD can be multifaceted, depending on the specific objectives and applications. In this paper, we focus on the most prevalent GAD scenario—the detection of anomalous nodes within a static attributed graph. Despite the plenty of methods proposed for this task, including traditional structural pattern mining algorithms [57, 4] and advanced deep learning techniques [21, 55], several limitations exist in the current model development and evaluation scheme:
- Absence of a comprehensive supervised GAD benchmark. We summarize the exiting GAD benchmarks and toolboxes in Table 1. The latest benchmark available for GAD, BOND [49], exclusively evaluates unsupervised methods. This overlooks the fact that many GAD models rely on labeled data to boost their performance. In comparison, there have been a variety of supervised
Table 1: Comparison of existing GAD benchmarks in terms of datasets, models, and scenarios.
Benchmark &Toolbox	#Datasets (Organic)	Max. Nodes /Edges	#Models	Model Type	Supervision Scenario
UGFraud [23]	1 (1)	45K/3M	6	GNN	Unsupervised
DGFraud [22]	3 (1)	45K/3M	9	GNN	Supervised
BOND [49]	9 (6)	3M/4M	14	GNN, Classic	Unsupervised
GADBench	10 (10)	5M/73M	29	GNN, Classic, Trees	Fully- and Semi-Supervised
anomaly detection benchmarks for time series [59, 61], images [68], videos [1], and tabular data [31, 77]. Semi-supervised approach is another setting that calls for attention because it can strike a balance between label annotation budgets and model performance.
- Insufficient comparative studies between tree ensembles and GNNs. Tree ensemble methods, including Random Forest [12], XGBoost [16], and Isolation Forest [48], have long been favored in the industry. They also showcase impressive results in a recent benchmark for anomaly detection within tabular datasets [31]. These models are also applicable to GAD datasets [83] with appropriate feature engineering. However, a systematic comparison between tree ensembles and GNNs in the context of GAD is still absent.
- Limited exploration on large-scale graphs. While many GNN models tailored for GAD have shown promising results on small-scale datasets, their efficacy on large graphs remains unexplored. On the other hand, although several large-scale GAD datasets have been proposed [36, 83, 74], they mainly focus on the comparisons of GNN variants, making it unclear how they perform compared with traditional GAD algorithms.
To redress these gaps and foster academia-industry synergy in GAD evaluation, we propose GAD-Bench, which serves as the first comprehensive benchmark for supervised GAD. Our evaluation encompasses 7 non-graph models, 10 standard GNNs, and 10 state-of-the-art GNNs specifically designed for GAD. Recognizing the proven success of tree ensembles in the anomaly detection of tabular data, we additionally employ two tree ensemble models with simple neighborhood aggregations. All models are evaluated on 10 real-world GAD datasets ranging from thousands to millions of nodes. They are tested in both semi-supervised and fully-supervised settings, with and without hyperparameter tuning.
Through extensive experiments, we discover that (1) surprisingly, tree ensembles with neighbor aggregation have the superior performance among all models; (2) most standard GNNs are not suitable for the GAD task; (3) GNNs specifically designed for GAD require hyperparameter tuning to achieve satisfactory performance. We highlight that our findings can offer the research community a clearer insight into the current progress in GAD. In summary, our contributions can be organized into three main aspects:
- We introduce GADBench, the first comprehensive benchmark for supervised anomalous node detection on static attributed graphs. This includes a comparison of 29 well-known GAD methods across a collection of 10 real-world datasets in both semi-supervised and fully-supervised settings.
- To ensure a rigorous and fair comparison, we identify the limitations inherent in the existing evaluation scheme for GAD, instituting enhancements from dataset selection, metric utilization, model training, and hyperparameter tuning.
- We integrate all models, datasets, and evaluation protocols mentioned in this paper into an open-source repository. Users can reproduce the results and evaluate their own datasets/models with minimal effort.
Task definition. We focus on the detection of anomalous nodes within a static attributed graph. Formally, consider a graph  $\mathcal{G} = \{\mathcal{V},\mathbf{A},\mathbf{X}\}$ , where  $\mathcal{V} = \{v_{1},v_{2},\dots ,v_{N}\}$  denotes a set of  $N$  nodes. The non-negative adjacency matrix  $\mathbf{A}\in \mathbb{R}^{N\times N}$  is defined such that  $\mathbf{A}_{ij} > 0$  if and only if there is an edge between  $v_{i}$  and  $v_{j}$ .  $\mathbf{X}\in \mathbb{R}^{N\times d}$  is a feature matrix, of which the  $i$ -th row vector  $\mathbf{x}_i = \mathbf{X}(i,:)$  is the  $d$ -dimensional feature vector of node  $v_{i}$ . Given a subset of labeled nodes  $\mathcal{V}^*\subset \mathcal{V}$ , the task is to classify the remaining nodes into either normal or anomalous categories. In the context of
a heterogeneous graph, multiple adjacency matrices  $\{A_1, A_2, \dots\}$  might exist, each representing different types of relationships among nodes.
Although the GAD task can be regarded as a binary node classification problem, it introduces several additional challenges. Firstly, anomalous nodes typically constitute a small portion of the total nodes, resulting in a significant data imbalance [50, 74]. Secondly, graphs with anomalies often exhibit strong heterophily, where connected nodes possess varying features and labels [27, 26]. This necessitates strategies to handle neighborhood feature disparities during message passing. Lastly, anomalous nodes have a tendency to camouflage their features and connections, blending seamlessly by mimicking the normal patterns in the graph [22, 51]. This demands attention to intentionally manipulated edges and node features. Before presenting our benchmark, we provide a brief overview of both classic methods and deep learning based GAD models. For a more thorough review of GAD models, please refer to these relevant surveys [55, 4, 38].
Classical methods. Classical methods primarily detect graph anomalies by leveraging graph statistical patterns [57, 34], community and clustering structures [14, 43, 86], egonet information [3], spectral analysis [25], random walk-based techniques [81, 29], etc. Other than these heuristic approaches and handcraft feature engineering, another research line leverages learning methods for a more flexible encoding of graph information to identify anomalies. Examples include residual learning [45, 63], relational learning [40, 66], and Bayesian models [33]. Despite their advantages, most methods are primarily designed for plain graphs and struggle to handle attributed graphs. Additionally, they predominantly emphasize unsupervised settings and are generally inefficient in utilizing node labels.
GNNs for GAD. With superb ability to encode both structure and attribute information simultaneously, GNNs have recently gained popularity in mining graph data [76, 39, 85, 30]. To tackle the unique challenges of graph anomalies, such as imbalance, heterophily, and camouflage, several adaptations of standard GNNs have been proposed [20, 96, 24, 53, 52, 21, 79, 54, 90]. On one hand, spatial GNNs have primarily been redesigned at the level of their inherent mechanisms, such as message passing and aggregation [99]. For instance, GAS [44] employs a structure-enhanced pre-processing strategy to establish implicit connections between anomalies. CARE-GNN [22] and GraphConsis [51] combat the camouflage behavior by designing camouflage-resistant message passing and aggregation processes. H2-FDetector [72] introduces a novel information aggregation strategy that enables homophilic connections to propagate similar information, while heterophilic connections disseminate differing information. With the consideration of label imbalance, solutions like PC-GNN [50] and DAGAD [47] use imbalance-aware data sampling and graph augmentation to highlight the importance of anomalies during training. On the other hand, spectral GNNs provide a fresh viewpoint that associates graph anomalies with high frequency spectral distributions [26]. For example, BWGNN [74] applies Beta kernel to manage higher frequency anomalies via flexible and localized band-pass filters. AMNet [13] captures both low-frequency and high-frequency signals, adaptively integrating signals of varying frequencies.
Ensemble models. Tree ensembles, such as Random Forest [12, 9, 6, 5] and XGBoost [16, 78, 97, 58], have shown superior performance in anomaly detection tasks related to tabular data [93, 10, 2, 100]. For example, XGBOD [93], an extension of the XGBoost algorithm specifically designed for anomaly detection, incorporates scores from other models such as Isolation Forest [48] as additional feature and achieves superior performance. Despite their potential, few research has explored the application of tree ensembles in leveraging both structural and feature information simultaneously for GAD. Beyond trees, various base models can be integrated into ensembles for anomaly detection, as suggested by [89]. The use of neighborhood aggregation has been demonstrated to enhance anomaly detection performance, which can be utilized during various stages such as pre-processing [87], model-training [17], and post-processing [88] phases. In this work, we revisit tree ensemble models and demonstrate their potential to outperform GNNs after combining a straightforward structural neighborhood aggregation.
3 The Setup of GADBench
In this section, we present a comprehensive overview of the setup for GADBench. We provide the general selection criteria and considerations for models (Section 3.1), datasets (Section 3.2), and other details (Section 3.3).
Table 2: Categorization of all models used in our evaluation.
Classic Methods	MLP [67], KNN [18], SVM [15], RF [12], XGBoost [16], XGBOD [93], NA [88]
Standard GNNs	GCN [39], SGC [84], GIN [85], GraphSAGE [30], GAT [76], GT [73], PNA [17]BGNN [37], RGCN [70], HGT [35]
Specialized GNNs	GAS [44], DCI [82], PC-GNN [50], GAT-sep [99], BernNet [32], AMNet [13],BWGNN [74], GHRN [26], CARE-GNN [22], H2-FDetector [72]
Tree Ensembles withNeighbor Aggregate	RF-Graph, XGB-Graph
3.1 Benchmark Models
Table 2 provides an overview of the 29 models assessed in GADBench. We briefly introduce each model in the following, and provide a more detailed description in Appendix A.
Classical methods. We select three basic algorithms for supervised classification: Multi-Layer Perceptron (MLP),  $k$ -Nearest Neighbors (KNN), and Support Vector Machine (SVM). Additionally, we incorporate three representative decision tree ensembles, including a bagging-based model, Random Forest (RF); a boosting-based model, Extreme Gradient Boosting Tree (XGBoost); and an advanced XGBoost model named Extreme Boosting Based Outlier Detection (XGBOD). We also assess a recent anomaly detection technique, Neighborhood Averaging (NA), which enhances the outlier scores of existing anomaly detectors by averaging them with the scores of neighboring objects.
Standard GNNs. We evaluate several standard GNNs that have proven to be effective across diverse graph learning tasks, including Graph Convolutional Network (GCN), Chebyshev Spectral Convolution Network (ChebNet), Graph Isomorphism Network (GIN), Graph Sample and Aggregate (GraphSAGE), Graph Attentional Network (GAT), Graph Transformer (GT), Principle Neighbor Aggregation (PNA), Boosted Graph Neural Network (BGNN). We also consider two widely used heterogeneous GNNs, including Relational Graph Convolution Network (RGCN) and Heterogeneous Graph Transformer (HGT).
Specialized GNNs. This group contains GNNs specifically designed for anomaly detection. We evaluate five spatial GNNs including the Graph-based Anti-Spam Model (GAS), Deep Cluster Infomax (DCI), Pick and Choose GNN (PC-GNN), and the GAT with ego- and neighbor-embedding separation (GAT-sep). For spectral GNNs, we evaluate the graph spectral filter via Bernstein Approximation (BernNet), Adaptive Multi-frequency GNN (AMNet), Beta Wavelet GNN (BWGNN), and the Graph Heterophily Reduction Network (GHRN). Additionally, we assess two GNNs optimized for heterogeneous graphs: the CAMouflage-RESistant GNN (CARE-GNN) and the Fraud Detector with Homophilic and Heterophilic Interactions (H2-FDetector).
Tree ensembles with neighbor aggregation. Decision tree ensembles have shown their effectiveness in anomaly detection with tabular data [31], prompting us to adapt them for GAD. To incorporate graph structure information, we follow the idea from a subclass of GNNs that independently manage message passing and node feature transformation [84, 98, 91]. Consequently, we devise tree ensembles with Neighbor aggregation that adhere to the following computational paradigm:
$$
\boldsymbol {h} _ {v _ {i}} ^ {(l)} = \text {A g g r e g a t e} \left\{\boldsymbol {h} _ {v _ {j}} ^ {(l - 1)} \mid v _ {i} \in \text {N e i g h b o r} (v _ {j}) \right\}
$$
$$
\operatorname {S c o r e} \left(v _ {i}\right) = \operatorname {T r e e E n s e m b l e} \left(\left[ h _ {v _ {i}} ^ {0} \right| \left| h _ {v _ {i}} ^ {1} \right| | \dots | \left| h _ {v _ {i}} ^ {L} \right|\right).
$$
In this scheme,  $\pmb{h}_{v_i}^{(0)} = \mathbf{x}_i$  denotes the initial node attributes, and  $\pmb{h}_{v_i}^{(l)}$  represents the feature of node  $v_i$  after  $l$ -layers of neighbor aggregation. Aggregate() can take on any aggregation function such as mean, max, or sum pooling. Same as [84, 98, 91], the aggregation process is parameter-free. TreeEnsemble() can be any tree ensembles that takes the aggregated features as input to predict the anomaly score of node  $v_i$ . In GADBench, we utilize Random Forest and XGBoost to instantiate two new tree ensemble baselines with neighbor aggregation, referred to as RF-Graph and XGB-Graph.
In GADBench, we have gathered 10 diverse and representative datasets, as detailed in table 3, which are chosen based on the following criteria:
Table 3: Statistics of all datasets in GADBench including the number of nodes and edges, the node feature dimension, the ratio of anomalous labels, the training ratio in the fully-supervised setting, the concept of relations, and the type of node features. Misc. indicates the node features are a combination of heterogeneous attributes, possibly including categorical, numerical, and temporal information, More details are shown in Appendix B.
#Nodes	#Edges	#Feat.	Anomaly	Train	Relation Concept	Feature Type
Reddit[42, 49]	10,984	168,016	64	3.3%	40%	Under Same Post	Text Embedding
Weibo[92, 49]	8,405	407,963	400	10.3%	40%	Under Same Hashtag	Text Embedding
Amazon[56, 22]	11,944	4,398,392	25	9.5%	70%	Review Correlation	Misc. Information
YelpChi[66, 22]	45,954	3,846,979	32	14.5%	70%	Reviewer Interaction	Misc. Information
Tolokers[65]	11,758	519,000	10	21.8%	40%	Work Collaboration	Misc. Information
Questions[65]	48,921	153,540	301	3.0%	52%	Question Answering	Text Embedding
T-Finance[74]	39,357	21,222,543	10	4.6%	50%	Transaction Record	Misc. Information
Elliptic[83]	203,769	234,355	166	9.8%	50%	Payment Flow	Misc. Information
DGraph-Fin[36]	3,700,550	4,300,999	17	1.3%	70%	Loan Guarantor	Misc. Information
T-Social[74]	5,781,065	73,105,508	10	3.0%	40%	Social Friendship	Misc. Information
- Organic anomalies. Datasets in GADBench exclusively contain anomalies that naturally emerge in real-world scenarios, a distinction from previous studies that employ synthetic anomalies for GAD evaluations [21, 49]. These earlier works typically inject artificial node attributes and structures into normal graphs like Cora [71], resulting in anomalies that are relatively straightforward to be identified and obviously different from real-world anomalies.
- Various domains. Datasets in GADBench span multiple domains, including social media, e-commerce, e-finance, and crowd-sourcing. As presented in Table 3, the graph edge in each dataset embodies unique relation concepts, which shows a diverse distribution of applications.
- Diverse scale. GADBench datasets cover a wide scale, from thousands to millions of nodes. We have consciously excluded datasets with fewer than 5,000 nodes, such as Bitcoin-Alpha [41], Disney, and Books [69].
- Imbalance ratio. We have ensured that the number and ratio of anomalies within the datasets meet a specific criteria: each dataset contains more than 100 anomalies, to ensure stable experimental results, and no more than a  $25\%$  anomaly ratio, preserving the inherent imbalance nature of GAD. This criterion leads to the exclusion of the Enron dataset [69].
Among the datasets in GADBench, Weibo, Reddit, Questions, and T-Social are designed to identify anomalous accounts on social media platforms. Tolokers, Amazon and YelpChi datasets aim to detect fraudulent workers, reviews and reviewers on crowd-sourcing or e-commerce platforms. T-Finance, Elliptic, and DGraph-Fin concentrate on identifying fraudulent users, illicit entities and overdue loans in financial networks, respectively. For a more comprehensive description of each dataset, please refer to Appendix B.
3.3 Other Details
Data split. We employ both fully-supervised and semi-supervised settings for model evaluation. In a fully-supervised setting, we preserve pre-existing data splits when available. If such divisions are not provided, we follow the approach suggested by [74], randomly partitioning nodes into three subsets:  $40\%$  for training,  $20\%$  for validation, and the remaining  $40\%$  for testing. For each dataset, the specific training ratio is reported in table 3. The semi-supervised setting typically involves a smaller training ratio, e.g.,  $1\%$  or  $5\%$  in previous studies [22, 74]. However, due to the variance in graph sizes present in GADBench, a fixed training ratio might lead to substantial discrepancies in the scale of training sets. To more accurately mimic real-world semi-supervised scenarios, we standardize the training set across all datasets to include a total of 100 labels—20 positive labels (anomalous nodes) and 80 negative labels (normal nodes). To ensure robustness in our findings, we execute ten random splits on each dataset and analyze the average performance of the model.
Metrics. According to existing anomaly detection benchmarks [31, 49], we select Area Under the Receiver Operating Characteristic Curve (AUROC), Area Under the Precision Recall Curve (AUPRC) calculated by average precision, and the Recall score within top-  $k$  predictions  $(\mathbf{Rec}@\mathbf{K})$  as performance metrics for the GAD task. We set  $k$  as the number of anomalies within the test set. For all metrics, anomalies are considered as the positive class, and higher scores indicate better model
performance. Among these metrics, AUROC primarily focuses on overall performance and is not sensitive to top- $k$  predictions, Rec@K only cares top- $k$  performance, and AUPRC strikes a balance between the two. Suppose the test set includes 10 anomalies within 1000 data points and a model ranks them from positions 11th to 20th, it would attain an AUROC of 0.99, an AUPRC of 0.33, and a Rec@10 of 0. We also document the running time and memory consumption of each model.
Hyperparameter Optimization. To control the effect of hyperparameter selection and ensure fairness [11], we standardize the evaluation process with and without hyperparameter tuning. Initially, we employ default hyperparameters as stated in the original papers. To ensure fairness in hyperparameter tuning, we then utilize random search [7] to optimize hyperparameters. During one trial on each dataset, we randomly select a set of hyperparameters from the predefined search space for each model. For more information about metrics, default hyperparameters, search spaces, and other implementation details, please refer to Appendix C.
In this section, we study the experimental results of all the benchmarked models. We first provide a comprehensive comparison of all models, taking into account both default and optimally tuned hyperparameters. Following that, we aim to conduct an in-depth comparison between tree ensembles with neighbor aggregation and GNN-based methods.
4.1 Overall Comparison
In Figure 1, we present an overview of model performance across 10 datasets for all metrics, excluding four GNNs that are specific to heterogeneous graphs. In Table 4, we take a close look at the model performance regarding the AUPRC score after hyper-parameter tuning on each dataset. For comprehensive experimental results, please refer to Appendix D. Our key findings include: