GADBench: Revisiting and Benchmarking Supervised Graph Anomaly Detection
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Graph Anomaly Detection
Graph Neural Networks
TL;DR: We introduce a comprehensive benchmark for supervised anomalous node detection on static attributed graphs, covering 29 distinct models and 10 real-world datasets.

Abstract: With a long history of traditional Graph Anomaly Detection (GAD) algorithms and recently popular Graph Neural Networks (GNNs), it is still not clear (1) how they perform under a standard comprehensive setting, (2) whether GNNs can outperform traditional algorithms such as tree ensembles, and (3) how about their efficiency on large-scale graphs. In response, we introduce GADBench---a benchmark tool dedicated to supervised anomalous node detection in static graphs. GADBench facilitates a detailed comparison across 29 distinct models on ten real-world GAD datasets, encompassing thousands to millions (~6M) nodes. Our main finding is that tree ensembles with simple neighborhood aggregation can outperform the latest GNNs tailored for the GAD task. We shed light on the current progress of GAD, setting a robust groundwork for subsequent investigations in this domain. GADBench is open-sourced at https://github.com/squareRoot3/GADBench.

GADBench: Revisiting and Benchmarking Supervised Graph Anomaly Detection
Jianheng Tangl Fengrui Hua', Ziqi !22 Peilin Zhao Jia Li Hong Kong University of Science and Technology (Guangzhou) Hong Kong University of Science and Technology. Tencent AI Lab (jtangbf, zgaoat (ecaoaat)@conneet..oo hk, nuafengrui@out com, masonzhao@tennentt.com com, jialee@ust.he hk
Abstract
With a long history of traditional Graph Anomaly Detection (GAD) algorithms and recently popular Graph Neural Networks (GNNs), is still not clear (1) how they perform under standard comprehensive setting, (2) whether GNNs can outperform traditional algorithms such tree ensembles, and (3) how about their efficiency on large-scale graphs. In response, we introduce GADBench-a benchmark tool dedicated to supervised anomalous node detection static graphs. GADBench facilitates a detailed comparison across 29 distinct models on ten real-world GAD datasets, encompassing thousands millions (~6M) nodes. Our main finding is that tree ensembles with simple neighborhood aggregation can outperform the latest GNNs tailored for the GAD task. We shed light on the current progress of GAD, setting a robust groundwork for subsequent investigations in this domain. GADBench is open-sourced at https: thub. cww/ ccmmm/uuurreooot3///ABennnc
Introduction
Graph Anomaly Detection (GAD) is the process of identifying uncommon graph objects, such as nodes, edges, substructures, that significantly deviate from the majority of reference objects within a graph database [4] (55]. With a notable history spanning over two decades, GAD has proven its
limited to, the prevention of financial fraud (36 and money-laundering [83, the prediction of network intrusions [57 64]
effectiveness across a variety of applications. These include, but
and device failures 46). the identification of spam reviews (56) and fake news (8). Unlike outlier detection in tabular data, GAD considers the inter-dependencies among a group of objects, which can often yield additional insights for identifying fraudulent patterns. Meanwhile, GAD also presents unique challenges in terms of modeling efficiency and necessitates strategies to address issues such
as label imbalance [50], feature heterophily 26 and relation camouflage
The definitions GAD can be multifaceted, depending on the specific objectives and applications. In this paper, we focus the most prevalent GAD scenario the detection of anomalous nodes within a static attributed graph. Despite the plenty of methods proposed for this task, including traditional structural pattern mining algorithms 55 and advanced deep learning techniques 21 [55) several
limitations exist in the current model development and evaluation scheme:
A bsence of comprehensive supervised GAD benchmark. We summarize the exiting GAD benchmarks and toolboxes in Table [ The latest benchmark available for GAD, BOND [49]. exclusively evaluates unsupervised methods. This overlooks the fact that many GAD models rely on labeled data to boost their performance. In comparison, there have been variety of supervised
WWork done during an internship at Tencent Al Lab.
1 "oorreppnniing Author.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Table 1: Comparison of existing GAD benchmarks in terms of datasets, models, and scenarios.
Benchmark
#Datasets (Organic)
Max. Nodes
#Models
Model Type
Supervision Scenario
& <Toolbox
/Edges
UGFraud
23
1 (1)
4SK/3M
6
GNN
Unsupervised Supervised Unsupervised
DGFraud
22
(1)
4SK/3M
9
GNN
BOND
49
(6)
3M/MM
14
GNN, Classic
GADBench
(10)
5M/73M
29
GNN, Classic, Trees
Fully- and Semi-Supervised
anomaly detection benchmarks for time series 59 61 images [68], videos [1 and tabular data
Semi-supervised approach is another setting that calls for attention because can strike balance between label annotation budgets and model performance.
31
Insufficient comparative studies between tree ensembles and GNNs. Tree ensemble methods,
including Random Forest [2]. XGBoost [16], and Isolation Forest 488] have long been favored in the industry. They also showcase impressive results in a recent benchmark for anomaly detection within tabular datasets 33.. These models are also applicable to GAD datasets [83] with appropri-
ate feature engineering. However, systematic comparison between tree ensembles and GNNs in the context of GAD is still absent.
Limited exploration on large-scale graphs. While many GNN models tailored for GAD have shown promising results small-scale datasets, their efficacy large graphs remains unexplored.
On the other hand, although several large-scale GAD datasets have been proposed [36 83 74), they mainly focus on the comparisons of GNN variants, making unclear how they perform compared with traditional GAD algorithms.
To redress these gaps and foster academia-industty synergy in GAD evaluation, we propose GAD- Bench, which serves as the first comprehensive benchmark for supervised GAD. Our evaluation encompasses 7 non-graph models, 10 standard GNNs, and 10 state-of-the-ard GNNs specifically designed for GAD. Recognizing the proven success of tree ensembles in the anomaly detection of tabular data, we additionally employ two tree ensemble models with simple neighborhood aggrega- tions. All models are evaluated on real-world GAD datasets ranging from thousands to millions of nodes. They are tested in both semi-supervised and fully-supervised settings, with and without hyperparameter tuning.
Through extensive experiments, we discover that (1) surprisingly. tree ensembles with neighbor aggregation have the superior performance among all models; (2) most standard GNNs are not suitable for the GAD task; (3) GNNs specifically designed for GAD require hyperparameter tuning achieve satisfactory performance. We highlight that our findings can offer the research community a clearer insight into the current progress in GAD. In summary, our contributions can be organized into three main aspects:
We introduce GADBench, the first comprehensive benchmark for supervised anomalous node detection static attributed graphs. This includes comparison of 29 well-known GAD methods across a collection of 10 real-world datasets in both semi-supervised and sully-supervieed settings.
To ensure rigorous and fair comparison, we identify the limitations inherent in the existing evaluation scheme for GAD, instituting enhancements from dataset selection, metric utilization, model training, and hyperparameter tuning.
We integrate all models, datasets, and evaluation protocols mentioned in this paper into open- source repository. Users can reproduce the results and evaluate their own datasets/models with minimal effort.
2 Preliminaries and Related Work
Task definition. We focus on the detection of anomalous nodes within a static attributed graph.
Formally, consider a graph = (V, A, X). where V = {", u.. Enr denotes set of N nodes. The non-negative adjacency matrix A E RNNN is defined such that A.j > 0 if and only if there is an edge between and vy. X E R is feature matrix, of which the i-th row vector the d-dimensional feature vector of node v;- Given subset of labeled nodes 1 < V. the task is to classify the remaining nodes into either normal or anomalous categories. In the context of
X (i.:)

---

a heterogeneous graph, multiple adjacency matrices AA1 might exist, each representing different types of relationships among nodes.
Although the GAD task can be regarded as a binary node classification problem, it introduces several additional challenges. Firstly, anomalous nodes typically constitute a small portion of the total nodes, resulting in significant data imbalance [50, 74). Secondly, graphs with anomalies often exhibit strong heterophily, where connected nodes possess varying features and labels [27 26). This necessitates strategies to handle neighborhood feature disparities during message passing. Lastly, anomalous nodes have tendency to camouflage their features and connections, blending seamlessly by mimicking the normal patterns in the graph [22] [ST]. This demands attention to intentionally manipulated edges and node features. Before presenting our benchmark, we provide brief overview of both classic methods and deep learning based GAD models. For more thorough review of GAD
models, please refer to these relevant surveys 555
Classical methods. Classical methods primarily detect graph anomalies by leveraging graph statistical patterns [57 (44, community and clustering structures [14 43 (86), egonet information [3], spectral analysis 225 random walk-based techniques [8I etc. Other than these heuristic approaches and handeraft feature engineering, another research line leverages learning methods for more flexible encoding of graph information to identify anomalies. Examples include residual learning [45] 63),
relational learning 40 66]. and Bayesian models [3]. Despite their advantages, most methods are primarily designed for plain graphs and struggle to handle attributed graphs. Additionally, they predominantly emphasize unsupervised settings and are generally inefficient in utilizing node labels.
GNNs for GAD. With superb ability to encode hoth structure and attribute information simultaneously, GNNs have recently gained popularity in mining graph data [76] 39]
85
30
To tackle the unique
challenges of graph anomalies, such as imbalance, heterophily, and camouflage, several adaptations 90). On one hand, spatial GNNs have primarily been redesigned at the level of their inherent mechanisms, such as message passing and aggregation [99]. For instance, GAS 4 employs structure-enhanced pre-processing strategy to establish implicit connections between anomalies. CARE-GNT GNN [22) and GraphConsis [5]] combat the camouflage behavior by designing sumouflage-resisaan message passing and aggregation processes. H2-FDetector (72) introduces novel information aggregation strategy that enables homophilic connections to propagate similar information, while heterophilic connections disseminate differing information. With the consideration of label imbalance, solutions like PC-GNN (50 and DAGAD 47 use imbalance-awan data sampling and graph augmentation to highlight the importance of anomalies during training. On the other hand, spectral GNNs provide a fresh viewpoint that associates graph anomalies with high frequency spectral distributions (26). For example, BWGNN 74 applies Beta kernel to manage higher frequency anomalies via flexible and localized band-pass filters. AMNet [13] captures both low-frequency and high-frequency signals, adaptively integrating signals of varying frequencies.
of standard GNNs have been proposed (20 96 24
53 52 21
54
Ensemble models. Tree ensembles, such as Random Forest 12,
and XGBoost
16
78
58
have shown superior performance in anomaly detection tasks related to tabular data 93
10
2
100)
For example, XGBOD (33, an extension of the XGBoost algorithm specifically designed for anomaly
detection, incorporates scores from other models such as Isolation Forest [8] as additional feature and achieves superior performance. Despite their potential, few research has explored the application of tree ensembles in leveraging both structural and feature information simultaneously for GAD. Beyond trees, various base models can be integrated into ensembles for anomaly detection, as suggested by (89) The use of neighborhood aggregation has been demonstrated to enhance anomaly detection performance, which can be utilized during various stages such as pre-pre pre-processing [87]. model-training [17], and post-processing (88) phases. In this work, we revisit tree ensemble models and demonstrate their potential to outperform GNNs after combining a straightforward structural neighborhood aggregation.
3 The Setup of GADBench
In this section, we present a comprehensive overview of the setup for GADBench. We provide the general selection criteria and considerations for models (Section 3. datasets (Section 3.2). and other details (Section 3.3
3

---

Table 2: Categorization of all models used in our evaluation.
Classic Methods
ML. 67
KNN
18
SVM
15),
RF
12. XGRoost
[66. XGBOD
93
NA
88
GCN 39
SGC
[84]
GIN
885 GraphSAGE
30
GAT 76.
GT
73
PNA
Standard GNNs
BGNN 37 RGCN 770, HGT
35
GAS 44 DCI [22, PC-GNN 50
GAT sep
99 BemNet 33. AMNet 13
Specialized GNNs
BWGNN 774,, GHRN (6), CARE-GNN 22
H2-FDelector
72
Tree Ensembles with Neighbor Aggregate
Gr-Grpph, XGB-
3.1 Benchmark Models
Table provides an overview of the 29 models assessed in GADBench. We briefly introduce each model the following, and provide more detailed description in Appendix [A]
Classical methods. We select three basic algorithms for supervised classification: Multi-Layer Perceptron (MIP), k-Nearest Neighbors (KNN), and Support Vector Machine (SVM). Additionally, we incorporate three representative decision tree ensembles, including bagging- model, Random Forest (RF); boosting-based model, Extreme Gradient Boosting Tree (XGBoost); and an advanced XGBoost model named Extreme Boosting Based Outlier Detection (XGBOD). We also assess a recent anomaly detection technique, Neighborhood Averaging (NA), which enhances the outlier scores existing anomaly detectors by averaging them with the scores of neighboring objects.
Standard GNNs. We evaluate several standard GNNs that have proven to be effective across diverse graph learning tasks, including Graph Convolutional Network (GCN), Chebyshev Spectral Convolution Network (ChebNet). Graph Isomorphism Network (GIN), Graph Sample and Aggregate (GraphSAGE)), Graph Attentional Network (GAT), Graph Transformer (GT), Principle Neighbor Aggregation (PNA), Boosted Graph Neural Network (BGNN). We also consider two widely used heterogeneous GNNs, including Relational Graph Convolution Network (RGCN) and Heterogeneous Graph Transformer (HGT).
Specialized GNNs.
This group contains GNNs specifically designed for anomaly detection. We evaluate five spatial GNNs including the Graph-based Anti-Spam Model (GAS), Deep Cluster In- fomax (DCI), Pick and Choose GNN (PC-GNN), and the GAT with ego- and nnighbor-embeddiing separation (GAT-sep). For spectral GNNs, we evaluate the graph spectral filter via Bernstein Approx- imation (BemNeet), Adaptive Aulti-frequency GNN (AMNet), Beta Wavelet GNN (BWGNN), and the Graph Heterophily Reduction Network (GHRN). Additionally, we assess two GNNs optimized for heterogeneous graphs: the CAAmoufage-RRsiistee Esistant GNN (CARE-GNN) and the Fraud Detector with Homophilic and Heterophilic Interactions (H2-FDetector).
Tree ensembles with neighbor aggregation. Decision tree ensembles have shown their effectiveness in anomaly detection with tabular data (3D. prompting us to adapt them for GAD. To incorporate graph structure information, we follow the idea from a subclass of GNNs that independently man- age message passing and node feature transformation [4 98 91]. Consequently, we devise tree ensembles with Neighbor aggregation that adhere to the following computational paradigm:
h Aggregate ( (h E Neighbor(v;))
Score(v;) = TreeEnsemble (h., ||4 -.........1.. -.,,,,)),,,,
In this scheme, 1 = X; denotes the initial node attributes, and (.(()) represents the feature of node i after (-layers of neighbor aggregation. Aggregate(-) can take any aggregation function such as mean, max, or sum pooling. Same as [84, 98 9I], the aggregation process is parameter-fre TreeFnnemblee() can be any tree ensembles that takes the aggregated features as input to predict the anomaly score of node n. In GADBench, we utilize Random Forest and XGBoost to instantiate two new tree ensemble baselines with neighbor aggregation, referred as RF-Graph and XGB-Graph.
3.2 Benchmark Datasets
In GADBench, we have gathered 10 diverse and representative datasets, as detailed in table which are chosen based on the following criteria:

---

Table 3: Statistics of all datasets in GADBench including the number of nodes and edges, the node feature dimension, the ratio of anomalous labels, the training ratio in the fully-supervised setting, the concept of relations, and the type of node features. Misc. indicates the node features are combination of heterogeneous attributes, possibly including categorical, numerical, and temporal
information, More details are shown in Appendix B
#Nodes
#Edges
#Feat.
Anomaly
Train
Relation Coneept
Feature Type
Reddit 49
10,984
168.016
64
3.3%
40%
Under Same Post
Text Embedding
Weibo
8,405
407.963
400
10.3%
40%
Under Same Hashtag
Text Embedding Misc. Information
Amazon 55 21
11,944
4,398.392
25
9.5%
70%
Review Correlation
YelpChi 66 22
45,954
3,846,979
32
14.5%
70%
Reviewer Interaction
Misc. Information
Tolokers[65
11,758
$19,000
10
21.8%
40%
Work Collaboration
Misc. Information
Questions 65
48,921
153,540
301
3.0%
52%
Question Answering
Text Embedding
T-Fimance 74
39,357
21,222,543
10
4.6%
50%
Transaction Record
Misc. Information
Elliptic (83
203,769
234,355
166
9.8%
50%
Payment Flow
Misc. Information
DGraph-Find 36
3,700,550
4,300,999
17
1.3%
70%
Loan Guarantor
Misc. Information
T-Social (74
5,781,065
73,105,508
10
3.0%
40%
Social Friendship
Misc. Information
Organic anomalies. Datasets in GADBench exclusively contain anomalies that naturally emerge in real-world scenarios, distinction from previous studies that employ synthetic anomalies for GAD evaluations 49]. These earlier works typically inject artificial node attributes and structures into normal graphs like Cora 71. resulting in anomalies that are relatively straightforward be identified and obviously different from real-world anomalies.
Various domains. Datasets in GADBench span multiple domains, including social media, commerce, e- f-finance, and crowd-sourcing. As presented in Table 3 the graph edge in each dataset embodies unique relation concepts, which shows a diverse distribution of applications.
Diverse scale. GADBench datasets cover a wide scale, from thousands to millions of nodes. We have consciously excluded datasets with fewer than 5,000 nodes, such as Bitcoin-Alpha (41) Disney, and Books [69].
Imbalance ratio. We have ensured that the number and ratio anomalies within the datasets meet specific criteria: each dataset contains more than 100 anomalies, to ensure stable experimental results, and no more than 25% anomaly ratio, preserving the inherent imbalance nature of GAD. This criterion leads to the exclusion of the Enron dataset (69
Among the datasets in GADBench, Weibo, Reddit, Questions, and T-Social are designed to identify anomalous accounts on social media platforms. Tolokers, Amazon and YelpChi datasets aim to detect fraudulent workers, reviews and reviewers on crowd-sourcing or e-commerce platforms. T-Finance, Elliptic, and DGraph-Fin concentrate on identifying fraudulent users, illicit entities and overdue loans in financial networks, respectively. For more comprehensive description of each dataset, please refer to Appendix B
3.3 Other Details
Data split. We employ both fully-supervised and cemi-supervised settings for model evaluation. In fully-supervised setting, we preserve pre-existing data splits when available. If such divisions are not provided, we follow the approach suggested by (74,, randomly partitioning nodes into three subsets: 40% for training, 20% for validation, and the remaining 40% for testing. For each dataset, the specific training ratio is reported table 3 The cemi-supervised setting typically involves a smaller training ratio, e.g., 1% or 5% in previous studies [22 74). However, due to the variance in graph sizes present in GADBench, fixed training ratio might lead to substantial discrepancies
the scale of training sets. To more accurately mimic real-world semi-supervised scenarios, we standardize the training set across all datasets to include a total of 100 labels- 20 positive labels (anomalous nodes) and 80 negative labels (normal nodes). To ensure robustness in findings, we execute ten random splits on each dataset and analyze the average performance of the model.
we select Area Under the Receiver Operating Characteristic Curve (AUROC), Area Under the Prevision Recall Curve (AUPRC) calculated by average precision, and the Recall score within top-k predictions (Ree@ K) as performance metries for the GAD task. We set & as the number of anomalies within the test set. For all metrics, anomalies are considered as the positive class, and higher scores indicate better model
Metrics. According to existing anomaly detection benchmarks 31 49
5

---

performance. Among these metrics, AUROC primarily focuses on overall performance and is not sensitive to top-k predictions, Rec@ K only cares top-k performance, and AUPRC strikes a balance between the two. Suppose the test set includes 10 anomalies within 1000 data points and model ranks them from positions 11th to 20th, would attain an AUROC of 0.99, an AUPRC of 0.33, and a Rec@@ @ 10 of O. We also document the running time and memory consumption of each model.
Hyperparameter Optimization. To control the effect heperarrameeer selection and ensure fairness [. we standardize the evaluation process with and without hyperparameter tuning. Initially, we employ default hyperparameters as stated in the original papers. To ensure faimess in hyperparameter tuning, we then utilize random search [7] to optimize hyperparameters. During one trial on each dataset, we randomly select a set of hyperparameters from the predefined search space for each model. For more information about metrics, default hyperparameters, search spaces, and other implementation details, please refer to Appendix
4 Experimental Results
In this section, we study the experimental results of all the benchmarked models. We first provide a comprehensive comparison of all models, taking into account both default and optimally tuned hyperparameters. Following that, we aim conduct an in-depth comparison between tree ensembles with neighbor aggregation and GNN-based methods.
4.1 Overall Comparison
In Figure I we present an overview of model performance across 10 datasets for all metries, excluding four GNNs that are specific to heterogeneous graphs. In Table we take close look at the model performance regarding the AUPRC score after hyper-parameter tuning on each dataset. For comprehensive experimental results, please refer to Appendix D Our key findings include:
Ensemble trees with neighbor aggregation have superior performance. As highlighted in Figure I1 XGB-Graph and RF-Graph consistently surpass other compared models across all metrics using default hyperparameters. The performance gap becomes particularly significant the fully-supervised setting, i.c., XGB- surpasses BWGNN- the best GNN model in this setting by an absolute average improvement of 2.0% on AUROC, 12.9% on AUPRC, and 9.8% on Ree @ K. In the semi- supervised context, RF-Graph presents an absolute average improvement of 2.8% on AUROC, 8.0% on AUPRC, and 3. % on Rec@K, as compared GHRN, the best GNN model in this setting. is important to highlight that the improvement in AUPRC and Rec@K K is more pronounced than that AUROC due to the imbalanced issue, suggesting that RF-Graph and XGB-Graph are more proficient in predicting top-k high-confidence anomalies. Further, as shown in the bottom of Figure T. tree ensembles with neighbor aggregation not only outperform GNNs in terms of efficiency but also exhibit lower memory consumption. Although the performance gap of different models narrow after hyperparameter tuning as in Table 4 RF-Graph and XGB- Graph still prevail among 6 out of 10 datasets. Accordingly, our observations show the superior effeectivnness and efficiency of RF-Graph and KGB-Graph across diverse GAD datasets and scenarios.
Most standard GNNs prove unsuitable for GAD. As shown in Figure I it becomes clear that the majority of standard GNNs encounter difficulties when dealing with GAD tasks. To illustrate,
the performance of GCN and GIN is on par with that of MLP-a method that does not take graph structure information into account. This indicates that standard GNNs often struggle to effectively handle structure camouflage or feature heterophily problems induced by anomalies. In Table 4 while hyperparameter tuning does improve the results of all standard GNNs, they remain subpar compared to methods in other categories. An exception is GraphSAGE, which displays an average absolute improvement of 10.4% on average AUPRC when optimal hyperparameters are used, making it competitive with specialized GNNs. BGNN, while impressive on specific datasets such as T- Social, exhibits poor performance on other datasets. This inconsistency may result from the inherent instability of its joint training scheme, especially when compared to the more stable two-step approach in XGB-Graph and RF-Graph.
Specialized GNNs require hyperparameter tuning to achieve satisfactory performance. Gen- erally, specialized GNNs outperform standard GNNs in Figure T indicating that GNNs tailored for GAD indeed enhance anomaly detection capabilities. However, the performance of these

---

Semi Supervised
AUROC
088
MLP
STM
RF
XGRIO
OEN
stC
6N
0-340E
PMA
Bot
645
Poot
O4Feep
Bootir
140
awotty
GHRS
RE-Cup
XGlk Cappl
AUPRC
0.0
MLP
ERN
HP
XOROD
S0C
G-M0E
G4T
PIA
BOWN
043
Ac]
RoW
Hemiie
AMii
#WOWN
GIRN
RF-Grsp
X00 Kap--Gaph
0.8
Rec@
84
MLP
ENW
SWM
XOBleal
XORiD
M
OEN
80c
GB
03AGE
G47
67
PNA
Brews
GA5
Rc7
PCDRS
OATww
Aentir
GIRS
RP-domi
XGI--ool Campi
Pally-Sopervised
AUROS
0.8
KNN
SYM
RF
XGBoog
KGRop
84
GCY
CN
GS40E
G47
G7
PM
BGWN
Ddj
REGN
64789
Remile
AMMi
RNGUN
GHRN
Xapp-Gopp
0.0
M.P
SYM
Ar
XüRlest
SOROD
OCN
GN
6-340E
GA7
07
PASA
Brew
GA5
nc1
PCORN
G475p
AMSi
Bwwww
RF-Gra
XGB Gompl
0.8
Recigk
94
MUP
BYM
8F
XGBoao
XGRO0
GEN
50c
GNN
OsSGGE
PNA
AGNN
045
(478p
Beoki
AWOKN
GITN
Xtal Geeph
105.
Memory (GB)
t. G
Time
ME2
NA
GON
SGC
GIN
G-AGG
GAYI
GI
POGNN
GATER
BemMel
AMMC
Be coeek
NCB Glup
KAN
STM
XCiB-on
XXBOD
NA
GON
SGC
GIN
G-AA
QI
BONN
CAS
POCKN
DATES
AMMO
GOURN
X6B
Figure 1: Comparison of the anomaly detection performance, wall-clock time (on all datasets), and peak CPU/GPU memory utilization (on DGraph-Fin among all models with default hyperparameters. Top three lines are in semi-supervised settings and the others are in fully-supervised settings, The color of the box plot represents the average score for each metrie, while the central line within the box indicates the median score.
specialized GNNs strongly depends on hyperparameter tuning. As indicated in the last column of Table all these methods witness performance improvement after hyperparameter search, For instance, when optimized hyperparameters are employed, BWGNN can surpass RF-Graph and XGB-Graph on the Reddit dataset. This demonstrates that under certain conditions, some specialized GNNs can deliver commendable performance. However, as illustrated the bottom of Figure 1, these GNNs often demand more training time and memory. The inherent limitations of hyperparameter
7

---

Table 4: Comparison of the AUPRC score of each model with optimal hyperparameters through random search. Best results are highlighted in bold. In the last two columns, Ave. signifies the average score across the first 9 datasets without T-Social, while Imp. denotes the absolute increase in this average score when compared to the default hyperparameters. OOT means the model could not
complete training within day. Results for other metrics can be found in Appendix D and Table 13
Model
Reddit
Weibo
Amazon
Yelp
I-Fin.
Ellip.
Tolo.
Quest.
DGraph.
T-Social
Ave.
Imp.
MLP
5.91
84.88
87.34
47.68
74.21
43.77
38.29
15.34
2.69
9.69
44.46
2.67
KNN
6.12
81.12
84.41
54.39
74.97
60.98
35.30
15.37
1.67
36.32
46.04
9.13
SVM
6,88
84,91
85.80
41.01
78.10
20.98
37.90
15.37
2.65
0OT
41.51
4.53
RF
4.63
93.52
91.18
77.77
81.99
78.42
38.6d 64
14.37
2.57
41.56
53.68
0.81
XGBoost
5.56
94.49
91.88
84.00
82.64
76.93
40.05
16.24
2.75
16.60
$4.95
0.73
XGBOD
8.27
95,70
92.15
79.46
82.32
74.86
40.65
16.08
1.95
OOT
54.61
1.62
NA
9.70
94.09
91,56
63.93
88.78
29.14
51.06
14.32
4.13
79.21
49.64
3,38
GCN
4.63
94.64
45.65
20.88
78.22
25.37
40.57
14.06
3.80
76.35
36.42
1.54
SGC
6.04
91.16
42.69
19.87
68.68
17.82
39.59
10.53
2.49
16.28
33.21
5.66
GIN
6.41
91.67
84.61
33.63
78.35
26.21
40.36
60.79
42.04
2.57
GraphSAGE
5.56
94.02
82.45
46.64
84.71
57.82
51.41
17.50
3.77
75.32
49.32
10.44
GAT
7.20
92.91
87.94
43.62
82.72
27.53
45.25
15.51
3.85
32.07
45.17
2.80
GT
7.68
89.85
84,90
44.60
83.14
25.90
45.71
17.08
3.83
36.14
44.74
5.42
PNA
7.75
96.04
35.24
29.95
76.67
27.81
41.74
11.38
21.24
64
BGNN
6.87
95.99
67.92
29.71
83.72
62.03
45.35
9.43
4.24
99.09
45.03
0.95
GAS
4.43
96.76
81.43
35.11
85,95
29.80
47.21
15.48
3.65
62.36
44.42
6.62
DCI
7.74
91.77
85.17
39
63.68
27.39
37.73
14.59
3.31
12.97
41.25
1.01
PCGNN
7.73
89.07
89.33
44.51
83.31
42.66
44.85
15.59
3.42
80.29
46.72
4.69
BernNet
7.82
92.38
84.89
51.92
89.17
38.25
43.69
17.25
3.27
44.30
47.63
2.90
AMNet
7.87
94.99
88,36
46.86
88.87
25.18
40.74
15.63
2.81
37.70
45.70
2.49
GAT-sep
7.19
93.40
84.72
45.49
84.01
26.35
46.66
17.90
3.84
33.39
45.50
2.98
BWGNN
8.32
94.01
91.48
61.53
89.38
29.31
49.58
18.57
3.97
78.93
49.57
2.12
GHRN
4.66
95.27
89.52
55.42
87.60
43.90
47.45
18.31
86.78
49.55
1.77
RF-Graph
5.13
96.95
90.53
83.92
89.23
78.86
52.34
14.44
2.15
97.63
57.06
1.21
XGB-Graph
5.29
97.06
93.33
91.11
90.12
77.78
53.92
18.19
3.79
97.34
58.95
1.34
search also pose significant challenges, especially in real-world applications where there might be scarcity of annotated labels or computational resources. Given these constraints, ensembles with neighborhood aggregation might still be the preferred choice.
NA is versatile technique adaptable to any model in GADBench. We apply it to XGBoost and observe a remarkable enhancement in the semi-supervised setting, where the average AUPRC across 10 datasets increases from 37.5% to 38.9%. However, the boost is not significant in the fully-supervised setting. These findings highlight NA's potential as an effective strategy to address challenges associated with label scarcity. For additional results related to the application of NA other models, please refer to Appendix DD.
Finally, observe that all methods perform poorly on the DGraph-Fin dataset. This can be attributed to the highly imbalanced and sparse graph structure, with an average degree of 1.16. Furthermore, nearly all anomalous nodes
we find that node features in this dataset are highly indistinguishable, share identical features with normal nodes. Indeed, the AUROC scores of all models on this dataset align with those reported in the original paper, demonstrated in Appendix
4.2 Specialized Experiments Heterogeneous and Inductive Settings
Table 5: Comparison of heterogeneous and homogeneous GAD methods Amazon and Yelp datasets under both fully-supervised and semi-supervised settings. Results are averaged over 10 runs.
Amazon (Semi Supervised)
Amazon (Fully. Supervised)
Yelp (Semi Supervised)
Yelp hully-Supervieed)
Model
AUROC:
AUPRC
Rec
AUROK
AUPRC
ReciKK
AUROC
AUPRC
Rec
AUROC
AUPRC
Rec
BWGNN
91.83
$1.68
77.71
97.95
89.09
85.00
64.30
23.66
26.44
84.89
55,06
52.18
RGCN
84.17
41.07
45.57
92.03
67.97
65.49
72.20
26.46
28.54
78.34
34,57
34.98
28.19
CARE-GNN
86.00
58.95
59.12
90.84
72.64
67.72
91.19
68.69
63.35
95.23
81.06
74.85
H2Detootor
71.00
29.27
32.61
66
39.95
44.35
67.28
22.23
24.08
89.07
59.40
57.54
Grah-raphh
84.38
78.17
98.69
92.61
85.87
64.03
24.84
26.81
96.22
87.03
78.76
8

---

Table 6: Performance comparison on the Elliptic and DGraph-Fin datasets under inductive and transductive settings, with all results being averaged over 10 runs.
Elliptic (Inductive)
Elliptic (Transductive)
IXGraphb-Fi (Iaductive)
DGGraph-Fin (Transductive)
Model
AUROC
AUPRC
Rec@K
AUROC
AUPRC
Rec@K
AUROC
AUPRC
Ree@K
AUROC
AUPRC
Ree@K
GCN
75.79
14.97
16.73
92.40
73.57
69.99
73.99
3.35
5.61
75.85
3.98
7.05
GrupSAAE
79.51
20.59
3.06
BWGNN
82.29
22.49
28.24
96.12
86.58
81.14
73.85
3.24
5.83
76.26
4.01
7.52
GHRN 84.74
25:42
28.54
96.05
86.57
81.11
76.20
4.03
7.48
76.14
3.99
7.54
XGB-Graph
90.36
76.20
70.64
96.80
71.23
74.64
Dealing with heterogeneous graphs. We evaluated the performance of four GAD methods that consider heterogemeity- RGCN, HGT, CARE-GNI and H2-FDettetto--ure datasets, each comprising three different types of edges as detailed in Appendix B For comparison, also tested three other equivalently. As shown in table 5 considering heterogeneity does not enhance performance on the Amazon dataset, but it leads to improvements the Yelp dataset. Specifically, CARE-GNN outperforms all other methods under label-scarce conditions.
the Amazon and Yelp
BWGNN, and XGB-Graph- that treat all edge types
Performance in the inductive setting. Our primary experiments focus the transductive setting, characterized by the assumption that all nodes are visible in the training process. To offer a holistic evaluation, we also conduct experiments in the inductive setting using DGraph-Fin and Elliptic datasets which have temporal features. In this setting, features and structures associated with test nodes not accessible during the training phase. As presented in table 6 the model performance is generally impacted in the inductive setting of two datasets. Specifically, XGB- XGB-Graph outperforms other models across all metries on Elliptic, while GHRN stands out the most robust model on DGraph-Fin.
The impact of different number of neighbor aggregation layers. Figure 2 illustrates the performance change in XGB-Graph and RF- Graph with varying numbers of neighbor ag- gregation layers. Observably, the performance on most datasets improves when the number of neighbor aggregation layers increases from 0 to
Reddk
Amazzz Yel
Tinaace
Telokess
Dyuphfia Taxia
Questioss
XGB-Graph
RF Griph
AUPRC
2, confirming the effectiveness of the neighbor aggregation process. However, further incre- ments in the number of layers do not contribute to any significant improvement in the model performance. Consequently, in most instances, two layers are adequate for XGB-Graph and RF- Graph, and utilizing more layers could lead to unnecessary computational overhead and mem- ory usage.
022
60
Actrogation Layers
Aggrapsion Layers
Figure 2: The impact of different number of neigh- bor aggregation layers on the performance of XGB- Graph and RF-Graph.
4.3
Why and When Do Tree Ensembles with Neighbor Aggregation Outperform GNNs?
An initial study on decision boundaries. Inspired by a recent benchmark about ensemble trees and neural networks on the tabular data (28), we explore the possible reasons for the superior performance of ensemble trees with neighborhood aggregation. Specifically, our primary investigation focuses on the models' decision boundaries.
In the left panel of Figure 3 we visualize the decision boundaries of GIN and RF-Graph on Amazon dataset. For detailed implementations, plcase see Appendix E It is observed that the normal and anomalous nodes are closely intertwined, making them hard to separate. Unfortunately, GIN tends to produce simple and smooth decision boundaries, leading to frequent misclassification of normal nodes in the right bottom corners. Differently, RF- Graph can produce more intricate decision boundaries. demonstrating greater proficiency in distinguishing anomalous data. In the right panel of Figure 3 we visualize the decision boundaries of BWGNN and XGB-Graph on Weibo dataset. As can be seen, the anomalous nodes are grouped into several dispersed clusters. With this dispersed distributions, BWGNN is hard to achieve accurate classification due simple and continuous decision boundaries. In contrast, XGB-Graph successfully classifies anomalies within each cluster.

---

GIN (Amazon)
RF-Crrahh (Amazon)
BWGNN (Weibo)
GGah (Weiho)
Figure 3: Decision boundaries comparison of different approaches. Blue points represent anomalies while red points are normal nodes. Similarly, the blue/red regions correspond to model predictions for anomalous/normal classes.
In summary, anomaly instances tend to form multiple dispersed clusters and are coupled with normal instances, which fall in the categories of the inductive bias of RF-Graph and XGB-Graph that favor complex and disjoint decision boundaries. In contrast, as GNNs typically employ MLP as the final layer, they tend to generate simple and continuous decision boundaries, which makes GNNs sub-optimal on some challenging GAD datasets.
The impact of dataset feature types on model performance. As indicated in Table 3] out of the 10 datasets in GADBench, datasets purely use text embeddings as node features, while in the remaining 7 datasets, node features contain miscellaneous information such as the combination of numerical, categorical, and temporal features. Notably, for datasets that rely on text-based features-namely Reddit, Weibo, and Questions GNNss showcase competitive performance in comparison to other methods including tree ensembles. This could be attributed to the nature of text embeddings: they often represent oow-dimensional manifolds in a nigh-dimensional feature space, where dimensions tend to be highly correlated. A GNN can process all these dimensions simultancously, whereas an individual decision tree might only consider limited subset of feature columns. Conversely, in the other 7 datasets with diverse feature types that have low correlation (for instance, gender and age information), tree ensembles with neighbor aggregation typically exhibit superior performance.
In conclusion, in common GAD scenarios such as fraud detection, node features mainly originate from user profiles, which may encompass varied feature types. Thus, tree ensembles with neighbor aggregation often emerge as the preferred choice. However, for specific tasks like fake news and rumor detection, where text data is pivotal, GNNs still present compelling option.
5 Conclusion and Future Plan
In this paper, we introduce GADBench, the first comprehensive benchmark for supervised anomalous node detection static attributed graphs. Our evaluation of 29 models on 10 real-world datasets shows that tree ensembles with simple neighborhood aggregation generally outperform other models, including GNNs specifically designed for the GAD task. The rationale behind this finding is initially examined from the standpoints of decision boundary and node feature type. Our results challenge the prevailing belief about the superiority of GNNs in GAD and underline the importance of a fair and comprehensive comparison in accurately understanding the capabilities of various models. By making GADBench open-source, aim to foster further research and refinement of GAD algorithms, as well as their more informed evaluations and comparisons.
We regard GADBench a long-term evolving project and are dedicated to its continuous development. Our roadmap for the future includes expanding its scope to include a broader spectrum of GAD scenarios, incorporating more cutting-edge models, and integrating newer datasets. At present, primarily focus on treating datasets as static graphs to ensure compatibility with most baselines. We have only embarked on preliminary studies concerning heterogeneous and inductive settings. Looking ahead, we envision extending our evaluations to more complex types of graphs and anomalies. Our ultimate goal is to transform GADBench into features like automated model selection [95].
robust, scalable GAD toolbox, with advanced
Acknowledgement This research was supported by NSFC Grant No. 62206067, Tencent AI Lab Rhino Rhino-Bird Focused Research Program and Cuangyhou--HKUSSCCCZ Joint Funding Scheme 202340300673.
10

---

References
[1] Andra Acsintoae. Andrei Florescu, Mariana-Iuliant Georgescu, Tudor Mare, Paul Sumedrea, Radu Tudor lonescu, Fahad Shahbaz Khan, and Mubarak Shah. Ubnormal: New benchmark for supervised open-set video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20143-20153, June 2022.
[2] Charu C Aggarwal. Outlier ensembles: position paper. ACM SIGKDD Explorations Newsletter, 14(2):49-
58, 2013.
[3] Leman Akoglu, Mary McGlohon, and Christos Faloutsos. Oddball: Spotting anomalies in weighted graphs. In Advances in Knowledge Discovery and Data Mining: 14th Pacific-Asia Conference, PAKDD 2010, Hyderabud, India, June 21-24 21-24, 2010. Proceedings. Part 14, pages 410-421. Springer, 2010.
[4] Leman Akoglu, Hanghang Tong, and Danai Koutra. Graph based anomaly detection and description: a survey. Data mining and knowledge discovery, 2963):626-688, 2015.
[5] Susan Athey, Julie Tibshirani, and Stefan Wager. Generalized random forests. The Annals of Statistics, 47())1148-1178. 2019.
[6] Mariana Belgiu and Lucian Drägut, Random forest in remote sensing: A review of applications and future directions. ISPRS journal of photogrammetry and remote sensing, 1 114:24- 31, 2016.
[7] James Bergstra, Daniel Yamins, and David Cox. Making science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference machine learning, pages 115-123. 123. PMLR, 2013.
[8] Tian Bian, Xi Xiao, Tingyang Xu, Peilin Zhao, Wenbing Huang, Yu Rong, and Junzhou Huang. Rumor detection on social media with bi-directional graph convolutional networks. In Proceedings of the AAAI Conference Artificial Intelligence, 2020.
[9] Gérard Biau. Analysis of random forests model. The Journal of Machine Learning Research, 13(1):1063- 1095, 2012.
[10] Azzedine Boukerche, Lining Zheng, and Omar Alfandi. Outlier detection: Methods, models, and classification. ACM Computing Surveys (CSUR), 53(3):1337, 2020.
[11] Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, et al. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems, 3:747-769, 2021.
[12] Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
[13] Ziwei Chai, Siqi You, Yang Yang, Shiliang Pu, Jiarong Xu, Haoyang Cai, and Weihao Jiang. Can abnormality detected by graph neural networks? In IJCAI, 2022.
[14] Deepayan Chakrabarti. Autopart: Parameter-fin graph partitioning and outlier detection. In European conference on principles of data mining and knowledge discovery, pages 12-1224 Springer, 2004.
[15] Chih-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM transaction on intelligent systems and technology (TIST), 2(3):1-27, 2011.
[16] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
[17] Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbour- hood aggregation for graph nets. Advances in Neural Information Processing Systems, 33:13260-11277 2020.
[18] Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1)21127,, 1967.
[19] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. NeurIPS, pages
2016.
[20] Ailin Deng and Bryan Hooi. Graph neural network bessork-based anomaly detection in multivariate time series.
In Proceedings of the AAAI conference artificial intelligence, pages 4027-4035, 2021.

---

[21] Kaize Ding, Jundong Li, Rohit Bhanushali, and Huan Liu, Deep anomaly detection on attributed networks. In Proceedings of the SDM, pages 594-602. SIAM, 2019.
[22] Yingtong Dou, Zhiwei Liu, Li Sun, Yutong Deng, Hao Peng, and Philip S Yu. Enhancing graph neural network-based fraud detectors against camouflaged fraudsters. In CIKM, pages 315-324, 2020.
[23] Yingtong Dou, Guixiang Ma, Philip S Yu, and Sihong Xie. Robust spammer detection by nash rein- forcement learning. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
[24] Jingean Duan, Siwei Wang, Pei Zhang. En Zhu, Jingtao Hu, Hu Jin, Yue Liu, and Zhibin Dong. Graph anomaly detection via mulli-scall contrastive learning networks with augmented view, arXiv preprint 1rX1v:2212:00535, 2022.
[25] Zhouyu Fu, Weiming Hu, and Tieniu Tan. Similarity based vehicle trajectory clustering and anomaly detection. In IEEE International Conference on Image Processing 2005, volume 2, pages II-602, Iccc, 2005.
[26] Yuan Gao, Xiang Wang, Xiangnan He, Zhenguang Liu, Huamin Feng, and Yongdong Zhang. Addressing heterophily graph anomaly detection: perspective graph spectrum. Proceedings of the ACM Web Conference, 2023.
[27] Yuan Gao, Xiang Wang. Xiangnan He, Zhenguang Liu, Huamin Feng, and Yongdong Zhang. Alleviating structural distribution shift in graph anomaly detection. In Proceedings of the Sixteenth ACM International Conference Web Search and Data Mining, pages 357-365, 2023.
[28] Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[29] Zoltán Gyöngyi, Hector Garcia-Molina and Jan O. Pedersen. Combating web spam with trustrank. In (e)Prcceedings of the Thirtieth International Conference on Very Large Duta Basex, VLDB, 2004.
[30] William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurIPS, 2017.
[31] Songqiao Han, Xiyang Hu, Hailiang Huang, Minggi Jiang, and Yue Zhao. ADBench: Anomaly detection benchmark. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[32] Mingguo Hc, Zhewei Wei, Hongteng Xu, et al. Bernnet: Learning arbitrary graph spectral filters via 14251, 2021.
bemsstei approximation. Advances in Neural Information Processing Systems,
[33] NA Heard, David J Weston, K Platanioti, and DJ Hand. Bayesian anomaly detection methods for social networks. The Annals of Applied Statistics, 4(2):655 (62266666...222 2010.
[34] Keith Henderson, Tina Eliassi-Rad, Christos Faloutsos, Leman Akoglu, Lei Li, Koji Maruhashi, B Aditya Prakash, and Hanghang Tong. Metric forensics: a multi-level approach for mining volatile graphs. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 163-172, 2010.
[35] Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In Proceedings the web conference 2020, pages 2704-2710. 2020.
[36] Xuanwen Huang, Yang Yang, Yang Wang, Chunping Wang. Zhisheng Zhang, Jiarong Xu, Lei Chen, and Michalis Vazirgiannis. Dgraph: A large-scale financial dataset for graph anomaly detection. In Thirty-sixhh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[37] Sergei Ivanov and Liudmila Prokhorenkova. Boost then convolve: Gradient boosting meets graph neural networks. In ICI.R, 2021.
[38] Minqi Jiang, Chaochuan Hou, Ao Zheng, Xiyang Hu, Songqiao Han, Hailiang Huang, Xiangnan He, Philip Yu, and Yue Zhao. Weakly supervised anomaly detection: A survey, arXiv preprint 200.::2302.04549 2023.
[39] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
12

---

[40] Danai Koutra, Toi Ke, U Kang, Duen Horng Polo Chau, Hsing- Lising-Kuo Kenneth Pao, and Christos Faloutsos. Unifying guilt-by-associtiinn approaches: Theorems and fast algorithms. In Joint European Conference on Machine Learning and Knowledge Discovery Databases, pages 245-260. Springer, 2011.
[41] Srijan Kumar, Bryan Hooi, Disha Makhija, Mohit Kumar, Christos Faloutsos, and VS Subrahmanian. Rev2: Fraudulent user prediction in rating platforms. In Proceedings of the Fleventh ACM International Conference on Web Search and Data Mining, pages 333-341, 2018.
[42] Srijan Kumar, Xikun Zhang, and Jure Leskovec. Predicting dynamic embedding trajectory in temporal interaction networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1269-1278, 2019.
[43] Kingsly Leung and Christopher Leckie. Unsupervised anomaly detection in network intrusion detection using clusters. Proceedings of the Twenty-eighth Australasian conference on Computer Science-Volumm 38, pages 333-342, 2005.
[44] Ao Li, Zhou Qin, Runshi Liu, Yiqun Yang. and Dong Li. Spam review detection with graph convolutional networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 2703-2711, 2019.
[45] Jundong Li, Harsh Dani, Xia Hu, and Huan Liu. Radar: Residual analysis for anomaly detection in attributed networks. In IJCAI, pages 2152 2158, 2017.
[46] Tianfu Li, Zheng Zhou, Sinan Li, Chuang Sun, Ruqiang Yan, and Xuefeng Chen. The emerging graph neural networks for intelligent fault diagnostics and prognostics: A guideline and henchmark study. Mechanical Systems and Signal Processing, 168:108653, 2022.
[47] Fanzhen Liu, Xiaoxiao Ma, Jia Wu, Jian Yang. Shan Xue, Amin Beheshti, Chuan Zhou, Hao Peng. Quan Z Sheng, and Charu C Aggarwal. Dagad: Data augmentation for graph anomaly detection. arXiv preprint arXiv:22 w0.:2210.0990000 1000666,...... 2022.
[48] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation-base basion anomaly detection. ACM Transactions on Knowledge Discovery from Data (TKDD), 6(1):1-39, 2012.
[49] Kay Liu, Yingtong Dou, Yue Zhao, Xueying Ding. Xiyang Hu, Ruitong Zhang, Kaize Ding, Canyu Chen, Hao Peng, Kai Shu, Lichao Sun, Jundong Li, George H Chen, Zhihao Jia, and Philip S Yu. Bond: Benchmarking unsupervised outlier node detection on static attributed graphs. In Advances in Neural Information Processing Systems, volume 35, 2022.
[50] Yang Liu, Xiang Ao, Zidi Qin, Jianfeng Chi, Jinghua Feng, Hao Yang. and Qing He. Pick and choose: A gnn-basee gnn-based imbalanced learning approach for fraud detection. In Proceedings of the Web Conference 2021, 2021.
[51] Zhiwei Liu, Yingtong Dou, Philip S Yu, Yutong Deng, and Hao Peng. Alleviating the inconsistency problem of applying graph neural network to fraud detection. In SIGIR, pages 1569-1572. 2020.
[52] Zhiyuan Liu, Chunjie Cao, and Jingzhang Sun. Mul-gad: a semi- supervipee graph anomaly detection framework via aggregating multi-view information. arXiv preprint 22Xx:2222.005478 2022.
[53] Zhiyuan Liu, Chunjie Cao, Fangjian Tao, and Jingzhang Sun. Revisiting graph contrastive learning for anomaly detection. arXiv preprint 2XXx:3305.02496 2023.
[54] Ziqi Liu, Chaochao Chen, Longfei Li, Jun Zhou, Xiaolong Li, Le Song, and Yuan Qi. Geniepath: Graph neural networks with adaptive receptive paths. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4424 4431, 2019.
[55] Xiaoxiao Ma. Jia Wu, Shan Xue, Jian Yang, Chuan Zhou, Quan Z Sheng. Hui Xiong, and Leman Akoglu. A comprehensive survey on graph anomaly detection with deep learning. IEEE Transactions on Knowledge and Data Engineering, 2021.
[56] Julian John McAuley and Jure Leskovec. From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. In WWW, 2013.
[57] Caleb C Noble and Diane J Cook. Graph-based anomaly detection. In KDD, 2003.
[58] Adeola Ogunleye and Qing-Gon Guo Wang. Xgboost model for chronic kidney disease diagnosis, JEEE/ACM transactions computational biology and bioinformatics, 17(6):2131-214 2019.
13

---

[59] John Paparrizos, Yuhao Kang, Paul Boniol, Ruey S Tsay, Themis Palpanas, and Michael J Franklin. Tsb-uad: end-to-end benchmark suite for univariate time-series anomaly detection. Proceedings of the VLDB Endowment, 2022.
[60] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen. Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. NewrIPS, 8037,
[61] Dhaval Patel, Giridhar Ganapavarapu, Srideepika Jayaraman, Shuxin Lin, Anuradha Bhamidipaty, and Jayant Kalagnanam. Anomalykits: Anomaly detection toolkit for time series. In Proceedings of the AAAI Conference on Artificial Intelligence, pages $3209-13211, 2022.
[62] Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, al. Scikit-leam: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
[63] Zhen Peng, Minnan Luo, Jundong Li, Huan Liu, and Qinghua Zheng. Anomalous: A joint modeling approach for anomaly detection on attributed networks. In IJCAI, pages 3513 35199, 2018.
[64] Muhammad Shakil Pervez and Dewan Md Farid, Feature selection and intrusion classification in nsl- kdd cup 99 dataset employing svms. In The 8th International Conference Software, Knowledge, Information Management and Applications (SKIMA 2014), pages 1-6. IEEE, 2014.
[65] Oleg Platonov. Denis Kuznedelev. Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova. A critical look at the evaluation of gnns under heterophily: are we really making progress? In ICLR, 2023.
[66] Shebuti Rayana and Leman Akoglu. Collective opinion spam detection: Bridging review networks and metadata. In KDD, pages 985-994, 2015.
[67] Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review, 65(6))386, 1958.
[68] Lukas Ruff. Robert A Vandermeulen, Billy Joe Franks, Klaus-Robed Müller, and Marius Kloft. Rethink- ing assumptions in deep anomaly detection. arXiv preprint arXiv:2006.003: 2020.
[69] Patricia Iglesias Sánchez, Emmanuel Müller, Fahian Laforet, Fabian Keller, and Klemens Böhm. Statistical selection of congruent subspaces mining attributed graphs. In 2013 IEEE 13th international conference on data mining, pages 647-656. IEEE, 2013.
[70] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. The Semantic Web: 15th International Conference, pages 593-607, 2018.
[71] Prithviraj Sen, Galilco Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data, Al magazine, 29(3):93-93, 2008
[72] Fengzhao Shi, Yanan Cao, Yanmin Shang, Yuchen Zhou, Chuan Zhou, and Jia Wu. H2-fdetector: A gnn-based fraud detector with homophilic and heterophilic connections. In Proceedings of the ACM Web Conference 2022, pages 1486-1494, 2022.
[73] Yunsheng Shi, Zhengjie Huang. Shikun Feng, Hui Zhong, Wenjing Wang. and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. In IJCAI, 2021.
[74] Jianheng Tang, Jiajin Li, Ziqi Gao, and Jia Li. Rethinking graph neural networks for anomaly detection. In International Conference on Machine Learning, 2022.
[75] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using L-sne. Journal of machine learning research, 911), 2008.
[76] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, and Yoshua Bengio. Graph attention networks. In ICLR, 2017.
[77] Miryam Elizabeth Villa-Pérez. Miguel Alvarez- Alvarez-Carmona, Medina-Párez, Pérez, Juan Carlos Velazco-Rossell and Kim-Kwang Raymond Choo. Semi-supervised anomaly detection algorithms: A comparative summary and future research directions. Knowledge-Band Based Systems, 218: 10678, 2021.
Carmona, Octavio Loyola-Gonzalez, Gonzalez, Miguel Angel
[78] Chen Wang. Chengyuan Deng, and Suzhen Wang. imbalance-xgboosd leveraging weighted and focal losses for binary aabel-imbalanee classification with xgboost. Pattern Recognition Letters, 136: 190-197, 2020.
14

---

[79] Daixin Wang, Jianbin Lin, Peng Cui, Quanhui Jia, Zhen Wang, Yanming Fang, Quan Yu, Jun Zhou, Shuang Yang, and Yuan Qi. semi-supervised graph attentive network for financial fraud detection. In ICDM, pages 598-607. IEEE, 2019.
[80] Minjie Wang. Da Zheng. Zihao Ye, Quan Can. Mufei Li, Xiang Song. Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai. Tianjun Xiao, Tong He. George Karypis. Jinyang Li, and Zheng Zhang. Deep graph library: A graph-centric, perfoytterfommant package for graph neural networks. 00XXiv:999.003/55. 2019.
[81] Xiang Wang and lan Davidson. Discovering contexts and contextual outliers using random walks in graphs. In 2009 Ninth IEEE International Conference on Data Mining, pages 1034-1039. IEEE, 2009.
[82] Yanling Wang. Jing Zhang, Shasha Guo, Hongzhi Yin, Cuiping Li, and Hong Chen. Decoupling representation learning and classification for gnn-based anomaly detection. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1239-1248, 2021.
[83] Mark Weber, Giacomo Domeniconi, Jie Chen, Daniel Karl Weidele, Claudio Bellei, Tom Robinson, and Charles E. Leiserson. Anti-money laundering in bitcoin: Experimenting with graph convolutional networks for financial forensics. arXiv preprint arXiv:1908.02550/ 2019.
[84] Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinherger. Simplifying graph convolutional networks. In ICML. pages 6861-6871, 2019.
[85] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? ICLR, 2019.
[86] Xiaowei Xu, Nurcan Yuruk. Zhidan Feng, and Thomas AJ Schweiger. Scan: a structural clustering algorithm for networks. In Proceedings of the 13th ACM SIGKDD international conference Knowledge discovery and data mining, pages 823,833, 2007.
[87] Jiawei Yang. Yu Chen, and Sylwan Rahardja. Neighborhood representative for improving outlier detectors. Information Sciences, 625:192 205, 2023.
[88] Jiawei Yang. Susanto Rahardja, and Pasi Franti. Neighborhood averaging for improving outlier detectors. arXiv preprint arXiv:2303.00 09972, 2023.
[89] Jiawei Yang. Sylwan Rahardja, and Susanto Rahardja. Foor: Be careful for outlier-score outliers when
using unsupervised outlier ensembles. IEEE Transactions on Computational Social Systems, 2023.
[90] Ge Zhang, Jia Wu, Jian Yang. Amin Beheshti, Shan Xuc, Chuan Zhou, and Quan Z Sheng. Fraudre: Fraud detection dual resistant graph inconsistency and imbalance. In 2021 IEEE International Conference on Data Mining (ICDM), pages 867-876. IEEE, 2021.
[91] Wentao Zhang, Ziqi Yin, Zeang Sheng, Yang Li, Wen Ouyang, Xiaosen Li, Yangyu Tao, Zhi Yang, and Bin Cui. Graph attention multi-layer pereeptron. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4560-4570, 2022.
[92] Tong Zhao, Chuchen Deng, Kaifeng Yu, Tianwen Jiang, Daheng Wang, and Meng Jiang. Error-hounded graph anomaly loss for gnns. In CIKM, 2020.
[93] Yue Zhao and Maciej K Hryniewicki. Xgbod: improving supervised outlier detection with unsupervised representation learning. In 2018 International Joint Conference on Neural Networks (IJCNN, pages 1-8. IEEE, 2018
[94] Yue Zhao, Zain Nastullah, and Zheng Li. PyOD: A python toolbox for scalable outlier detection. Journal of Machine Learning Research, 20(96): 2019.
[95] Yue Zhao, Ryan Rossi, and Leman Akoglu. Automatic unsupervised outlier model selection. Proceedings of the NeurIPS. 34:4489-4500 2021.
[96] Li Zheng, Zhenpeng Li, Jian Li, Zhao Li, and Jun Gao. Addgraph: Anomaly detection in dynamic graph using attention- based temporal gen. In IJCAI, pages 4419 4425, 2019.
[97] Jiancheng Zhong, Yusui Sun. Wei Peng. Minzhu Xie, Jiahong Yang, and Xiwei Tang. Xgbfemf: an xgboost-based framework for essential protein prediction. IEEE transactions on nanobioscience. 17(3):243-255 2018.
[98] Hao Zhu and Piotr Koniusz. Simple spectral graph convolution. In International conference on learning representations, 2021.
15

---

[99] Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, 33, 2020.
[100] Arthur Zimek, Ricardo JGB Campello, and Jörg Sander. Ensembles for unsupervised outlier detection: challenges and research questions position paper, Acm Sigkdd Explorations Newsletter. 15(1):11 2014.
16

---

A Detailed Description of Models in GADBench
Classic Methods
MLP (Multi-Layer Perceptron 667 A type of feedforward neural network composed of multiple
layers of interconnected artificial neurons that can learn and make predictions by adjusting the weights and biases of the connections between the neurons.
KNN (k-Nearest Neighbors [18): non-parametric classification algorithm that assigns labels to data points based on the labels of their k nearest neighbors within the feature space.
SVM (Support Vector Machine [15]:: A supervised learning algorithm that classifies data by identifying an optimal hyperplane that maximally separates different classes in a high-dimensional feature space.
An ensemble learning algorithm that utilizes bagging train collection of individual decision tree models on subsets of the original dataset. The predictions from these individual models then combined through averaging or voting, resulting in a robust and accurate prediction model.
RF (Random Forest 12):
XGBoost (eXtreme Gradient Boosting [16]:: A gradient boosting decision tree framework that uses a gradient descent algorithm to minimize the loss function and iteratively adds new trees into the model to correct the errors made by the previous trees.
XGBOD (Extreme Boosting Based Outlier Detection ): An enhanced XGBoost model that integrates unsupervised outlier detectors to generate abnormality scores as additional features for the original data. These generated features concatenated to the original feature set and used in an XGBoost classifier for outlier detection.
Neighborhood Averaging [8] A method that post-processes the outlier scores provided by any existing outlier detector by averaging with the scores of its neighbors in the feature space. In GADBench, we integrated this technique into BWGNN case study for in-depth analysis.
Standard GNN Architectures
method that utilizes convolution operation on the graph to propagate information from node to its neighboring nodes, enabling the network to learn a representation for each node based on its local neighborhood.
GCN (Graph Convolutional Network 1:
ChebNet Chebyshev Spectral Convolution Network (19):: A variant of GCN that applies Chebyshev polynomials approximate the spectral graph convolution operator. This approach allows the model to capture both local and global graph structures, making it scalable for larger graphs.
GIN (Graph Isomorphism Network (85): type of GNN that learns to capture the structure of graph while respecting graph isomorphism. This means it generates identical embeddings for graphs that are structurally identical, regardless of permutations in their node labels.
GraphSAGE (Graph Sample and AggregatE (30]:) A general inductive learning framework that generates node embeddings by sampling and aggregating features from node's local neigh- borhood.
GAT (Graph Attention Networks (76)): A GNN framework that incorporates the attention mechanism. assigns varying levels of importance to different nodes during the neighborhood information aggregation process, allowing the model to focus on the most informative parts.
GT (Graph Transformer (73)): An adaptation of the neural network architecture that applies the principles of the Transformer model to graph-structured data. It uses masks in the self-attention process leverage the graph structure and enhance model efficiency
PNA (Principle Neighbor Aggregation) [17 A novel graph neural network architecture combin- ing multiple aggregators with degree-scaless scalers in the neighborhood aggregation process.
BGNN (Gradient Boosting Meets Graph Neural Networks) [37] An end-to-end framework
that trains GBDT and GNN jointly by allowing new trees to fit the gradient updates of GNN.
RGCN (Relational Graph Convolution Network) [70]: A specialized framework derived from
Graph Convolutional Networks, tailored for handling relational data.
HGT (Heterogeneous Graph Transformer) 35): GNN framework to model Web-scale het- erogeneous graphs by designing node- and edge-type dependent parameters for characterizing heterogeneous attention over each edge, thus facilitating the maintenance of dedicated representa- tions for different types of nodes and edges.
17

---

GNNs Specialized for Graph Anomaly Detection
GAS (GCN-based Anti-Spam [44]) A highly scalable method for detecting spam reviews. extends GCN to handle heterogeneous and heterophilic graphs and adapts to the graph structure of specific GAD applications using the KNN algorithm.
DCI (Deep Cluster Infomax [ A self-supervised learning scheme that decouples node representation learning from classification for anomaly detection. It mitigates inconsistencies between node behavior patterns and label semantics, and captures intrinsic graph properties in concentrated feature spaces by clustering the entire graph into multiple parts.
PC-GNN (Pick and Choose Graph Neural Network [50): A framework designed for imbalanced GNN learning in fraud detection. It uses a label-balanced sampler to select nodes and edges for training, resulting in balanced label distribution in the induced sub-graph. Furthermore, it employs a learnable parameterized distance function to select neighbors, filtering out redundant links and adding beneficial ones for fraud prediction.
BernNet [32]: A GNN variant that offers a robust scheme for designing and learning arbitrary graph spectral filters. It uses an order-K Bernstein polynomial approximation to estimate any filter over the normalized Laplacian spectrum of graph.
GAT-sep [99]: GNN designed to enhance learning from graph structures under high het- erophily. It combines key designs such ego- and neighbor-embedding separation, higher-order neighborhoods, and intermediate representation combinations.
AMNet (Adaptive Multi-frequency Graph Neural Network [13): : A method designed to
capture both -rw-freuency and high-frequency signals through stacking multiple BernNets, and adaptively combine signals of different frequencies.
BWGNN (Beta Wavelet Graph Neural Network (74)): A method proposed to tackle the 'right- shift' phenomenon of graph anomalies, i.e., the spectral energy distribution concentrates less on low frequencies and more high frequencies. It employs the Beta kernel to address higher frequency anomalies through multiple flexible, spatial//pecctaallocate localized, and band-pass filters GHRN (Graph Heterophily Reduction Network (26)): A method that addresses the heterophily issue in the spectral domain of graph anomaly detection. The approach prune inter-class edges to emphasize and delineate the graph's -eghffreueenc components.
GNN) [2]: A GNN-based fraud detector designed for multi-relation graphs, which is equipped with three modules that enhance its performance against camouflaged fraudsters.
CARE-GNN (CAmouflage-Rsistat
H2-FDetector (Fraud Detector with Homophilic and Heterophilic Interactions) [72 A het- erogeneous GNN framework for fraud detection that facilitates the propagation of analogous information through homophilic connections and varied information via heterophilic connection.
B Detailed Description of Datasets in GADBench
This dataset contains a user-subreddit graph, capturing one month's worth of posts shared across various subreddits. Verified labels of banned users are included. The dataset focuses on the 1,000 most active subreddits and the 10,000 most engaged users, leading to a total of 672,447 interactions. Posts were transformed into feature vectors, each representing the Linguistic Inquiry and Word Count (LIWC) categories of the text.
Reddit 42
Weibo 42 This dataset features a graph of users and their associated hashtags from the Tencent- Weibo platform, consisting of 8,405 users and 61, 11,964 hashtags. Suspicious activities defined as two posts made within specific timeframes, such as 60 seconds. Users that engaged at least five such activities are labeled as "suspicious", while the rest are categorized "benign". This process yielded 868 suspicious and 7,537 benign users. The raw feature vector is composed of the location of micro-blog post and bag-of-words features.
Chlp 66):
This dataset is designed to identify anomalous reviews that unfairly promote or demote products businesses Yelp.com. The graph includes three types of edges: R-U-R (reviews posted by the same user), R-S-R (reviews for the same product with the same star rating), and R-T-R (reviews for the same product posted in the same month).
Amazon [56] The goal of this dataset is identify users paid to write fake reviews for products in the Musical Instrument category on Amazon.com The graph includes three types of relations:
18

---

Table 7: Overview of datasets in GADBench with their corresponding node feature types, feature dimension, and detailed descriptions.
Dataset
Node Feature Type
#Dim.
Detailed Feature Description
Reddit Weibo
Text Embedding
64
LIWC text embedding for posts Bag-of-words features from posts Hand-crafted user features and statistics Hand-craftee review features and statistics 10
Text Embedding
400
Amazon
Misc. Information
25
YelpChi
Misc. Information
32
Tolokers
Misc. Information
User profile with task performance statistics FastText embeddings for user descriptions
Questions
Text Embedding
301
T-Finance
Misc. Information
10
User profile details such as registration days Timestamps and transaction information Timestamps and user profiles details User profile details such as logging activities
Elliptic
Misc. Information
166
DGraph-Fin
Misc. Information
17
T-Social
Misc. Information
10
U-P-U (users reviewing at least one same product), U-S-U (users giving at least one same star rating within one week), and U-V-U (users with top-5% mutual review similarities).
T-Finance 74
This dataset aims to find the anomaly accounts in transaction networks. The nodes are unique anonymized accounts with 10-dimension features related to registration days, logging activities and interaction frequency. The edges in the graph represent two accounts that have transaction records. Human experts annotate nodes as anomalies if they fall into categories like fraud, money laundering and online gambling.
Tolokers [65] : This dataset is derived from the Toloka crowd-sourcing platform. Nodes represent workers who have participated in at least one of 13 selected projects, and an edge connects two workers if they worked on the same task. The task is to predict which worker has been banned in one of the projects. Node features are based on worker's 's profile and task performance statistics.
Questions 655 This dataset collected from the question-answerrin website Yandex Q. Nodes are users, and an edge connects two users if one user answered the other user's 's question during one-year period (September 2021 to August 2022). The dataset focuses on users interested in the topic "medicine". The task is to predict which users remained active on the website at the end of the period. Node features are the mean of FastTexx embeddings for words in the user description, with an additional binary feature indicating users without descriptions.
This dataset includes a graph of over 200,000 Bitcoin transactions (nodes), 234,000 directed payment flows (edges). and 166 node features. The dataset maps Bitcoin transactions to real-world entities associated with licit categories, such as exchanges, wallet providers, miners, and legal services, as well as illicit categories like scams, malware, terrorist organizations, ransomware, and Ponzi schemes.
Elliptic [33]
DGraph-Fin [36]: This dataset is a real-world, large-scale dynamic graph provided by the Finvolution Group, representing social network within the financial industry. In DGraph-Fin, a node represents a Finvolution user, and an edge between two users indicates that one user lists the other as their emergency contact. Anomalous nodes in DGraph-Fin represent users who exhibit overdue behaviors. The dataset comprises over 3 million nodes, 4 million dynamic edges, and more than 1 million extremely unbalanced ground-truth nodes.
This datasct aims to find the anomaly accounts in social networks. has the same node annotations and features T-Finance, while two nodes are connected if they maintain the friend relationship for more than three months. Same as T-Finance, human experts annotate nodes as anomalies if they fall into categories like fraud, money laundering and online gambling.
T-Social 74:
C
Other Information in GADBench
C.1
Metries
AUPRC (Area Under the Precision-Recall Curve). AUPRC a metric that evaluates the perfor- mance of classification models by computing the area beneath the Precision-Recall curve. This curve
19

---

illustrates the relationship between precision (i.e., the ratio of true positive predictions to all positive predictions) and recall (i.e., the ratio of true positive predictions to all positive labels) at different threshold levels. AUPRC can be calculated by the weighted mean of precisions at each threshold, where the increase in recall from the previous threshold serves as the weight.
AUROC (Area Under the Receiver Operating Characteristic Curve). It evaluates a model's ability to discriminate between positive and negative classes by measuring the area under the ROC curve. The ROC curve plots the true positive rate against the false positive rate for varying decision thresholds. An AUROC of indicates perfect discrimination, while an AUROC of 0.5 suggests that the model is no better than random guessing.
Rec@K K (Recall at k). It is determined by calculating the recall of the true anomalies among the top-k predictions that the model ranks with the highest confidence. We set the value of l: as the number of actual outliers in the test dataset. It is noteworthy that in this specifie scenario, Rec @ K is equivalent to both precision at k and the F1 score at k.
Runtime. To assess the efficiency of the various models, including both traditional algorithms and neural networks, we measure the runtime as the duration from the beginning to the completion of the experiments. This measurement does not distinguish between computation times on CPUs and GPUs.
Memory. We record the peak utilization of both CPU and GPU memory during the entire supervised training phase for each algorithm on the DGraph- Fin dataset. This metric is crucial for assessing the computational resource requirements of each model in practical implementations.
C.2 Additional Experimental Details
Implementation Details. To ensure a comprehensive evaluation and maintain fairness across a broad spectrum of models, we develop an open-source toolkit named GADBenct This toolkit is built on top of Pytorch 1.12 [60] and DGL 1.0 [80] We implement all standard and specialized the
GNNs in GADBench using the DGL library. For classic models KNN, SVM, and RF, implementations provided in the Scikit-Learn library (62). XGBoost is integrated using its official implementation [16), and XGBOD is included via the PyOD library (94).
Hardware Specifications. All our experiments were carried out on a Linux server equipped with an AMD EPYC 75F3 32-Core CPU processor, 64GB RAM, and NVIDIA RTX A6000 GPU with 48G memory.
Hyperparameter Settings. Tables 8 and provide a comprehensive list of all hyperparameters used in our random search, complete with their default values and respective distributions or search spaces. For all configurations, we retain the model that yields the best AUPRC score on the validation set and report the corresponding test performance. Due to limited data, we did not perform hyperparameter search for heterogencous graph neural networks (RGCN, CARE-GNN, and H2 11-FFDetector.
New Models and Datasets. GADBench is highly extensible due its seamless integration with the DGL library. Datasets structured in the dg1. agl.graph format, accompanied by binary node labels, can be effortlessly integrated and leveraged by all models within GADBench. In a similar vein, models that accept @gl..grah as input and yield an anomaly score for each node can be smoothly assessed across all datasets in GADBench. We are committed to the ongoing maintenance of GADBench and will persist in incorporating new models and datasets.
D Additional Experimental Results
In accordance with Figure [ Tables TT and 12 provide the performance metrics of each model under semi-supervised and fully-supervised settings, respectively, when default hyperparameters are employed. Additionally, Table 13 consolidates the information in Table 4 and exhibits the performance of the models in terms of AUROC and Rec K scores subsequent to hyperparameter tuning for each dataset.
In table TO we evaluated the effect of incorporating Neighborhood Averaging (NA) on the perfor- mance of four representative graph- based methods in GADBench, namely GIN, BWGNN, XGB- Graph, and RF-Graph. In the semi-supervised setting, all of the techniques demonstrate an im-
20

---

GIN (T-Finance)
RF-Graph (T-Finance)
BWGNN (Tolokers)
NGB-Graph (Tolokers)
Figure 4: More decision boundaries comparison examples as a complement to Figure 3 Blue points represent anomalies while red points normal nodes. Similarly, the blue/red regions correspond to model predictions for anomalous/normal classes.
provement in performance subsequent to the integration of NA. For example, the AUPRC score is improved by 1.4%~ 3.7%. In the fully-supervised setting, NA can enhance weaker baselines like GIN. However, for stronger baselines, the performance remains similar (on BWGNN) might decrease (on XGB-Graph and RF-Graph).
E More Details About Plotting Decision Boundaries
As GNNs require graph structure the input, it is not feasible directly illustrate their decision boundaries. In the left panel of Figure 3 we use GIN removing the linear layer to embed nodes into the hidden space, following which we employ t-SNE 77] reduce node embeddings to two dimensions. We then apply MLP and Random Forest to classify these node embeddings, approximating comparison between GIN and RF-Graph. It is observed that the embeddings of normal and anomalous nodes are entwined, making them hard to separate. Unfortunately. GIN tends produce simple and smooth decision boundaries, leading to recurring misclassification of normal nodes in the right bottom corners, thus compromising model performance. Conversely, XGBoost can produce more intricate decision boundaries, demonstrating greater proficiency in distinguishing anomalous data.
The right panel of Figure showcases trained BWGNN model on the Weibo dataset, visualizing the node embeddings preceding the final MLP layer. Employing t-SNE again to reduce the node these embeddings,
embeddings, we compare the decision boundaries of MLP and XGBoost simulating comparison between BWGNN and XGB-Graph. As shown in the right panel of Figure the embeddings of anomalous nodes are grouped into several dispersed clusters after training. However, the MLP model struggles with accurate classification due to the lack of simple boundary between normal and anomalous data. contrast, XGBoost successfully classifies anomalies within each cluster. In Figure 4 we extend our visualization to include two additional datasets, T-Finance and Tolokers. This serves complement to Figure 3 and the observations remain consistent.
21

---

Table 8: Default hyperparameters and random search space for MLP and GNN models. Elements in '[.J" is randomly selected with equal probability during each trial. RandInt(u, returns an random integer between and b (both included).
Model
Hyperparameter
Default value
Distribution / Search Space
Shared hyperparameters for all neural networks
Common
learning rate dropout rate
0.01
10 Umformm -3,11
10.0.1,0.2.0.3] [16,32,64]
hidden dimension
32
epochs
100
Specific hyperparameters for each model
MLP
layers
2
[1,2,344]
activation
ReLU
[ReLU, LeakyReLU, Tanh]
GCN
layers
[1,2,3]
activation
ReLU
[ReLU, LeakyReLU, Tanh]
SGC
number of hops
[1.2.3,4]
activation
ReLU
[ReLU, LeakyReLU, Tanh]
MLP layers
[1,2]
GIN
layers aggregation
[1,2,3]
mean ReLU
[sum, mean, max]
activation
[ReLU, LeakyReUU Tanh]
GraphSAGE
layers
[1,2,3]
aggregation
mean RcLU
[mean, GCN, pool]
activation
[RcLU, LeakyReLU, Tanh]
GAT
layers
[1,2,3]
attention heads
[1,2,48]
GT
layers
2
[1,2,]]
attention heads
[1,2,48]
PNA
layers
[1,2,3,4]
activation
ReLU
[RcLU, LeakyReLU, Tanh]
BGNN
depth
[4,5,6.7]
iteration per epoch
10
[2.5.10.20]
GBDT learning rate
10 -2-05)
normalize features
False
[True, Falsel
activation
ReLU
[RcLU, LeakyReLU, Tanh]
GAS
layers
[1,2.3.4]
number of neighbors
RandInt(3.50)
distance function
cosine
feuclidean, cosine]
DCI
layers
[1,2,3.4]
pretain epochs 100
100)
number of clusters
RandInt(2.30)
PC-GNN
layers
[1,2,3]
samll ratio
Uniform(O. (.0.0.
over-sample ratio distance function
0.3
Uniform(0.01.0.3
cosine
[euclidean, cosine]
GAT-sep
layers
[1,2,3]
attention heads
[1.2,48]
BernNet
MLP layers
[1.2]
orders
[2,3,4.5]
AMNet
layers
[1,2,3]
orders
[2,3]
activation
ReLU
[ReLU, LeakyReLU, Tanh]
BWGNN
layers
[1,2,3,4]
MLP layers
[1.2]
activation
ReLU
[ReLU, LeakyReLU, Tanh]
GHRN
layers
[12,,3.4]
MLP layers
[1.2]
deletion ratio
0.015
10 (2-11)
22

---

Table 9: Default hyperparameters and random search space for non-deep learning models. Elements in 'IJJ" randomly selected with equal probability during each trial. RandInt(a, returns an random integer between and (both included).
Model
Hyperparameter
Default value
Distribution / Search Space
SVM
weights
uniform
[uniform, distance] 10
L2 regularization C
RF
number of estimators
100
RandInt( 10,200)
split criterion
gini
[gini, entropy]
max samples
Uniform(O.
XGBoost
number of estimators
100
RandInt(100200)
learning rate y
0.3
0.5 + 10 -1.0)
L,2 regularization À
1
[0,1,10] [0.5,0.75.1]
subsample rate booster
gbtree
[gbtree, dart]
XGBOD
Same XGBoost
NA
Same as XGBoost, with additional hyperparameter: RandInt(0,50)
number of neighbors 5
RF-Graph
Same as RF, with 2 additional hyperparameters:
aggregation layers 2
[1,2,3,4]
aggregation function
[sum, mean, max]
XGB-Graph
Same as XGBoost, with additional hyperparameters:
aggregation layers 2
[1,2,3,4]
aggregation function mean
[sum, mean, max]
Table 10: The impact of introducing neighborhood averaging (NA) the performance of four representative methods in GADBench.
Semi- Semi-Supervised
mally-Supervised AUPRC
Model
AUROC
AUPRC
Rec@K K
AUROC
Rec@ @ K
GIN
73.61
30.26
32.14
79.97
36.54
37.52
with NA
74.78
33.91
34.18
80.43
39.03
40.01
BWGNN
76.00
35.42
37.36
84.73
48.19
48.01
with NA
76.95
36.96
37.34
84.70
47.93
48.08
XGB-Graph with NA
77.74
42.91
41.29
86.71
61.09
57.82
78.41
44.30
42.07
86.05
59.58
57.41
RF-Graph with NA
78.94
43.37
40.83
85.35
59.65
56.90
79.56
45.14
42.28
85.17
58.54
55.33
23

---

Table 11: Comparison of AUPRC (top), AUROC (middle), and Rec@ K (bottom) for each model employing default hyperparameters in the semi-supervised setting. Each model is executed 10 times with varying random seeds, and the mean scores along with standard deviations are reported. Ave. denotes the average score across all 10 datasets.
AUPRC
Reddit
Weibo
Amazon
Yelp
T-Fin.
Ellip.
Talo.
Quest.
DGraph.
T:Social
Mean
MLP
4.4208
56.246.0
83.0216
23.6-23
53,4=1399
45.0=55.
33.3821
1.7
2.3:
3.9-08
31.3
KNN
4.+004
41.1+722
2622+11
19.3+17
49.1+6.5
25.1+337
28.3+144
4.7+1.1
1.7+0.11
38+00
25.3
SVM
4.7+0055
44 14.1010.00
62.2421.5
25.6-22
20.6-10.
26.5-42
33.6+116
7,1+166
1.9+0..3
3.6+06
23.0
RF
4.3:904
61.6=1000
819942
27.4
74.9
85.0=119
33.1-23
11.34336
2.14
7.0=:13
38.9
XGiBoost
4.1+04
62.1+776
N40++1
26.4442
709+50
79.5+22
31,0+19
9.1+34
20++0
6.1+13
XGBOD
4.3+06
6488-6.
268-38
70,3+44
78.4+17
30.9-224
9,043.2
2.0+002
NA
4.13804
69.34444
85.6:11.
27.3-33
73.9
79.2-13.3
31.2-220
10.1
2.1
6.4213
38.9
GCN
4.24088
86.0267
12
16.4222
60.5:
43.1::46
33.0=3.6
6,190.9
2.34:03
8.43388
29.3
SOC
3,++07
63.9+74
29.5+26
16.1+19
51,3-186
24-2+2.1
32.2+177
6.1+0.7
1.7+0.
4.++11
GIN
4.3+06
67.6+774
75.4+433
23.7+55
44.8+7.1
40.1+332
31,8+332
6.741.1
2.0+0.1
6.24117
30.3
GrahSAGE
4.5:1:66
58.526.
42.5
20.9:335
11.7::52
43.14:6
34.0=2.1
5.5
20002
7.
23.0
GAT
4.7+07
73.3+7.3
816+17
25.0++29
28.9+86
44.2+66
33.D=200
7,3+1.2
2.240.
9.2+20
30.9
4.3+07
78.2+447
71.4+55
23.7=330
17,2=1099
46.480
34.5-20
1,2+1.6
2.2+0.3
8.6+1.5
29.4
PNA
4.0.8.5
83.42-40
34.6433
168827
27.499.7
31.3229
32.68223
5.7
18402
22.8-555
26.0
RGNN
4.7+00.5
81.8+6.3
39.0+-5
17,6+113
75.8++2.
35.3+553
33.1++36
5,7+0.7
2,0+0.3
58.2+146
35.3
GAS
4.7+07
65.7+8844
10.7+17
21.7+333
45.7+134
46.0+49
31,7++30
6,3+2.0
2.5+003
8.6+-2.4
31.4
DC1
4.3+04
76.24443
72.5+79
24.0=488
51.0-72
43.4=49
32.1=42
6,1+1.3
2.0+0.2
7.4+255
31.9
POGNN
3.4:10.5
69.3+997
819219
25.0=:..5
58.1-113
40.34166
33.9-17
6,2188
24404
8.21166
32,9
GAT-sap
4.6++-0.
76.5+-5.8
80.94266
24.4-1.7
34.2-10.1
468+88
31.6++22
7
2.40033
10.4-19
32.1
BernNer
4.9-8:33
66.6-
81.2+224
23.9=277
51,8=12A
40.0=441
28.9=335
6,7+221
2.5+002
4.2+112
31.1
AMNe
67.1+:5.1
82.4422
23,9:35
6022:82
33.3-48
28.6::15
74414
2.240.3
3.11:3
RWGNN
4.2+007
N0.6+-4.7
81.7+22
23.7+229
609-1338
43.4-55
35.3+22
6.5+1.7
2.14033
15.9-62
GHRN
4.244:0.
77,0+662
80.7++1.7
23.8-28
63.41144
44.2-5
35.9-20
6.5+17
2.3+0.3
16.2:46
35.4
RF-Graph
4.5::04
73.819.5
70.745..
23.6:2..
$1.1127
808:11
35.8124
10.1128
2,01002
51.3:162
43,4
XGB-GGrph
4.1:::55
75.9:66.
8..41111
24.83:1
78.3+3.1
77,2432
34.1228
7.752.1
19902
40.6276
AUROC
Reddit
Weibo
Amazon
Yelp
T-Fin.
Ellip.
Tolo.
Quest.
DGraph.
T-Social
Avc.
MLP
59.11:43
66.6177
92:24:33
64.7
89.91144
19.4
68.1
61.213.1
69.11:1
59.1
KNN
58 38.9423
72.5+33
5+09
60.3-133
87.7+0.
78.6+447
63.0-1.
58.6+22
598++4
569-28
68.6
SVM
&1224
62
89.41477
834199
83.0
68.9-11
62.0422
65.243.1
57.6254
70.7
RF
61.0126
94.111.
91.110
68.0
93.0109
91.1199
66.5121
615149
63.31445
67.3
76.4
XGBoost
58.9+2.5
938-088
67.0=5.1
92,4+111
91.6119
59.5+55
66.1222
66.7=28
75.4
XGBOD
59.143.1
94.1
95.24013
56.8
92.511
916117
64.1
60.6139
64.21221
61.0246
74.9
NA
59.51223
93.7..0.6
91.1113
69.3
92.01122
90.9122
65.
59.71555
66.8114
64.9-3.1
75.7
GCN
56.9450
93.5+666
82.0+03
$1,2+337
88.3+25
M6-+-19
64.2+48
60.04222
66.4225
71 71.6+10
SOC
53.6+52
81.1+665
80.2-15
$1.1=29
74.5=12.3
80.6=12
63.6+-5.0
59.5+3.3
67.3
GIN
60.0:::.1
83.81
9161117
62.9-73
84.5145
86.21009
66.81552
62.2122.
65.7
70.4274
73.6
GraphSAGE
60.3.133
8188166
$1.1100
58.92433
68.91155
87.1110
67.614222
61.2122
61.813.3
72.0229
70.4
GAT
60.5+:3
76
92.4+199
65.6-441
85.0+45
88:5=21
68.1
62.3+14
67.2+19
75.4=488
75.1
59.11444
94.4144
58.6422
64.5443
73.51722
88.9116
69.7
60.7133.6
67.7225
714421
73.9
PNA
58.41444
93.21118
79.1129
53.2..55
78.419.7
83.7
65.71.3.1
62.11229
60.7139
84.4
71.9
BGNN
63.1+33.
94.4=53
83.7+255
14,4=112
93.7+007
83.3=23
65.1=-41
59.8+-311
59.9+335
93.6=114
75.1
GAS
60.6::00
81,8-70
91.6119
61.1-52
88.7+1.1
89.0=14
62.7-28
$7.5+44
69.920
72.1=88
73.5
DCI
61.033.11
89.3
89,4130
64.1
S8.01322
88.5
67.6
62.24255
65.32233
74.2-33
75.0
PCGNN
52.8+334
83.9+&11
93.2+112
65.1+448
92,0+111
87.5+144
67.4-2.1
59.0+-4.0
684442
69:1-24
73.8
GAT-sep
60.3+b4.4
87.8=577
914422
65.0=4455
$6.3+39
89.3-221
69.1=22.5
61,92422
69.0=19
74.5=237
75.5
BernNet
63.11:17
60.11669
92.1122
65.02337
91,2110
87.0.1177
61.9156
6181164
69.0114
59,8-433
73.1
AMNer
62.9+188
82,4+46
92.8+211
648+52
92.6+09
85.4+17
61.7+411
63.6+288
67.1+33
53.7-14
72.7
BWGNN
$7.745.0
93.6--4.1
91.8+23
64.3-14
92.1+277
88.7-13
68.54227
60.248.6
65.543.1
77.5-443
76.0
GHRN
57.52245
91.62444
90.91199
64.523.1
92.620777
89.0=213
69.02:22
60.51877
67.14300
78.7=30
76.1
RI-GGraph
61.4424
96.3111
92.5413
61.62227
9500007
93.9221
70.4#23
6A.7436
64.9332
66.6116
XGR-Graph
59.242.7
96.4007
94.7409
64.0+35
94.8+0.6
91.9+13
67.5+344
61.4+2.9
62.4+4.11
85.2-18
77.8
Rec
Reddi
Welbo
Amazon
Yolp
T-Fin.
Ellip:
Tolo.
Quest.
DGraph.
T-Social
MLP
6.5120
53.2155.2
79.3110
26.52227
59.921022
$1.5662
35.5126
12.012.1
3.41199
3.212.11
33.4
KNN
5.7116
46.4278
75.1
12.02311
47.9144
318454
31.422
8.3233
2.211117
5.7
27.6
SVM
64112
48.01...
61.81214
28.522.
18.1
266617
16.1116
11.213.
1.6100.5
166204
24.0
$61119
56.4266
73.1178
30.5218
69.111
770022
34.925
12,9199
2.81066
11.52:11
XGBoosi
50:19
56.8-4555
77.5+29
29.5=50
669123
72.1-25
33.1-28
11553.3
181199
9.2-3333
36.3
XGBOD
4.91177
59,3113.5
77.9155
30,2
66.91333
70.5-20
32.61225
11.3133
2.310777
6.8.1226
36,3
NA
4.6:119
61.81255
79.2114
31.02:48
67.1117
72.5
33.3-25
12.1138
2.11066
10.622.3
37.4
GCN
79.2+44.
369922
16.933
60
49.7 49.7-142
33.4-35
9,84122
3.6104
10.2-81
SGC
5.9129
64.1
33.6123
16.4222
35
26.9144
32.7
10.01144
2440
4.0
23.2
GIN
4.81199
66.51133
70.415.7
26.526.1
$1.1150
47.611.1
33.61130
10.311111
2.110.5
$.1129
32.2
GaptSAGE
5.81008
63/1160
48.01566
22.9236
1851194
35:2-22
8.812.5
2.5100.1
9,51229
26.3
GAT
6.51233
70.2244.
77.1111
28.4
36.2-1033
5L4258
35.1
10.91009
3.11
116630
GT
5.611.7
75.1133
70.244.7
26.8
24.111577
53.1
36.3116
10.7.1.
2.81100
11.5200
31.7
PNA
4.2119
78.714.1
40.4:3..1
17.5245
58.6114A
38.66:366
34.02222
8A122
2.2110.7
29.4
29.2
BGNN
5.612.11
79.71.3
43.24662
17.5211
72.9119
43.4
34.7
9.0
3.6111
58.241124
36.8
GAS
6.6225
62.0266
77.4117
24.624411
54.24955
519452
33.0
9.18229
3.404
11.5246
DCI
4.5114
68.5
68.3177
26.8
58.516.3
50.0..3.
33.5
9.91199
2.31
6.3.688
32.9
PCGNN
3.0:2.1
65.1
78.0+15
27,8
63.946.
46.5-733
34.3-11
10.14:39
3.7+-00
13.5:23.1
34.6
seP
5.8.122
71.1149
77.5425
27.2-42
43.6211.7
53.9169
35.2122
11.2119
3.41199
14.9229
34.4
BernNett
6.1111
60.9144
77.2121
26.8.1111
60.5111.1
17.044
30.1238
10.31.2.7
3.810.66
3.3
AMNet
6.8115
62.1
77.8123
26.62443
65.726.3
37.8-6.7
30.5119
12,75:25
2.61088
1.61005
32,4
BWGNN
6.0.114
75.1
77.7116
26.4232
64.921177
49.7.6.
35.5
10.91332
3.1.11
24.3
37.4
GHRN
631115
72/11:6
77.7113
26.923.1
67.714.3
50.813
36.1
11.1134
3/110.1
21.627.0
37.7
RF-Gimph
5.8+17
65.5+788
63.54455
24.3--13
74.5+3.2
72.143.3
38.-225
12,943.0
2.4004
49.0-57
40.8
XGB-
49-19
68.9-5.77
78.24115
26.8-3.11
72.4&33.
08.9-13.
36.6-330
10.6+229
2.5
43.0=766
41.3
24

---

Table 12: Comparison of AUPRC (top), AUROC (middle), and Rec@ K (bottom) for each model employing default hyperparameters in the fully-supervised setting. Each model is executed 10 times with varying random seeds, and the mean scores along with standard deviations are reported. Ave. denotes the average score across all 10 datasets.
AUPRC
Reddit
Weibo
Amazon
Yelp
T-Fin.
Ellipp
Tolo.
Quest
DGGraph.
T-Social
Ave.
MLP
6.0%11.1
818211
88.01.2.2
47.72:17
70.52:2
22.74:777
38.51:111
1522.10
2.71:0.0
14.727.
39.1
KNN
4.8:00
7.
78.2:00.0
31.5:0000
66.5216
30.520.0
31.8400.
14.72:10
1.3:0000
24.8:0.1
35.7
SVM
6.5+088
72.04-4.0
80.1+6.6
40.6+00.0
S8.D-174
19.++0.2
37.2+0.9
16.413
24++0.
3.740.8
33.7
5.3106
92.02:08
89.72.0.4
76.3:0.3
$0.6113
77.5104
37.711.1
14.31100
2.5000
41.010.11
51.7
XGBoosi
5.6:2077
95.1=007
91.5000.0
82.7=000
19.9
77440.0
38.1
15.100.9
2.700.0
15.0:00.
50.3
XGBOD
7,5.:144
95.02.04
$9.4
66.31004
$1.1206
75.910.0
413
18.411.7
2.010.00
8.2.0000
48.5
NA
5.7
92,92009
91.1.20.0
76.22000
79.221.3
78.6:0.00
39.111.3
15.6109
2.710.00
15.510..2
49.7
GCN
6,5+14
94.5-110
21.1-0066
73.9+52
21.942.
44.5422
1255+4
4.0+0.1
147777
52.9
SGC
4.6:208
92.34:16
33.4
15.92115
39.51:4455
12.4
36.0.577
10.0
2.4-0.1
5.241.66
25.3
GIN
5.7+0.9
90.6++13
79.2+1.5
36.5+15
64,2+46
23.5+49
196+18
12,7+122
3.5+0.1
10.1+16
36.5
GrapbSAGE
5.5+0.8
85,4+223
69.0=12.5
$4.0+-10
34,8=1099
32.96600
47.9+177
166+088
388+0.1
12.7433
36.3
GAT
5.9+100
11
46.7+11
434446
63.1=110
25.2456
15
3920.1
9.24200
39.0
GT
5.7+00.9
88.7+19
N1.4+400
55.0+554
31,6+1477
25.14455
476+19
159+14
3.9+0.1
9.5+2.11
36.3
PNA
5.4+0044
93.6+-088
42.5+-6.
30.5=00.
28,5=12.3
27,5++.3
42.3+1177
1199009
3.5+0.1
26.4+7.3
31.2
BONN
7,74066
96.24008
26.9t:177
82.6+0099
64.24466
98.7+002
49.5
GAS
7.141.77
92.9200
48.9:1..11
22.3
76.9
27.916.6
45.7
14.8&1.9
3.8110.1
9.3.2244
35.0
DCI
6.12:09
89.6.:1.
81.51:222
39,5
62.61117
15.414.7
39.9113
14.111.
3.610..1
13.8.1400
37,6
PCGNN
4.2+0.5
81.9=1177
87.8
43.7=:26
69.8
10.
38.1+2.1
14.4409
2.80000
8.7424
38.7
SAP
6.1.1..
90.91121
$7.1
$4.9128
70.9=1199
26.414.5
46,5122
16.5110
39F0-1
10.4119
41.3
BerNNet
7.12:12
89.72:19
86.12:277
50.1
72.11699
21.6129
13.5.11.3
152211
2910.0
5.1.1.0
39,4
AMNet
7.34:1.0
89.7=222
87.3+2211
1..8=113
74.3=22
14.7+29
43.2+122
14.+112
2.8
3.1
38.6
BWGNN
6.91166
93.0.114
19.12116
55.12116
86.6200
26.0.1.5
49.7119
16.711.3
4.0.0.11
54,9116.5
48.2
GHRN
7,2+1.7
91,8++12
89.5+122
56.6+1.7
86.+11
27.7+666
49.942.1
16.741.2
4.0+0.1
16.3+4.5
44.6
RP-Graph
54404
97.0+033
91.3+0011
76.7+05
89.0009
77.8+0.3
49.3+116
14.2+12
2,0+0.0
93.8+0.1
59.6
XGB XGa-Graph
5.6200
97,5#006
92.6:00.0
87,02044
89,620.8
78.2:0.00
48.9112
15.340.
3,74000
92.410.1
61.1
AUROC
Reddit
Weibo
Amazon
Yelp.
T-Fin.
Ellip.
Tolo.
Quest.
DGraph.
T-Social
Ave.
MLP
65.5+228
90
97.3++1.7
82.0+0.7
91,4+1.1
84. 242.3
73. 73.34-0.7
69. 3+2.3
72.5+0011
80.247.3
56.5+1.2
88.6+110
90.2+0000
75.3+00
86.6+1.
67.5+-19
6100+11
514400.0
73.0+0.1
75.1
SVM
6.
84.8230
3.
71.2:00.0
91
84.2:10.2
72.24:0.6
66.72:18
72.000.0
58.71:5.6
77.0
RF
62.0+-28
98.7+02
97.2+003
93,3+0.1
94.2+0.5
901+0099
71.7+008
57.6116
69.1+0.11
79.0+00.1
81.3
XGRoost
61.6+-355
99.1+0.1
97.9+0.0
95.2=000
94,4+04
91.2400.0
73.0+-0.
57.8+14
714+000
41400.1
82.3
XGBOD
68.62:16
99.1=0011
98 3:.33
8944:044
95.0003
93.0+0.00
75.24:08
69.3
64.910.00
69.2000.0
82.2
NA
62.4+132
98.4+0.3
97.5++0.0
94.0+0.0
93.9+0.3
92.0+0.00
73.5++0.9
59.3+116
71,8+0.0
81 4++0.1
82.5
OCN
65.21:48
98.01005
62.41.0.4
57.8100.6
92.412.3
$141188
75.6.1..2
70,911..
75,9100.
80.217..3
78.0
SGC
$1.51:4.2
97.920.4
80.3::200
$4.11224
76.62:3.8
75.111.2
68.3144
71001..1
66.110.3
66.1118.7
71.0
GIN
65.51:32
95
93.8
77.0=009
S8.21442
82.7420
15.1
694113
74.0
78.41:2.22
30.0
GrapbS.AGE
62.81:511
94
69.61.1.4
$5.2
$2.8119
85.310077
7931.99
72.1
75.6100
79.2.1-.0
$0.7
GAT
65,34-3.1
95.3=1144
96.7=110
79,5=119
92.8115
849H1.9
79.1+11
711116
75.94-12
75.4448
81.6
63.72444
95.5
92.4
$4.5122
$1.465
15.1
79.61:1
70.91122
85.8
72.113..
80.1
PNA
64.6+115
97,8+005
83.2+-4.5
74.3+0.
74.3-12.0
84.0+1.0
75.+-10
71.8+0.99
73.4-022
78.2++110
77.7
BGNN
72.2
98.8=0033
92.3+-0.5
64.6+:335
95.6+004
90.1
74.4++199
62.5+11.2
68.5+11.3
99.940000
81.9
GAS
67.515.3
97.52005
16.15
59.1::0.5
93.3121.
85.6111.
77:110
69.41.
76.01:022
76.91:36
78.9
DCI
66.52:13
94
6-1-0.9
7788=7888
86.8+:455
82.8+1.5
75.++
69-24-1.3
74.7+-0.1
80.660
S0.3
PCGNN
53.21:2.1
90.21:1.5
97,310088
79.7
93.311.0
85.811.
72.8.2.0
69.911144
72.0100.3
69 2114.44
78.3
GATS<p
66.1::26
96.3+112
97.0200.5
84.3:0088
93.5:22.
860214
79.11.
69.11999
76.0:1022
74.51:3.7
$2.3
BernNet
66.8--3.7
1.
96.2+11.4
83.0=066
92
15.1-14
769946
68 84299
73.240.11
81122
AMNet
68.11:2.4
9..3
97.011.6
82.61055
93.71007
77-34255
768197
68.1119
73.119.1
53.6133.33
78.6
BWGNN
65.42..3
97.3:099
98.02007
84.92007
96.5
85.221.1
804409
71,8122
76.310.11
92.01552
84.7
GHRN
66.01.4.5
96.71-1.1
98.1.0.3
85.31006
96.010.88
8541199
804108
71.1199
76.1
79.0112
83.5
RF-Graph
62.81:24
99.52:01
97,52033
93.31002
96.7103
919105
812407
65.81122
65.61n3
99.21:00
85.4
GGB--Graph
6288-32
99.5+=01
98.7+0.0
96.2+022
96.8+0.4
93.400.0
80.1+0.5
65.7+0.88
74.6+00
99.3++0.0
86.7
Rec@K
Redda
Waibo
Amazon
Yelp
T-Fin.
Ellip
Tolo.
Ques.
DGraph.
T-Social
Ave.
MLP
7,5229
1
$4.4
46.9 46.91:13
67.8226
23.91116
19.11-11
19331177
4.24022
20.19:119
39.3
KNN
9.31116
70.32:16
78.810.0
29.4100
68.111.
44.5.0.0
35.1144
174116
18100
34.10022
39,0
SVM
80=26
60
75.1+78
41,7-00
60.6-1311
15.2407
370+13
194+18
4.2022
2,111.1
33.2
RF
5.911.5
22.32-1
85.71..
69.4
74.
72.50.33
39.410
179415
4.20000
45.3400.1
49.8
XGBoost
7,5114
87.0
$6.410.0
74.8-000
72.9
72.01000
40.111.5
17.911.1
4.1100
20.01112
48.3
XGBOD
10.9:39
88.0=05
83.8+220
61,0=004
74.6+0.8
69.7410.
42.24-1.5
18.5+1.3
1.+000
140000
46.4
6.921.4
86.62:1.4
$7.5
73.1
73.2
72.600
40.34:1.2
18.8133
4.4.00.0
20.3
48.4
GCN
1000:3.2
90.3
36.93 1.
23.7:006
70.0E5.2
25.0:6600
16.5+220 43.1
7.1+002
20.11:109
34.3
SGC
7,8+29
88.2+114 1.4
35.2+443
19,7+2.6
41,1+233
11.0+1.7
35.0+34
16.1+112
4.2+0.2
4.1+2.7
26.2
GIN
6.81:244
1.
73.01:3.5
3842:12
65.51A8
27.318.3
39.01:1
17.511.7
5.110.2
14,435
37.5
GaPPSAGE
7.22:19
83.--2.
67.9
$1.3=2.1
46.0=119
37.54
4656-1
21.2::16
70404
156:144
38.4
GAT
6.91:2.
87.8
1..13
43.61:177
61.6115
27.91113
16.6.18
19.811.6
7/1102
11.6130
39.9
GT
7.72228
13
77.8128
51.12:322
19.81119
26.311111
47.118
19.71177
7.511022
13.2111.1
37.4
PNA
6.1.222
89.3
48.81.6
31.41006
36.5
23.515.8
42.411.2
17.018
6.6.0.3
36.5
34.8
BGNN
10.72:16
91.02:99
63.91228
27.51:14
77,31088
63.811.3
43.7.1..
8.611.5
6.210.4
95.410.3
48.8
GAS
9.9+443
86.6+11.5
49. 49.5++6.4
24 13.16
72.9+1.7
34.6++9.6
45.0+1
18:2+2.3
6.8+022
10.4+443
35.8
DCI
7.36211
85.5
76.764.0
40.3+:7.1
64.0+5.3
29.956.2
40:24:00
18.5414
6.740.3
20.6:7722
39.0
RUGNN
5.0+1.4
75.9--2.4
N3.2+-2.5
444-1.3
69.+55
40.+1120
37.6+200
185+-19
5.00022
14.5+499
39.4
GATSep
7.6+299
87.9+-288
NA.D--0.8
51.1+14
70.3+99.
31,3+888
4664220
20.0+118
7,5+0.3
149429
42.1
BernNel
10.5:224
85.02-2.3
83.2:12.9
48.7:00.
71.955.5
2399777
43.5t:1
19.3+2.1
4,6:0.11
60116
39.7
AMNei
10,3+2.9
65.3--2.
13.4+1.6
47,7=114
73.1+114
42
19942.1
4.2+0.1
1.6:--.5
BWGNN
10.7+288
87,4+117
85.0+227
52,2+1.3
81.1+07
31.7+62
48.742.1
21.4700
7,5+0.3
54.3+1244
48.0
GHRN
11.61:3.2
86.72:18
1.6
53.02:10
81.11114
33.3
49.11:15
21.3422
7.51102
24.6:17.0
45A4
RF-Graph
5.81114
92.2-007
84.4
69.6::0.6
84.0
72.510.3
$1.34166
184424
3.210.1
87.6410.1
56.9
XGB-Graph
7.621.1
93.1.:1.1
85.910000
78.8100.6
$4.611.1
71240.0
48.811..
16.511.3
6.7.0.0
$5.210.1
$7.8
25

---

Table 13: Comparison of AUROC (top) and Rec@ K (bottom) of each model with optimal hyperpa- rameters through random search. The Results for AUPRC is in Table Ave. signifies the average score across the first datasets without T-Social. OOT (out-of-time) means the model could not complete random search within day. Best results are highlighted in bold.
AUROC
Reddit
Yelp.
Kllip.
Quest.
IXGraph.
Ave.
MLP
60.71
92.89
96.94
81.64
92.24
88.32
73.78
60.15
72.34
73.06
79.89
KNN
62.53
93.57
94.91
84.60
92.69
88.03
72.09
64.00
58.50
77.64
78. 78.99
SVM
68.18
92.52
95.80
77.21
92.59
85.02
72.59
64.77
72.03
OOT
80.08
60.79
98.77
97.53
93.80
94.85
91.68
74.14
70.37
79.94
81.87
XGBoost
64.59
98.82
98.34
95.38
95.42
89.91
74.84
62.09
72.43
81.75
83.53
XGBOD
68,39
99.24
98.20
93.96
95.39
92.26
74.55
64.88
OOT
83.82
63.86
99.03
98.23
94.70
94.85
91.27
74.87
66.20
71.73
81.95
83.86
GCN
62.04
98.84
85.17
$8.62
94.62
81.69
73.80
68.20
75.51
96.63
77.61
SGC
66.95
98.58
87.84
$7.66
87.97
79.52
72.36
68.11
68.63
83.67
76.40
GIN
61.82
98.65
95.63
73.77
92.71
83.01
74.57
68.08
74.16
94.09
80.27
GraphSAGF:
63.28
97.75
90.90
82,90
95.62
87.59
80.92
73.04
75.60
95.73
83.07
GAT
64.15
98.54
97.13
79.14
95.75
86.28
78.75
68.30
75.53
90.33
82.62
67.39
97.23
93.37
80.32
94.39
86.10
78.94
69.62
75.71
90.79
82.56
PNA
63.45
98.78
82.14
74.39
92.47
84.86
75.17
66.32
73.39
84.02
79.00
BGNN
68.94
90.64
63,28
96.20
87.69
76.65
68.02
76.22
99.94
80.69
KNNGCN
60.31
99.17
90.69
69.13
94.69
88.53
74.23
66.95
75.72
87.80
79.94
GAS
60.57
99.11
92.69
76.43
96.45
86.66
78.19
68.15
75.98
94.96
81.58
DCI
66.87
97.94
95.39
78.38
87.85
85.73
73.86
64.77
73.94
83.96
80.53
PCGNN
65.41
95.14
98.01
80.82
94.03
86.50
76.63
67.66
72.76
96.91
81.88
GATSep
66.17
98.22
95.11
80.49
94.40
85.94
79.52
70.08
75.78
87.69
82.85
BemNet
69.83
98.00
95.79
83.54
96.67
87.54
77.51
68.58
73.58
93.73
83.45
AMNet
69.60
98.32
96.92
81.90
96.38
83.66
75.16
66.05
72.99
92.50
82.33
BWGNN
70.82
98.13
98.27
87.13
96.93
87.03
80.41
70.87
76.30
88
85.10
GHRN
61.02
99.18
98.29
84.60
89.50
80.08
72.16
76.13
97.12
84,16
RFGraph
65.35
99.43
96.73
95.24
97.28
93.21
81.88
64.77
67.78
99.69
84.63
XGBGraph
64.74
99.29
98.74
97.37
97.15
91.78
82.85
71,02
75.83
99.76
86.53
Rec@K
Reddit
Waibo
Amazon
Yelp.
T-Fin.
Ellip.
Tolo.
Quest.
DGraph.
T-Social
Ave.
MLP
9.52
78.39
83.15
46.62
68.93
57.43
39.88
16.99
4.04
16.86
44,99
KNN
10.20
73.20
79.35
51.92
70.04
56.60
37.85
19.18
1.98
44.08
44.48
SVM
8.84
77.81
82.07
41.77
72.40
19.21
39.72
18.08
4.04
OOT
40.44
RF
5.44
84.15
86.41
70.23
75.59
72.76
39.72
17.81
4.21
45.48
50.70
XGBoost
6.80
87.03
86.41
75.08
76.01
72.58
41.90
16.71
4,30
21.86
$1.87
XGBOD
10.20
87.32
86.96
71.62
75.87
69.07
42.21
16.71
1.68
OOT
$1.29
NA
8.84
85.59
87,50
73.38
75.31
71.74
42.21
18.36
4.04
21.70
$1.89
GCN
6.12
88.47
44.02
23.85
74.90
33.52
39.41
17.81
7.05
73.23
37.24
SGC
8.84
87.32
46.20
22.46
67.41
22.99
39.72
16.99
3.87
24.93
35.09
GIN
10.88
89.63
80.98
36.15
73.37
32.13
39.41
18.36
6.32
64.47
43.03
GraphSAGE
7.48
90.78
78.26
47.15
78.09
56.69
48.75
21.37
6.84
73.74
48.38
GAT
10.88
89.63
82.61
44.23
79.75
37.86
44.24
17.26
7.14
42.07
45.95
GT
12.93
85.88
78.80
44.62
81.55
30.75
44.39
20.27
6.92
43.58
45.12
PNA
10.88
39.67
90.78
30.00
72.54
36.47
42.68
13.15
5.16
26.95
37.93
BGNN
9.52
93.37
64.67
30.77
77.25
59.56
45.33
12.60
7.70
96.89
44.53
KNNGCN
5.44
89.63
71.74
33.23
75.87
41.55
40.34
18.08
7.18
48.67
42.56
GAS
4.08
91.93
80.43
38.00
79.75
37.49
47.04
18.90
6.02
64.58
44.85
DCI
8.16
89.05
80.98
40.46
71.98
35.27
37.85
17.26
5.85
18.27
42.99
PCGNN
14.97
84.15
85.33
43.77
79.06
43.77
43.15
18.08
6.66
73.53
46.55
GATSep
9.52
88.18
80.98
44.23
81.55
30.56
47.20
21.37
7.31
40.84
45.66
BemNett
10.88
89.34
82.61
49.69
83.63
49.77
44.70
19.73
5.55
48.23
48.43
AMNet
11.56
88.76
83.15
45.38
84.05
30.56
42.52
17.53
4.21
43.21
45.31
BWGNN
11,56
87.90
85.87
56.69
84.19
42.47
50.31
21.64
7.57
75.78
49.80
GHRN
5.44
89.34
85.33
51.85
81.97
50.51
46.57
22.19
6.96
82.33
48.91
RFGraph
3.40
91.93
83.15
75.31
84.05
72.58
52.18
15.89
3.22
93.58
53.52
XGBGraph
6.12
91.93
85.87
83.15
85.02
71.93
53.43
20.55
6.96
93.53
56.11
26


Reviews:
[{"review_id": "hXGSdO6Hpx", "reviewer": "Reviewer_7otD", "summary": "", "questions": "", "limitations": "The experimental results are also well described and convincing. However, the authors are suggested to consider more types of datasets and give more motivations regarding datasets selection.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "+ This paper formalizes a very interesting problem, i.e., graph anomaly detection. \n\n+ The paper is well-written and easy to follow.\n\n+ The datasets covered in this paper include 10 diverse topics.\n\n+ The authors provide complete experiments and analysis.\n ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "01i6ltEdva", "reviewer": "Reviewer_jFM3", "summary": "", "questions": "", "limitations": "1. Lack of baseline models. As this paper is a benchmark for GAD models, I would expect a comprehensive evaluation of existing models. But I find that this benchmark lacks some popular baseline models in the experiment, like Care-GNN[1], GraphConsis[2], FRAUDRE [3], H2-FDetector [4], to name a few. \n\n2. Findings lack of novelty. I find that some of the findings in section 4.1 is not informative. For example, the finding that most standard GNNs prove unsuitable for GAD is repeatedly mentioned in existing GAD works. Also, the finding that Specialized GNNs require hyperparameter tuning to achieve satisfactory performance is a common sense in the machine learning community. I would expect authors can derive more valuable findings based on experiments in this benchmark.\n\n3. Simplification of dataset causes information loss. Authors simply treat all datasets as static homogeneous graphs, ignoring temporal and heterogeneous information. That can bias the testing results as some baselines are specifically designed to utilize these kinds of information. In this way, we cannot get a fair evaluation on these models. \n\n[1] Yingtong Dou, et al. 2020. Enhancing graph neural network-based fraud detectors against camouflaged fraudsters. CIKM.\n\n[2] Liu, Zhiwei, et al. 2020. Alleviating the inconsistency problem of applying graph neural\nnetwork to fraud detection. SIGIR.\n\n[3] Ge Zhang, et al. 2021. FRAUDRE: fraud detection dual-resistant to graph inconsistency\nand imbalance. ICDM.\n\n[4] Fengzhao Shi, et al. 2022. H2-FDetector: a gnn-based fraud detector with homophilic and heterophilic connections. WWW.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1.  The finding that a tree ensemble with neighborhood aggregation can deliver the best performance is interesting and worth further investigations.\n\n2. GADBench is available as an open-source tool, which can help researchers test their model and keep track of current research frontier.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "97ryNxJ9W7", "reviewer": "Reviewer_qCKR", "summary": "", "questions": "", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper is highly original. It presents a comprehensive benchmark for supervised anomalous node detection on static graphs, which is an important and challenging problem in the field of graph analysis. The authors provide a thorough evaluation of traditional Graph Anomaly Detection algorithms and Graph Neural Networks, and compare their performance and efficiency on large-scale graphs. The significance of the paper is high, as it provides important insights into the current advancements of GAD and establishes a solid foundation for future research.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "O9tN5r9mAU", "reviewer": "Reviewer_eNf8", "summary": "", "questions": "", "limitations": "yes", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "Motivation is good.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "5qVKmVdnaZ", "reviewer": "Reviewer_VEyk", "summary": "", "questions": "", "limitations": "In Section 2, the task definition can be more general. The adjacency matrix may not necessarily be unweighted (e.g., multiple edges between two nodes). Also, the description seems to be transduction. Inductive settings are not considered and are not mentioned in the paper.\n\nIn Figure 1, the lower lines of tree ensembles with neighbor aggregation are not significantly higher than other methods. It indicates the instability of the tree ensembles.", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper is overall well-organized and easy to follow.\n\nThe evaluation is comprehensive, comparing 23 methods on 10 different datasets. It is valuable for both researchers and practitioners in the future. \n\nThe paper proposes to combine tree ensembles and neighbor aggregate, which is proven effective in the experiments. \n\nSection 4.2 provides insightful visualization of the decision boundary of different approaches and explains the reason why tree ensembles are effective.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Thank you for submitting your paper to NeurIPS.

The reviewers find your proposed datasets valuable for important applications. They appreciate the extensiveness of both the datasets and the experiments conducted, as well as your efforts to make the datasets easily accessible.

However, the reviewers have raised some concerns, which have not been fully addressed during the rebuttal period:
- Comparison with additional competitors
- Discussion of additional relevant studies.

Overall, all reviewers have a positive view of the paper and lean towards its acceptance. I believe that the additional concerns can be adequately addressed in the camera-ready version. Therefore, I recommend accepting the paper.
Decision: Accept (Poster)