SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Reddit [42]: This dataset contains a user-subreddit graph, capturing one month's worth of posts shared across various subreddits. Verified labels of banned users are included. The dataset focuses on the 1,000 most active subreddits and the 10,000 most engaged users, leading to a total of 672,447 interactions. Posts were transformed into feature vectors, each representing the Linguistic Inquiry and Word Count (LIWC) categories of the text.
Weibo [42]: This dataset features a graph of users and their associated hashtags from the Tencent-Weibo platform, consisting of 8,405 users and 61,964 hashtags. Suspicious activities are defined as two posts made within specific timeframes, such as 60 seconds. Users that engaged in at least five such activities are labeled as "suspicious", while the rest are categorized as "benign". This process yielded 868 suspicious and 7,537 benign users. The raw feature vector is composed of the location of a micro-blog post and bag-of-words features.
YelpChi [66]: This dataset is designed to identify anomalous reviews that unfairly promote or demote products or businesses on Yelp.com. The graph includes three types of edges: R-U-R (reviews posted by the same user), R-S-R (reviews for the same product with the same star rating), and R-T-R (reviews for the same product posted in the same month).
Amazon [56]: The goal of this dataset is to identify users paid to write fake reviews for products in the Musical Instrument category on Amazon.com. The graph includes three types of relations:
Table 7: Overview of datasets in GADBench with their corresponding node feature types, feature dimension, and detailed descriptions.
Dataset	Node Feature Type	#Dim.	Detailed Feature Description
Reddit	Text Embedding	64	LIWC text embedding for posts
Weibo	Text Embedding	400	Bag-of-words features from posts
Amazon	Misc. Information	25	Hand-crafted user features and statistics
YelpChi	Misc. Information	32	Hand-crafted review features and statistics
Tolokers	Misc. Information	10	User profile with task performance statistics
Questions	Text Embedding	301	FastText embeddings for user descriptions
T-Finance	Misc. Information	10	User profile details such as registration days
Elliptic	Misc. Information	166	Timestamps and transaction information
DGraph-Fin	Misc. Information	17	Timestamps and user profiles details
T-Social	Misc. Information	10	User profile details such as logging activities
U-P-U (users reviewing at least one same product), U-S-U (users giving at least one same star rating within one week), and U-V-U (users with top-5% mutual review similarities).
T-Finance [74]: This dataset aims to find the anomaly accounts in transaction networks. The nodes are unique anonymized accounts with 10-dimension features related to registration days, logging activities and interaction frequency. The edges in the graph represent two accounts that have transaction records. Human experts annotate nodes as anomalies if they fall into categories like fraud, money laundering and online gambling.
Tolokers [65]: This dataset is derived from the Toloka crowd-sourcing platform. Nodes represent workers who have participated in at least one of 13 selected projects, and an edge connects two workers if they worked on the same task. The task is to predict which worker has been banned in one of the projects. Node features are based on worker's profile and task performance statistics.
Questions [65]: This dataset is collected from the question-answering website Yandex Q. Nodes are users, and an edge connects two users if one user answered the other user's question during a one-year period (September 2021 to August 2022). The dataset focuses on users interested in the topic "medicine". The task is to predict which users remained active on the website at the end of the period. Node features are the mean of FastText embeddings for words in the user description, with an additional binary feature indicating users without descriptions.
Elliptic [83]: This dataset includes a graph of over 200,000 Bitcoin transactions (nodes), 234,000 directed payment flows (edges), and 166 node features. The dataset maps Bitcoin transactions to real-world entities associated with licit categories, such as exchanges, wallet providers, miners, and legal services, as well as illicit categories like scams, malware, terrorist organizations, ransomware, and Ponzi schemes.
DGraph-Fin [36]: This dataset is a real-world, large-scale dynamic graph provided by the Finvolution Group, representing a social network within the financial industry. In DGraph-Fin, a node represents a Finvolution user, and an edge between two users indicates that one user lists the other as their emergency contact. Anomalous nodes in DGraph-Fin represent users who exhibit overdue behaviors. The dataset comprises over 3 million nodes, 4 million dynamic edges, and more than 1 million extremely unbalanced ground-truth nodes.
T-Social [74]: This dataset aims to find the anomaly accounts in social networks. It has the same node annotations and features as T-Finance, while two nodes are connected if they maintain the friend relationship for more than three months. Same as T-Finance, human experts annotate nodes as anomalies if they fall into categories like fraud, money laundering and online gambling.
C Other Information in GADBench
C.1 Metrics
AUPRC (Area Under the Precision-Recall Curve). AUPRC is a metric that evaluates the performance of classification models by computing the area beneath the Precision-Recall curve. This curve
illustrates the relationship between precision (i.e., the ratio of true positive predictions to all positive predictions) and recall (i.e., the ratio of true positive predictions to all positive labels) at different threshold levels. AUPRC can be calculated by the weighted mean of precisions at each threshold, where the increase in recall from the previous threshold serves as the weight.
AUROC (Area Under the Receiver Operating Characteristic Curve). It evaluates a model's ability to discriminate between positive and negative classes by measuring the area under the ROC curve. The ROC curve plots the true positive rate against the false positive rate for varying decision thresholds. An AUROC of 1 indicates perfect discrimination, while an AUROC of 0.5 suggests that the model is no better than random guessing.
Rec@K (Recall at  $k$ ). It is determined by calculating the recall of the true anomalies among the top- $k$  predictions that the model ranks with the highest confidence. We set the value of  $k$  as the number of actual outliers in the test dataset. It is noteworthy that in this specific scenario, Rec@K is equivalent to both precision at  $k$  and the F1 score at  $k$ .
Runtime. To assess the efficiency of the various models, including both traditional algorithms and neural networks, we measure the runtime as the duration from the beginning to the completion of the experiments. This measurement does not distinguish between computation times on CPUs and GPUs.
Memory. We record the peak utilization of both CPU and GPU memory during the entire supervised training phase for each algorithm on the DGraph-Fin dataset. This metric is crucial for assessing the computational resource requirements of each model in practical implementations.
C.2 Additional Experimental Details
Implementation Details. To ensure a comprehensive evaluation and maintain fairness across a broad spectrum of models, we develop an open-source toolkit named GADBench<sup>3</sup>. This toolkit is built on top of Pytorch 1.12 [60] and DGL 1.0 [80]. We implement all standard and specialized GNNs in GADBench using the DGL library. For classic models KNN, SVM, and RF, we use the implementations provided in the Scikit-Learn library [62]. XGBoost is integrated using its official implementation [16], and XGBOD is included via the PyOD library [94].
Hardware Specifications. All our experiments were carried out on a Linux server equipped with an AMD EPYC 75F3 32-Core CPU processor, 64GB RAM, and an NVIDIA RTX A6000 GPU with 48G memory.
Hyperparameter Settings. Tables 8 and 9 provide a comprehensive list of all hyperparameters used in our random search, complete with their default values and respective distributions or search spaces. For all configurations, we retain the model that yields the best AUPRC score on the validation set and report the corresponding test performance. Due to limited data, we did not perform hyperparameter search for heterogeneous graph neural networks (RGCN, CARE-GNN, and H2-FDetector).
New Models and Datasets. GADBench is highly extensible due to its seamless integration with the DGL library. Datasets structured in the dg1 . graph format, accompanied by binary node labels, can be effortlessly integrated and leveraged by all models within GADBench. In a similar vein, models that accept dg1 . graph as input and yield an anomaly score for each node can be smoothly assessed across all datasets in GADBench. We are committed to the ongoing maintenance of GADBench and will persist in incorporating new models and datasets.
In accordance with Figure 1, Tables 11 and 12 provide the performance metrics of each model under semi-supervised and fully-supervised settings, respectively, when default hyperparameters are employed. Additionally, Table 13 consolidates the information in Table 4 and exhibits the performance of the models in terms of AUROC and Rec@K scores subsequent to hyperparameter tuning for each dataset.
In table 10, we evaluated the effect of incorporating Neighborhood Averaging (NA) on the performance of four representative graph-based methods in GADBench, namely GIN, BWGNN, XGB-Graph, and RF-Graph. In the semi-supervised setting, all of the techniques demonstrate an im
<sup>3</sup>https://github.com/squareRoot3/GADBench
Figure 4: More decision boundaries comparison examples as a complement to Figure 3. Blue points represent anomalies while red points are normal nodes. Similarly, the blue/red regions correspond to model predictions for anomalous/normal classes.
provement in performance subsequent to the integration of NA. For example, the AUPRC score is improved by  $1.4\% \sim 3.7\%$ . In the fully-supervised setting, NA can enhance weaker baselines like GIN. However, for stronger baselines, the performance remains similar (on BWGNN) or might even decrease (on XGB-Graph and RF-Graph).
E More Details About Plotting Decision Boundaries
As GNNs require graph structure as the input, it is not feasible to directly illustrate their decision boundaries. In the left panel of Figure 3, we use GIN removing the linear layer to embed nodes into the hidden space, following which we employ t-SNE [75] to reduce node embeddings to two dimensions. We then apply MLP and Random Forest to classify these node embeddings, approximating a comparison between GIN and RF-Graph. It is observed that the embeddings of normal and anomalous nodes are entwined, making them hard to separate. Unfortunately, GIN tends to produce simple and smooth decision boundaries, leading to recurring misclassification of normal nodes in the right bottom corners, thus compromising model performance. Conversely, XGBoost can produce more intricate decision boundaries, demonstrating greater proficiency in distinguishing anomalous data.
The right panel of Figure 3 showcases a trained BWGNN model on the Weibo dataset, visualizing the node embeddings preceding the final MLP layer. Employing t-SNE again to reduce the node embeddings, we compare the decision boundaries of MLP and XGBoost on these embeddings, simulating a comparison between BWGNN and XGB-Graph. As shown in the right panel of Figure 3, the embeddings of anomalous nodes are grouped into several dispersed clusters after training. However, the MLP model struggles with accurate classification due to the lack of a simple boundary between normal and anomalous data. In contrast, XGBoost successfully classifies anomalies within each cluster. In Figure 4, we extend our visualization to include two additional datasets, T-Finance and Tolokers. This serves as a complement to Figure 3, and the observations remain consistent.
Table 8: Default hyperparameters and random search space for MLP and GNN models. Elements in '[,' is randomly selected with equal probability during each trial. RandInt(a,b) returns an random integer between  $a$  and  $b$  (both included).
Model	Hyperparameter	Default value	Distribution / Search Space
Shared hyperparameters for all neural networks
Common	learning rate	0.01	10Uniform(-3,-1)
dropout rate	0	[0,0.1,0.2,0.3]
hidden dimension	32	[16,32,64]
epochs	100	-
Specific hyperparameters for each model
MLP	layers	2	[1,2,3,4]
activation	ReLU	[ReLU, LeakyReLU, Tanh]
GCN	layers	2	[1,2,3]
activation	ReLU	[ReLU, LeakyReLU, Tanh]
SGC	number of hops	2	[1,2,3,4]
activation	ReLU	[ReLU, LeakyReLU, Tanh]
MLP layers	1	[1,2]
GIN	layers	2	[1,2,3]
aggregation	mean	[sum, mean, max]
activation	ReLU	[ReLU, LeakyReLU, Tanh]
GraphSAGE	layers	2	[1,2,3]
aggregation	mean	[mean, GCN, pool]
activation	ReLU	[ReLU, LeakyReLU, Tanh]
GAT	layers	2	[1,2,3]
attention heads	4	[1,2,4,8]
GT	layers	2	[1,2,3]
attention heads	4	[1,2,4,8]
PNA	layers	2	[1,2,3,4]
activation	ReLU	[ReLU, LeakyReLU, Tanh]
BGNN	depth	6	[4,5,6,7]
iteration per epoch	10	[2,5,10,20]
GBDT learning rate	0.1	10Uniform(-2,-0.5)
normalize features	False	[True, False]
activation	ReLU	[ReLU, LeakyReLU, Tanh]
GAS	layers	2	[1,2,3,4]
number of neighbors	5	RandInt(3,50)
distance function	cosine	[euclidean, cosine]
DCI	layers	2	[1,2,3,4]
pretain epochs	100	[20,50,100]
number of clusters	2	RandInt(2,30)
PC-GNN	layers	2	[1,2,3]
under-sample ratio	0.7	Uniform(0.01,0.8)
over-sample ratio	0.3	Uniform(0.01,0.8)
distance function	cosine	[euclidean, cosine]
GAT-sep	layers	2	[1,2,3]
attention heads	4	[1,2,4,8]
BernNet	MLP layers	2	[1,2]
orders	2	[2,3,4,5]
AMNet	layers	3	[1,2,3]
orders	2	[2,3]
activation	ReLU	[ReLU, LeakyReLU, Tanh]
BWGNN	layers	2	[1,2,3,4]
MLP layers	2	[1,2]
activation	ReLU	[ReLU, LeakyReLU, Tanh]
GHRN	layers	2	[1,2,3,4]
MLP layers	2	[1,2]
deletion ratio	0.015	10Uniform(-2,-1)
Table 9: Default hyperparameters and random search space for non-deep learning models. Elements in '[,' is randomly selected with equal probability during each trial. RandInt(a,b) returns an random integer between  $a$  and  $b$  (both included).
Model	Hyperparameter	Default value	Distribution / Search Space
SVM	weights	uniform	[uniform, distance]
L2 regularization C	1	10Uniform(-1,1)
RF	number of estimators	100	RandInt(10,200)
split criterion	gini	[gini, entropy]
max samples	1	Uniform(0.1,1)
XGBoost	number of estimators	100	RandInt(10,200)
learning rate η	0.3	0.5 * 10Uniform(-1,0)
L2 regularization λ	1	[0,1,10]
subsample rate	1	[0.5,0.75,1]
booster	gbtree	[gbtree, dart]
XGBOD	Same as XGBoost
NA	Same as XGBoost, with 1 additional hyperparameter:
number of neighbors	5	RandInt(0,50)
RF-Graph	Same as RF, with 2 additional hyperparameters:
aggregation layers L	2	[1,2,3,4]
aggregation function	mean	[sum, mean, max]
XGB-Graph	Same as XGBoost, with 2 additional hyperparameters:
aggregation layers L	2	[1,2,3,4]
aggregation function	mean	[sum, mean, max]
Table 10: The impact of introducing neighborhood averaging (NA) on the performance of four representative methods in GADBench.
Model	Semi-Supervised	Fully-Supervised
AUROC	AUPRC	Rec@K	AUROC	AUPRC	Rec@K
GIN with NA	73.61	30.26	32.14	79.97	36.54	37.52
74.78	33.91	34.18	80.43	39.03	40.01
BWGNN with NA	76.00	35.42	37.36	84.73	48.19	48.01
76.95	36.96	37.34	84.70	47.93	48.08
XGB-Graph with NA	77.74	42.91	41.29	86.71	61.09	57.82
78.41	44.30	42.07	86.05	59.58	57.41
RF-Graph with NA	78.94	43.37	40.83	85.35	59.65	56.90
79.56	45.14	42.28	85.17	58.54	55.33