Unifying Unsupervised Graph-Level Anomaly Detection and Out-of-Distribution Detection: A Benchmark

## Abstract

Abstract To build safe and reliable graph machine learning systems, unsupervised graph-level anomaly detection (GLAD) and unsupervised graph-level out-of-distribution (OOD) detection (GLOD) have received significant attention in recent years. Though those two lines of research indeed share the same objective, they have been studied independently in the community due to distinct evaluation setups, creating a gap that hinders the application and evaluation of methods from one to the other. To bridge the gap, in this work, we present a U nified B enchmark for unsupervised G raph-level O OD and anoma L y D etection (UB-GOLD), a comprehensive evaluation framework that unifies GLAD and GLOD under the concept of generalized graph-level OOD detection. Our benchmark encompasses 35 datasets spanning four practical anomaly and OOD detection scenarios, facilitating the comparison of 16 representative GLAD/GLOD methods. We conduct multi-dimensional analyses to explore the effectiveness, generalizability, robustness, and efficiency of existing methods, shedding light on their strengths and limitations. Furthermore, we provide an open-source codebase ( https://github.com/UB-GOLD/UB-GOLD ) of UB-GOLD to foster reproducible research and outline potential directions for future investigations based on our insights.

## 1 Introduction

With the ubiquity of graph data, graph machine learning has been widely adopted in various scientific and industrial fields, ranging from bioinformatics to social networks [; ].
As one of the representative graph learning tasks, graph-level anomaly detection (GLAD) has been widely studied to identify abnormal graphs that show significant dissimilarity from the majority of graphs in a collection [; ]. GLAD task is crucial in various real-world applications, such as toxic molecule recognition and pathogenic brain mechanism discovery [; ]. Due to the high cost of data labeling, existing GLAD studies generally follow an unsupervised paradigm, eliminating the requirement of labeled anomaly samples for model training [; ; ].

In the meantime, another line of research – graph-level out-of-distribution detection (GLOD) – has drawn increasing attention in the research community lately [; ]. GLOD aims to identify whether each graph sample in a test set is in-distribution (ID), meaning it comes from the same distribution as the training data, or out-of-distribution (OOD), indicating it comes from different distributions. Considering the universality of distribution shift in open-world data, GLOD plays an important role in real-world high-stakes applications such as drug discovery and cyber-attack detection [; ; ]. Though a few post-hoc GLOD methods [; ] that rely on a well-trained graph classifier have been proposed, most of the existing GLOD methods [; ; ] are developed in an unsupervised fashion, where a specific OOD detection model is trained on unlabeled ID data, and then predicts a score for each test sample to indicate its ID/OOD status. In essence, unsupervised GLOD and unsupervised GLAD share the same goal, suggesting that research in one area could potentially be applied to solve problems in the other. Recent survey and benchmark papers [; ] also identify anomaly detection and OOD detection as two branches under the concept of “generalized OOD detection”, which further highlights the close conceptual connection between GLOD and GLAD tasks.

Figure: Figure 1: Diagram of generalized OOD detection scenarios supported by UB-GOLD.
Refer to caption: x1.png

Despite the inherent connection between the problems of unsupervised GLAD and GLOD, those two research sub-areas have been studied independently in the literature and have distinct evaluation setups. To fill the gap, in this paper, we develop a Unified Benchmark for unsupervised Graph-level OOD and anomaLy Detection (UB-GOLD for short).
As shown in Fig. 1, we unify GLAD and GLOD as “generalized graph-level OOD detection problem”, and consider two GLOD scenarios and two GLAD scenarios for comprehensive evaluation.
For GLAD task that aims to detect graph samples with semantic abnormality, we explore scenarios with intrinsic anomaly and class-based anomaly. Specifically, the first type of dataset inherently includes semantic anomalies, while the second type treats samples from one class (usually the minority class) as anomalies.
For GLOD task that emphasizes the distribution shift between ID and OOD samples, we consider two scenarios with different distribution shifts, i.e., inter-dataset shift and intra-dataset shift. Specifically, the former simulates distribution shifts by drawing ID and OOD samples from different datasets within the same domain [; ], while the latter refers to datasets with intrinsic distribution shifts regarding graph sizes or molecular scaffolds []. Based on 35 datasets belonging to 4 types, we establish a comprehensive benchmark to fairly compare 16 GLAD and GLOD methods under a unified experimental setting, exploring the effectiveness of state-of-the-art (SOTA) approaches across diverse scenarios and domains.

Apart from comparing performance, we further investigate the characteristics of GLAD/GLOD methods in terms of three dimensions: generalizability, robustness, and efficiency, through extensive experiments. For generalizability, we test a well-trained GLAD/GLOD model on OOD data with varying degrees of distribution shift, investigating each method’s ability to identify both near-OOD and far-OOD samples. For robustness, we introduce varying proportions of OOD samples into the training data to observe the impact of noisy ID data on the performance of existing methods. For efficiency, we perform efficiency evaluations for representative GLAD/GLOD approaches, focusing on time and space complexity. At the end of this paper, we discuss the future directions of this emerging research direction. To sum up, the main contribution of this paper is three-fold:

- •
Comprehensive benchmark. We introduce UB-GOLD, the first comprehensive and unified benchmark for GLAD and GLOD. UB-GOLD compares 16 representative GLAD/GLOD methods across 35 datasets from four practical anomaly and OOD detection scenarios.
- •
Multi-dimensional analysis. To explore their ability and limitations, we conduct a systematic analysis of existing methods from multiple dimensions, encompassing effectiveness on different datasets, generalizability to near-OOD and far-OOD, robustness against unreliable training data, and efficiency in terms of time and memory usage.
- •
Unified Codebase and future directions. To facilitate future research and quick implementation, we provide an open-source codebase for UB-GOLD. We also outline potential directions based on our findings to inspire future research.

Key findings. Through comprehensive comparison and analyses, we have the following remarkable observations:
1) The SOTA GLAD/GLOD methods show excellent performance across tasks;
2) Near-OOD samples are harder to detect compared to far-OOD samples;
3) Most methods are vulnerable to the perturbations of training sets; and
4) Certain end-to-end methods outperform 2-step methods in terms of both performance and computational costs.

## 2 Related Work and Problem Definition

In this section, we begin by briefly reviewing related work on GLAD and GLOD. Next, we define the generalized unsupervised graph-level OOD detection problem, formulating GLAD and GLOD into a unified learning task.

Graph-level anomaly detection (GLAD). GLAD aims to identify abnormal graphs that show significant dissimilarity from the majority of graphs in a collection. A simple solution for GLAD is to use graph kernels [; ] to extract graph-level features and utilize shallow anomaly detectors (e.g. OCSVM [] and iForest []) to identify anomalies. Recently, GNN-based end-to-end methods have demonstrated significant performance improvements in GLAD [; ]. Among them, some approaches assume that the anomaly labels of graphs are available for model training, and hence formulate GLAD as a supervised classification problem [; ; ]. Nevertheless, their heavy reliance on labeled anomalies makes it challenging to apply them to real-world scenarios where annotated anomalies are scarce or unavailable. To overcome this shortage, the majority of GNN-based GLAD methods focus on an unsupervised learning paradigm, learning the GLAD model only from normal data [; ; ; ; ; ]. Recent research [; ] indicates that GLAD methods also show great potential in handling GLOD problem, prompting us to unify these two fields into a generalized research problem and conduct a comprehensive benchmarking study.

Graph-level out-of-distribution detection (GLOD). GLOD
refers to the problem of identifying whether a graph sample in a test set is ID or OOD, i.e., whether it originates from the same distribution as the training data or from a different one []. A line of studies developed post-hoc OOD detectors that identify OOD samples by additional fine-tuned detectors on top of well-trained GNN classifiers [; ]. However, these methods require annotated ID data to train the backbone GNNs, which limits their applicability in scenarios where labeled data is unavailable.
In contrast, another line of research proposes training an OOD-specific GNN model using only ID data, without relying on any labels or OOD data [; ; ]. To learn discriminative patterns of ID data, they employ unsupervised learning techniques such as contrastive learning [] and diffusion model [].

Problem definition. Inspired by generalized OOD detection framework in computer vision [; ], we unify GLAD and GLOD into a high-level topic termed generalized graph-level OOD detection. The new learning task aims to distinguish generalized OOD samples (i.e., OOD or abnormal graphs) from generalized ID samples (i.e., ID or normal graphs).
In this paper, we focus on the unsupervised scenario, considering its universality. The research problem is formulated as:

###### Definition 1 (Unsupervised generalized graph-level OOD detection) .

We define the training dataset as $\mathcal{D}_{train}=\{G_{1},\cdots,G_{n}\}$, where each sample $G_{i}$ is a graph drawn from a specific distribution $\mathbb{P}^{in}$. We define the testing dataset as $\mathcal{D}_{test}=\mathcal{D}_{test}^{in}\cup\mathcal{D}_{test}^{out}$, where $\mathcal{D}_{test}^{in}$ contains graphs sampled from $\mathbb{P}^{in}$, and $\mathcal{D}_{test}^{out}$ consists of graphs drawn from an OOD distribution $\mathbb{P}^{out}$. Given $\mathcal{D}_{train}$ and $\mathcal{D}_{test}$, the learning objective is to train a detection model $f(\cdot)$ on $\mathcal{D}_{train}$ and then the model can predict whether each sample $G^{\prime}\in\mathcal{D}_{{test}}$ belongs to $\mathbb{P}^{in}$ or $\mathbb{P}^{out}$. In practice, $f(\cdot)$ is a scoring function, where a larger OOD score $s^{\prime}=f(G^{\prime})$ indicates a higher probability that $G^{\prime}$ is from $\mathbb{P}^{out}$ (i.e., abnormal samples).

###### Definition 1 (Unsupervised generalized graph-level OOD detection) .

## 3 Benchmark Design

In this section, we provide a comprehensive overview of UB-GOLD, covering datasets (Sec. 3.1), algorithms (Sec. 3.2), and evaluation metrics & implementation details (Sec. 3.3).

**Table 1: Statistics of datasets used in UB-GOLD.**
| Dataset Type | Full Name | Abbreviation | Domain | OOD Definition | # ID Train | # ID Test | # OOD Test |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Tox21_p53 | p53 | Molecules | Inherent Anomaly | 8088 | 241 | 28 |
| (Type I) | Tox21_HSE | HSE | Molecules | Inherent Anomaly | 423 | 257 | 10 |
| Intrinsic Anomaly | Tox21_MMP | MMP | Molecules | Inherent Anomaly | 6170 | 200 | 38 |
|  | Tox21_PPAR-gamma | PPAR | Molecules | Inherent Anomaly | 219 | 252 | 15 |
|  | COLLAB | - | Social Networks | Unseen Classes | 1920 | 480 | 520 |
| IMDB-BINARY | IMDB-B | Social Networks | Unseen Classes | 400 | 100 | 100 |  |
| REDDIT-BINARY | REDDIT-B | Social Networks | Unseen Classes | 800 | 200 | 200 |  |
| ENZYMES | - | Proteins | Unseen Classes | 400 | 100 | 20 |  |
| (Type II) | PROTEINS | - | Proteins | Unseen Classes | 360 | 90 | 133 |
| Class-based Anomaly | DD | - | Proteins | Unseen Classes | 390 | 97 | 139 |
|  | BZR | - | Molecules | Unseen Classes | 69 | 17 | 64 |
|  | AIDS | - | Molecules | Unseen Classes | 1280 | 320 | 80 |
|  | COX2 | - | Molecules | Unseen Classes | 81 | 21 | 73 |
|  | NCI1 | - | Molecules | Unseen Classes | 1646 | 411 | 411 |
|  | DHFR | - | Molecules | Unseen Classes | 368 | 93 | 59 |
|  | IMDB-MULTI→IMDB-BINARY | IM→IB | Social Networks | Unseen Datasets | 1350 | 150 | 150 |
| ENZYMES→PROTEINS | EN→PR | Proteins | Unseen Datasets | 540 | 60 | 60 |  |
| AIDS→DHFR | AI→DH | Molecules | Unseen Datasets | 1800 | 200 | 200 |  |
| BZR→COX2 | BZ→CO | Molecules | Unseen Datasets | 364 | 41 | 41 |  |
| (Type III) | ESOL→MUV | ES→MU | Molecules | Unseen Datasets | 1015 | 113 | 113 |
| Inter-Dataset Shift | TOX21→SIDER | TO→SI | Molecules | Unseen Datasets | 7047 | 784 | 784 |
|  | BBBP→BACE | BB→BA | Molecules | Unseen Datasets | 1835 | 204 | 204 |
|  | PTC_MR→MUTAG | PT→MU | Molecules | Unseen Datasets | 309 | 35 | 35 |
|  | FREESOLV→TOXCAST | FS→TC | Molecules | Unseen Datasets | 577 | 65 | 65 |
|  | CLINTOX→LIPO | CL→LI | Molecules | Unseen Datasets | 1329 | 148 | 148 |
|  | GOOD-HIV-Size | HIV-Size | Molecules | Size | 1000 | 500 | 500 |
| GOOD-ZINC-Size | ZINC-Size | Molecules | Size | 1000 | 500 | 500 |  |
| GOOD-HIV-Scaffold | HIV-Scaffold | Molecules | Scaffold | 1000 | 500 | 500 |  |
| GOOD-ZINC-Scaffold | ZINC-Scaffold | Molecules | Scaffold | 1000 | 500 | 500 |  |
| (Type IV) | DrugOOD-IC50-Size | IC50-Size | Molecules | Size | 1000 | 500 | 500 |
| Intra-Dataset Shift | DrugOOD-EC50-Size | EC50-Size | Molecules | Size | 1000 | 500 | 500 |
|  | DrugOOD-IC50-Scaffold | IC50-Scaffold | Molecules | Scaffold | 1000 | 500 | 500 |
|  | DrugOOD-EC50-Scaffold | EC50-Scaffold | Molecules | Scaffold | 1000 | 500 | 500 |
|  | DrugOOD-IC50-Assay | IC50-Assay | Molecules | Protein Target | 1000 | 500 | 500 |
|  | DrugOOD-EC50-Assay | EC50-Assay | Molecules | Protein Target | 1000 | 500 | 500 |

### 3.1 Benchmark Datasets

UB-GOLD includes 35 datasets in four types, each of which corresponds to either a GLAD or GLOD scenario categorized in Fig.1. Table 1 provides an overview of the benchmark datasets.
For GLAD task, we consider two types of datasets with intrinsic anomaly and class-based anomaly to test the models’ performance of detecting anomalous graphs. For GLOD task, we consider datasets with inter-dataset shift and intra-dataset shift that assess the models’ ability to distinguish between ID and OOD samples. For more detailed information on the datasets, please see Appendix A.

- •
Type I: Datasets with intrinsic anomaly. These datasets contain natural anomalies within chemical compounds, testing the robustness of GLAD methods. We use four datasets (i.e., Tox21_HSE, Tox21_MMP, Tox21_p53, and Tox21_PPAR-gamma) from the Tox21 challenge [] involving molecules with unexpected biological activities.
- •
Type II: Datasets with class-based anomaly. In these datasets, certain classes are designated as anomalies. We use 11 datasets (e.g., COLLAB, IMDB-BINARY, and ENZYMES) from the TU benchmark [] where minority or distinct class samples are treated as anomalies.
- •
Type III: Datasets with inter-dataset shift. We synthetic a benchmark dataset with inter-dataset shift by considering samples from one real-world dataset as ID and samples from another real-world dataset as OOD. For example, in “IMDB-MULTI→IMDB-BINARY”, graphs from IMDB-MULTI are ID, and graphs from IMDB-BINARY are OOD. The datasets belonging to the same domain and having close distribution shifts form a pair [].
- •
Type IV: Datasets with intra-dataset shift. Designed to assess GLOD methods under various types of intra-dataset shifts, these datasets are from graph OOD benchmarks, including GOOD [] and DrugOOD []. Specifically, datasets with three kinds of domain shifts, i.e., assay shift, scaffold shift, and size shift, are considered.

### 3.2 Benchmark Algorithms

In UB-GOLD, we consider four groups of GLAD/GLOD methods for a comprehensive evaluation:
1) 2-Step methods, including graph kernel with detector (GK-D) and self-supervised learning (SSL) with detector (SSL-D), and 2) End-to-End methods, including GNN-based GLAD and GLOD methods. All methods are unsupervised, aligning with the primary focus of UB-GOLD to provide a comprehensive and fair evaluation framework. For a detailed summary of the models and further details, please refer to Table 2 and Appendix B.

- •
Graph kernel with detector. These methods follow a two-step process: obtaining graph embeddings using graph kernels and applying outlier detectors in the embedding space. For kernel methods, we consider Weisfeiler-Leman subtree kernel (WL) [] and propagation kernel (PK) []. For detectors, we employ isolation forest (iF) [] and one-class SVM (OCSVM, SVM for short) [].
- •
SSL with detector. These approaches also follow a two-step process but use SSL methods to obtain graph embeddings. We consider two SSL methods, GraphCL (GCL for short) [] and InfoGraph (IG for short) [] to generate embeddings and use iF [] and SVM [] as detectors.

- •
GNN-based GLAD methods.
We consider 6 SOTA methods for GLAD, including OCGIN [], GLocalKD [], OCGTL [], SIGNET [], GLADC [], and CVTGAD []. These methods use different techniques, such as deep one-class classification, contrastive learning, and knowledge distillation, to detect anomalies in graph data.
- •
GNN-based GLOD methods.
We involve two representatives of unsupervised GLOD methods, i.e., GOOD-D [] and GraphDE [], for comparison. GOOD-D is a contrastive learning-based approach, while GraphDE is a generative model-based approach.

**Table 2: Categorization of all benchmark algorithms in UB-GOLD.**
| Method Type | Category | Models and References |
| --- | --- | --- |
| 2-Step | Graph kernel with detector [] (GK-D) | PK-SVM [; ], PK-iF [], WL-SVM [] , WL-iF [] |
| Self-supervised learning with detector [] (SSL-D) | IG-SVM[], IG-iF[], GCL-SVM[], GCL-iF[] |  |
| End-to-End | Graph neural network-based GLAD | OCGIN[], GLADC[], GLocalKD [], OCGTL [], SIGNET [], CVTGAD [] |
| Graph neural network-based GLOD | GOOD-D [], GraphDE [] |  |

### 3.3 Evaluation Metrics and Implementation Details

Evaluation metrics. For comprehensive comparison, UB-GOLD utilizes three commonly used metrics for anomaly/OOD detection [], i.e., AUROC, AUPRC, and FPR95. Higher AUROC and AUPRC values indicate better performance, while lower FPR95 values are preferable.

Data split. In our target scenarios (i.e., unsupervised GLAD/GLOD), all the samples in the training set are normal/ID, while the anomaly/OOD samples only occur in the testing set. In such an unsupervised case, the validation set with anomaly/OOD samples is usually unavailable during the training phase. Thus, following the implementation of OpenOOD [], we divide the datasets into training and testing sets, without using a validation set. Specifically, we adopted the splits from [] and [], applying them to the benchmark datasets. Detailed splits are provided in Table 1.

Hyperparameter search. To obtain the performance upper bounds of various methods on GLAD/GLOD tasks, we conduct a random search to find the optimal hyperparameters w.r.t. their performance on the testing set. The search space is detailed in Table 4. The random search is conducted 20 times or for a maximum of one day per method per dataset to ensure fairness.

For more details related to the experimental setup in UB-GOLD, please refer to Appendix C.

## 4 Experimental Results & Discussion

**Table 3: Comparison in terms of AUROC. The best three results are highlighted by 1st, 2nd, and 3rd. “Avg. AUROC” and “Avg. Rank” indicate the average AUROC and rank across all datasets.**
|  | GK-D (2-step) | SSL-D (2-step) | GNN-based GLAD (E2E) | GNN-based GLOD (E2E) |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | PK-SVM | PK-iF | WL-SVM | WL-iF | IG-SVM | IG-iF | GCL-SVM | GCL-iF | OCGIN | GLocalKD | OCGTL | SIGNET | GLADC | CVTGAD | GOOD-D | GraphDE |
| p53 | 49.17 | 54.05 | 57.69 | 54.40 | 68.11 | 60.85 | 68.61 | 64.60 | 68.35 | 65.43 | 67.58 | 68.10 | 65.82 | 69.40 | 67.82 | 62.59 |
| HSE | 60.72 | 56.49 | 63.27 | 52.98 | 60.33 | 22.77 | 67.40 | 63.95 | 71.42 | 60.21 | 63.36 | 64.56 | 61.37 | 70.52 | 68.71 | 62.47 |
| MMP | 51.03 | 49.95 | 55.50 | 51.98 | 57.72 | 52.58 | 69.91 | 71.31 | 69.37 | 68.12 | 67.51 | 71.23 | 70.03 | 70.58 | 71.41 | 60.12 |
| PPAR | 53.74 | 48.42 | 57.76 | 49.45 | 61.78 | 63.22 | 68.37 | 69.88 | 67.75 | 65.29 | 66.43 | 68.88 | 69.43 | 68.83 | 68.21 | 66.31 |
| COLLAB | 49.72 | 51.38 | 54.62 | 51.41 | 36.47 | 38.18 | 44.91 | 45.44 | 60.58 | 51.85 | 48.13 | 72.45 | 54.32 | 71.01 | 69.34 | 46.77 |
| IMDB-B | 51.75 | 52.83 | 52.98 | 51.79 | 40.89 | 45.64 | 68.00 | 63.88 | 61.47 | 53.31 | 65.27 | 70.12 | 65.94 | 69.82 | 66.68 | 59.25 |
| REDDIT-B | 48.36 | 46.19 | 49.50 | 49.84 | 60.32 | 52.51 | 84.49 | 82.64 | 82.10 | 80.32 | 89.92 | 85.24 | 78.87 | 87.43 | 89.43 | 63.42 |
| ENZYMES | 52.45 | 49.82 | 53.75 | 51.03 | 60.97 | 53.94 | 62.73 | 63.09 | 62.44 | 61.75 | 63.59 | 63.12 | 63.44 | 68.56 | 64.58 | 52.10 |
| PROTEINS | 49.43 | 61.24 | 53.85 | 65.75 | 61.15 | 52.78 | 72.61 | 72.60 | 76.46 | 77.29 | 72.89 | 75.86 | 77.43 | 76.49 | 76.02 | 68.81 |
| DD | 47.69 | 75.29 | 47.98 | 70.49 | 70.33 | 42.67 | 76.43 | 65.41 | 79.08 | 80.76 | 77.76 | 74.53 | 76.54 | 78.84 | 79.91 | 60.49 |
| BZR | 46.67 | 59.08 | 51.16 | 51.71 | 41.50 | 45.42 | 68.93 | 67.81 | 69.13 | 68.55 | 51.89 | 80.79 | 68.23 | 77.69 | 73.28 | 65.94 |
| AIDS | 50.93 | 52.01 | 52.56 | 61.42 | 87.20 | 97.96 | 95.44 | 98.80 | 96.89 | 96.93 | 99.36 | 97.60 | 98.02 | 99.21 | 97.10 | 70.82 |
| COX2 | 52.15 | 52.48 | 53.34 | 49.56 | 49.11 | 48.61 | 59.68 | 59.38 | 57.81 | 58.93 | 59.81 | 72.35 | 64.13 | 64.36 | 63.19 | 54.73 |
| NCI1 | 51.39 | 50.22 | 54.18 | 50.41 | 45.11 | 61.88 | 43.33 | 46.44 | 69.46 | 65.29 | 75.75 | 74.32 | 68.32 | 69.13 | 61.58 | 58.74 |
| DHFR | 48.31 | 52.79 | 50.30 | 51.64 | 45.58 | 63.15 | 58.21 | 57.01 | 61.09 | 61.79 | 59.82 | 72.87 | 61.25 | 63.23 | 64.48 | 53.23 |
| IM→IB | 49.80 | 51.23 | 53.45 | 53.03 | 56.26 | 51.32 | 74.45 | 78.62 | 80.98 | 81.25 | 66.73 | 71.10 | 78.28 | 80.23 | 80.94 | 52.67 |
| EN→PR | 52.53 | 53.36 | 53.92 | 51.90 | 46.01 | 33.52 | 59.76 | 63.23 | 61.77 | 59.36 | 67.18 | 62.42 | 56.95 | 64.31 | 63.84 | 54.48 |
| AI→DH | 51.18 | 51.69 | 52.28 | 50.95 | 84.33 | 83.27 | 97.11 | 98.48 | 95.68 | 94.33 | 98.95 | 96.82 | 95.42 | 99.10 | 99.27 | 94.58 |
| BZ→CO | 43.34 | 52.43 | 49.76 | 52.16 | 64.29 | 64.65 | 78.98 | 76.01 | 87.27 | 80.55 | 81.86 | 89.11 | 83.21 | 96.32 | 95.16 | 65.26 |
| ES→MU | 52.99 | 52.63 | 52.13 | 52.28 | 58.12 | 51.57 | 78.66 | 79.66 | 86.70 | 90.55 | 88.32 | 91.43 | 89.30 | 92.41 | 91.98 | 75.65 |
| TO→SI | 53.73 | 51.87 | 53.50 | 52.25 | 64.32 | 66.53 | 66.85 | 64.85 | 67.29 | 69.80 | 68.91 | 66.72 | 72.51 | 68.24 | 66.70 | 72.34 |
| BB→BA | 54.15 | 53.11 | 54.62 | 53.48 | 63.27 | 32.37 | 69.18 | 67.33 | 78.83 | 77.69 | 78.93 | 89.88 | 79.07 | 80.17 | 81.44 | 55.69 |
| PT→MU | 51.52 | 55.87 | 54.03 | 52.12 | 55.88 | 53.78 | 78.10 | 79.74 | 79.27 | 77.54 | 62.51 | 84.63 | 80.12 | 79.44 | 82.05 | 58.28 |
| FS→TC | 50.06 | 54.76 | 51.98 | 53.24 | 44.98 | 49.57 | 67.05 | 66.01 | 66.98 | 68.92 | 64.38 | 78.12 | 67.32 | 69.89 | 71.58 | 60.12 |
| CL→LI | 50.85 | 51.74 | 52.66 | 51.54 | 55.62 | 56.45 | 59.65 | 54.17 | 61.21 | 58.31 | 59.30 | 72.15 | 63.42 | 70.21 | 69.28 | 50.79 |
| HIV-Size | 48.94 | 49.96 | 66.11 | 45.10 | 31.39 | 32.67 | 26.73 | 35.80 | 38.55 | 42.94 | 96.34 | 91.86 | 47.56 | 56.23 | 74.12 | 68.31 |
| ZINC-Size | 48.66 | 50.12 | 50.58 | 48.96 | 53.07 | 53.47 | 52.61 | 53.46 | 54.22 | 54.21 | 59.41 | 57.29 | 52.53 | 58.78 | 68.43 | 55.63 |
| HIV-Scaffold | 49.36 | 48.43 | 44.72 | 54.57 | 58.78 | 58.74 | 61.00 | 59.26 | 56.82 | 54.38 | 58.05 | 70.93 | 63.23 | 59.49 | 62.28 | 53.48 |
| ZINC-Scaffold | 51.12 | 46.66 | 51.17 | 53.28 | 54.77 | 55.17 | 54.04 | 55.80 | 51.87 | 50.12 | 56.79 | 59.24 | 55.79 | 55.73 | 56.39 | 55.62 |
| IC50-Size | 67.08 | 59.35 | 90.76 | 52.49 | 32.61 | 36.15 | 24.44 | 30.96 | 42.58 | 41.29 | 97.36 | 81.43 | 39.63 | 50.37 | 50.71 | 45.24 |
| EC50-Size | 70.50 | 59.33 | 92.29 | 49.36 | 29.71 | 32.60 | 22.83 | 27.68 | 41.37 | 39.42 | 97.74 | 77.12 | 39.23 | 55.84 | 56.58 | 59.49 |
| IC50-Scaffold | 66.33 | 60.95 | 88.49 | 58.56 | 36.96 | 38.67 | 34.60 | 38.82 | 44.56 | 42.97 | 96.00 | 77.58 | 38.43 | 57.96 | 56.62 | 59.68 |
| EC50-Scaffold | 64.95 | 62.78 | 86.03 | 55.27 | 27.88 | 30.36 | 31.28 | 32.35 | 51.92 | 48.43 | 94.18 | 74.65 | 40.98 | 66.52 | 58.29 | 54.24 |
| IC50-Assay | 54.47 | 53.13 | 59.18 | 52.11 | 51.23 | 51.42 | 51.15 | 51.93 | 52.61 | 49.87 | 68.78 | 65.99 | 52.12 | 54.25 | 53.74 | 57.11 |
| EC50-Assay | 49.08 | 46.66 | 48.43 | 45.32 | 47.80 | 47.81 | 47.80 | 46.02 | 56.11 | 52.44 | 69.31 | 82.97 | 54.34 | 66.71 | 65.57 | 58.96 |
| Avg. AUROC | 52.69 | 53.67 | 57.56 | 52.91 | 52.11 | 50.35 | 61.29 | 61.50 | 66.00 | 64.29 | 73.14 | 75.86 | 65.50 | 71.07 | 71.05 | 60.38 |
| Avg. Rank | 12.09 | 11.94 | 10.46 | 11.77 | 12.06 | 11.65 | 8.14 | 8.31 | 6.69 | 8.23 | 4.91 | 3.34 | 7.14 | 4.51 | 4.54 | 10.14 |

In this section, we introduce the experimental setup and discuss the experimental results in UB-GOLD benchmark. Specifically, we aim to answer the following research questions:
⚫ RQ1 (Effectiveness): How do different GLAD and GLOD methods perform under various anomaly scenarios and distribution shifts? (Sec. 4.1)
⚫ RQ2 (Generalizability): How effective are GLAD and GLOD methods in detecting near-OOD and far-OOD graph samples? (Sec. 4.2)
⚫ RQ3 (Robustness): How does the inclusion of OOD samples in the ID training set affect the robustness of GLAD and GLOD methods? (Sec. 4.3)
⚫ RQ4 (Efficiency): Are the GLAD and GLOD methods efficient in terms of run time and memory usage? (Sec. 4.4)

### 4.1 Performance Comparison (RQ1)

Experiment Design.
We conduct a comprehensive comparison of the detection performance in terms of AUROC, AUPR, and FPR95 of 16 benchmark algorithms on 35 benchmark datasets. For each method and dataset, we conduct 5 runs of experiments and report the average performance.

Experimental Results.
Table 3 shows the performance comparison in terms of AUROC, and Fig. 2 provides box plots to overview the model performance across 35 datasets for all metrics. In Appendix E, detailed performance with standard deviation is demonstrated in Table 8, and the box plots of the ranking of each method are demonstrated in Fig. 6. We have the following observations.

Observation ❶: The SOTA GLAD/GLOD methods show excellent performance on both tasks.
The results in Table 3 highlight the excellent performance of the SOTA GLAD/GLOD methods on both detection tasks. Specifically, the GLAD methods, SIGNET and OCGTL, demonstrate outstanding performance in OOD detection tasks, achieving competitive results in several datasets. Meanwhile, the GLOD method GOOD-D, while not attaining the best performance in anomaly detection tasks, still performs commendably with an average ranking of 4.60, placing it among the top performers. This observation highlights the intercommunity between GLAD and GLOD tasks, emphasizing the need to apply powerful methods designed for one task to the other.

Observation ❷: No universally superior method.
Table 3 demonstrates that no single method consistently outperforms others on more than 12 (out of 35) datasets. Even the top-ranked method, SIGNET, can exhibit sub-optimal performance on several datasets belonging to different types, such as HSE, DD, ES→MU, and ZINC-Size.
In Fig. 2, although SIGNET has a higher median and a compact interquartile range, the top 25% of its performance is still lower than that of some other models. Similar unstable performance can also be found in other competitive methods, such as OCGTL, CVTGAD, and GOOD-D. This finding illustrates the diversity of our benchmark datasets, underscoring the challenge of identifying a “universal method” that performs well across all datasets.

Figure: Figure 2: Comparison of the detection performance on 35 datasets in terms of three metrics.
Refer to caption: x2.png

Observation ❸: Inconsistent performance in terms of different metrics.
From Fig. 2 and Table 8, we can observe that some methods performing well on certain metrics may show unstable performance on other metrics.
Specifically, CVTGOD and GOOD-D consistently achieve high AUC values on most datasets, but show more variability in other metrics, particularly in their sub-optimal AUPRC and FPR95 performance.
This indicates that methods performing well in overall discrimination (high AUROC) may struggle with precision-recall trade-offs (low AUPRC) and maintaining low false positive rates at high true positive rates (low FPR95). This finding highlights the importance of comprehensive evaluation using multiple metrics.

Observation ❹: End-to-end methods show consistent superiority over 2-step methods.
While 2-step methods show notable performance in certain scenarios, end-to-end methods consistently outperform them. Specifically, in Table 3, the average rankings of most end-to-end methods are below 8, while all 2-step methods have rankings above 8. This consistent superiority highlights the advantages of integrated and unified learning approaches over segmented and two-step processes.

### 4.2 Generalizability on Near-OOD and Far-OOD (RQ2)

Experiment Design.
To evaluate the generalizability of GLAD/GLOD methods in handling near and far OOD samples, we consider two different settings to define near-OOD and far-OOD: (A) intra-inter dataset setting and (B) size-based distance setting. In setting A, we define the intra-dataset samples with different class labels as the near-OOD, while samples from another dataset are considered far-OOD. In setting B, the size of graphs serves as the measure to divide near and far OOD. We redefine the size of training/testing sets for fair comparison. Please refer to Appendix C.3 for a more detailed description of experimental settings.

Experimental Results.
The performance of GLAD and GLOD methods in distinguishing between ID and near-OOD/far-OOD samples is demonstrated in Fig. 3 and Table 7, where Subfigures (a)-(c) are in setting A and (d) is in setting B. We have the following key observations.

Observation ❺: Near-OOD samples are harder to detect compared to far-OOD samples.
From Fig. 3, it is evident that end-to-end models generally perform better in detecting far-OOD samples than near-OOD samples.
This trend is consistent across various datasets.

Observation ❻: Poor generalizability of several GLAD/GLOD methods in specialized scenarios.
Despite the overall effectiveness of GLAD/GLOD methods, their performance has notable limitations, particularly in specialized scenarios.
Firstly, in setting A, GLOD approaches (i.e., GOOD-D and GraphDE) exhibit significant performance gaps between near-OOD and far-OOD conditions.
This suggests that when OOD samples are similar to ID samples, the detection capability of GLOD methods is significantly compromised.
This limitation is even more pronounced in setting B, where GLAD methods significantly underperform 2-step methods.

### 4.3 Robustness under Training Set Perturbation (RQ3)

Experiment Design.
In this study, we investigate the robustness of GLAD/GLOD methods against the perturbation of the normal/ID training set by contaminating anomaly/OOD samples. Specifically, we set the perturbation ratios as 0%, 10%, 20%, and 30% to explore the impacts of different perturbation strengths.
For more details, please refer to Appendix C.3.

Experimental Results.
The performance of each GLAD/GLOD method under different strengths of perturbation of the training dataset is demonstrated in Fig. 4, which brings the below observations.

Observation ❼: Performance degradation with increasing contamination ratio.
From Fig. 4, it is obvious that the performance of GLAD/GLOD methods generally deteriorates as the proportion of OOD samples in the ID training set increases. This trend is evident across multiple datasets and methods. This decline in performance suggests that the presence of anomaly/OOD samples in the training data introduces noise and confounds the models, making it increasingly difficult for them to distinguish between generalized ID and OOD samples during testing.

Figure: Figure 3: Near and Far OOD performance comparison.
Refer to caption: x3.png

Figure: Figure 4: Performance of models under different perturbation levels.
Refer to caption: x4.png

Observation ❽: The sensitivity of different methods/datasets can be diverse.
Although increasing perturbation strength affects most methods, some methods exhibit notable robustness to the perturbation. Specifically, GLADC shows minimal performance degradation across all datasets, while CVTGOD demonstrates impressive stability in the BZR. Conversely, several methods are highly sensitive to perturbations in some cases. For example, OCGTL’s performance on the PROTEINS declines sharply as the level of contamination increases. The unstable performance highlights the need for improving the robustness of such methods to ensure reliable performance in real-world scenarios where data contamination is common.

### 4.4 Efficiency Analysis (RQ4)

Experiment Design.
We evaluate the computational efficiency of GLAD/GLOD methods using default hyperparameter settings. Our assessment focuses on two main aspects: time efficiency and memory usage. See Appendix C.2 for a detailed description.

Experimental Results. We present the computational efficiency of all methods in terms of time and memory usage in Fig.5 and Fig.5, respectively. We highlight the below observation.

Figure: Figure 5: Time and memory usage comparison.
Refer to caption: x5.png

Observation ❾: Certain end-to-end methods outperform 2-step methods in terms of both performance and computational costs.
Observation ❹ demonstrates that end-to-end methods usually outperform the 2-step methods. In addition to their superior performance, most end-to-end methods exhibit comparable time efficiency and significantly better memory efficiency. As shown in Fig. 5, end-to-end methods achieve optimal results much faster than 2-step methods, except for GOOD-D and GraphDE. In contrast, the iF detector in 2-step methods significantly increases the time required to reach optimal performance, leading to substantially higher time costs compared to most existing methods.
In terms of memory usage, as illustrated in Fig. 5, end-to-end methods show much lower CPU and GPU consumption than 2-step methods. Although graph kernel methods do not utilize GPU resources, their high CPU consumption is a critical consideration. To sum up, the majority of end-to-end methods may be preferable for GLAD/GLOD tasks due to their advantages in effectiveness and efficiency.

## 5 Conclusion and Future Directions

In this paper, we introduce a unified benchmark, UB-GOLD, for graph-level anomaly detection (GLAD) and graph-level out-of-distribution detection (GLOD), which comprehensively compare the performance of 16 GLAD/GLOD methods on 35 datasets across various scenarios and domains.
Based on UB-GOLD, we conduct extensive experiments to analyze the effectiveness, generalizability, robustness, and efficiency of these methods.
Observation ❶ highlights the intercommunity between GLAD and GLOD tasks.
It reveals that many approaches show comparable performance across both tasks, suggesting a significant overlap in the underlying principles and techniques used for detecting anomalies and OOD samples.
Several insightful observations are summarized from the results, providing an in-depth understanding of existing methods and inspiration for future work. Our experiments reveal that SOTA methods for GLAD and GLOD exhibit superior performance across both tasks, underscoring the strong interconnection between these two forefront research domains.

Despite the promising results of existing approaches, several critical challenges and research directions remain worthy of future investigation:

- •
Universal approaches for diverse datasets. Observation ❷ suggests the existing GLAD/GLOD methods do not consistently perform well across diverse datasets. In this case, it is a promising opportunity for producing novel approaches that can generally work well on diverse datasets. Such universal approaches should be aware of various structural and attribute characteristics of graph data from diverse domains and be sensitive to different types of OOD and anomaly samples.
- •
Awareness of near-OOD samples. Observations ❺ and ❻ indicate that most existing methods struggle to detect near-OOD samples. Thus, future research is expected to discover more advanced solutions to effectively differentiate between near-OOD and ID samples. Considering the inaccessibility of OOD samples during model training, accurately capturing the patterns of ID data and establishing reliable decision boundaries can be the key to achieving the goal.
- •
Robust approaches against unclean training data. Observations ❼ and ❽ expose that most methods are vulnerable to perturbation (i.e., data contaminated by OOD samples) of training datasets. Considering the difficulty of acquiring clean training data in several real-world applications, future approaches are expected to be more robust against noisy training data.

Limitation. Since UB-GOLD mainly focuses on unsupervised GLAD/GLOD tasks, the methods that require class labels (e.g. AAGOD []), anomaly labels (e.g. i-GAD []), and domain-specific pre-trained models (e.g., PGR-MOOD []) are not included for fair comparison. As a long-term evolving project, UB-GOLD will include these methods in the codebase for quick implementation.

## Checklist

- 1.
For all authors…
(a)
Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b)
Did you describe the limitations of your work?
[Yes] See Section 5: Conclusion and Future Directions
(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
(d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
- 2.
If you are including theoretical results…
(a)
Did you state the full set of assumptions of all theoretical results?
[N/A]
(b)
Did you include complete proofs of all theoretical results?
[N/A]
- 3.
If you ran experiments (e.g. for benchmarks)…
(a)
Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Abstract.
(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 3.3:Evaluation Metrics and Implementation Details.
(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Appendix D:Additional Experimental Results.
(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C.2: Additional Experimental Detail.
- 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…
(a)
If your work uses existing assets, did you cite the creators?
[Yes]
(b)
Did you mention the license of the assets?
[Yes] See Appendix D: Reproducibility
(c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes] See Appendix D: Reproducibility
(d)
Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] See Appendix D: Reproducibility, and cite their paper.
(e)
Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] See Appendix D: Reproducibility
- 5.
If you used crowdsourcing or conducted research with human subjects…
(a)
Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b)
Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
(c)
Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]

## Appendix A Detailed Description of datasets in UB-GOLD

The description of datasets in UB-GOLD is given as follows.

⚫ Type I: Datasets with intrinsic anomaly.(^1^11 [Tox21 Challenge Data.](https://tripod.nih.gov/tox21/challenge/data.jsp))

- •
HSE []: Focuses on human stress response elements, essential for studying stress-induced cellular changes.
- •
MMP []: Targets matrix metalloproteinases, crucial for understanding processes like cancer metastasis.
- •
p53 []: Involves the p53 protein, a key player in cancer prevention and apoptosis.
- •
PPAR-gamma []: Examines peroxisome proliferator-activated receptor gamma, important for metabolic and inflammatory processes.

⚫ Type II: Datasets with class-based anomaly.(^2^22 [TUDataset.](https://chrsmrrs.github.io/datasets/docs/datasets/))

- •
COLLAB []: Scientific collaboration networks, representing authors and their co-authored papers in various scientific fields.
- •
IMDB-BINARY []: Movie collaboration networks, where nodes represent actors and edges denote their co-appearance in films. This dataset helps in analyzing the collaboration patterns within the film industry.
- •
REDDIT-BINARY []: Discussion threads from Reddit, each graph representing a thread with nodes as users and edges as interactions. It captures the structure of online discussions and their dynamics.
- •
ENZYMES []: Protein tertiary structures, each graph represents an enzyme with nodes as secondary structure elements and edges as spatial adjacencies, crucial for biochemical and functional studies.
- •
PROTEINS []: Comprehensive protein structures and interaction data, where nodes represent amino acids and edges denote interactions, providing insights into protein functions and interactions.
- •
DD []: Protein-protein interaction networks, capturing the intricate relationships between proteins within biological systems, essential for understanding cellular functions.
- •
BZR []: Benzodiazepine receptor ligands, with nodes representing atoms and edges representing bonds, used for studying molecular interactions with benzodiazepine receptors.
- •
AIDS []: Chemical compounds screened for antiviral activity against HIV, where each graph represents a molecule, useful in drug discovery and antiviral research.
- •
COX2 [] : Cyclooxygenase-2 inhibitors, where nodes and edges represent molecular structures, important for studying anti-inflammatory drug properties.
- •
NCI1 []: Chemical compounds screened for anti-cancer activity, each graph representing a molecule, used for evaluating potential anti-cancer drugs.
- •
DHFR []: Dihydrofolate reductase inhibitors, with graphs representing molecular structures, focusing on compounds inhibiting the DHFR enzyme, vital for cancer and bacterial infection treatments.

⚫ Type III: Datasets with inter-dataset shift.

All graphs in the following dataset represent molecules, where nodes are atoms, and edges are chemical bonds.

- •
BBBP []: BBBP is the Blood–brain barrier penetration (BBBP) dataset includes binary labels for over 2000 compounds on their permeability properties.
- •
BACE []: BACE provides a series of human $\beta$-secretases as well as their binary label, and all data are experimental values reported in the scientific literature over the last decade.
- •
CLINTOX []: The ClinTox dataset compares drugs that have been approved by the FDA with drugs that have failed in clinical trials for toxicity reasons.
- •
LIPO []: The full name of LIPO is Lipophilicity, which is an important feature of drug molecules that affects both membrane permeability and solubility.
- •
FREESOLV []: The Free Solvation database (FreeSolv) provides experimental and calculated free energies of hydration of small molecules in water.
- •
TOXCAST []: ToxCast providing toxicology data for a library of compounds based on high-throughput screening and includes qualitative results of over 600 experiments on 8615 compounds.
- •
ESOL []: ESOL is a small dataset containing water solubility data for 1128 compounds with the goal of estimating solubility from chemical structures.
- •
MUV []: The Maximum Unbiased Validation (MUV) group is another benchmark dataset selected from PubChem BioAssay by applying a modified nearest neighbor analysis
- •
TOX21 []: The dataset contains qualitative toxicity measurements of 8014 compounds against 12 different targets, including nuclear receptors and stress response pathways.
- •
SIDER []: The Side Effect Resource (SIDER) is a database of marketed drugs and adverse drug reactions which measured for 1427 approved drugs.
- •
PTC-MR []: PTC dataset labels compounds based on their carcinogenicity where MR Indicates that their rodent is a male rat.

⚫ Type IV: Datasets with intra-dataset shift.

- •
GOOD []: A systematic benchmark specifically tailored for the graph OOD problem. We utilize two molecular datasets for OOD detection tasks: (1) GOOD-HIV is a small-scale real-world molecular dataset which aim to predict whether this molecule can inhibit HIV replication. (2) GOOD-ZINC is a real-world molecular property regression dataset from ZINC database and aims at predicting molecular solubility. Each dataset
comprises two ID-OOD splitting strategies (scaffold and size), resulting in a total of 4 distinct datasets.
- •
DrugOOD []: This OOD benchmark is designed for AI-aided drug discovery. It includes three ID-OOD splitting strategies: assay, scaffold, and size. These strategies are applied to two measurements (IC50 and EC50), resulting in six datasets. Each dataset comprises a binary classification task aimed at predicting drug target binding affinity.

## Appendix B Detailed Description of Algorithms in UB-GOLD

The description of benchmarking algorithms in UB-GOLD is demonstrated as follows.

⚫ Graph kernels.

- •
Weisfeiler-Leman Subtree Kernel (WL) [] : This kernel generates graph embeddings by iteratively refining node labels based on subtree patterns. It effectively captures structural similarities within graphs, making it a powerful tool for embedding generation.
- •
Propagation Kernel (PK) []: This method propagates labels through the graph structure, resulting in embeddings that reflect the graph’s topology. It captures relational information within the graph, providing a robust basis for subsequent outlier detection.

⚫ Self-supervised learning methods.

- •
GraphCL (GCL) []: GCL leverages augmentations to learn robust graph-level representations. By contrasting different views of the same graph, the model learns to capture essential graph structures and properties, making it highly effective for various graph-based tasks. This method is known for its strong performance in unsupervised learning settings.
- •
InfoGraph (IG) []: IG maximizes the mutual information between local and global graph representations to achieve unsupervised and semi-supervised graph-level representation learning. By capturing meaningful information across different levels of the graph, IG effectively learns comprehensive graph embeddings that are useful for a wide range of downstream tasks.

⚫ Outlier Detectors.

- •
Isolation Forest (iF) []: This algorithm isolates observations by constructing an ensemble of trees, each built by randomly selecting features and split values. An anomaly score is calculated based on the path length from the root to the leaf node, with shorter paths indicating anomalies.
- •
One-Class SVM (OCSVM) []: OCSVM aims to separate the data from the origin using a hyperplane in a high-dimensional space. Points that lie far from the hyperplane are considered outliers.

⚫ Graph neural network-based GLAD methods.

- •
OCGIN []: Utilizes a GIN encoder optimized with a Support Vector Data Description (SVDD) objective to identify anomalies within the graph structure. This method employs an end-to-end GNN model with one-class classification for effective anomaly detection.
- •
GLocalKD []: Jointly learns two GNNs and performs graph-level and node-level random knowledge distillation between their learned representations. By leveraging both local and global knowledge, this approach enhances the detection of anomalies.
- •
OCGTL []: Extends deep one-class classification to a self-supervised detection approach using neural transformations and graph transformation learning as regularization. This technique improves the model’s unsupervised anomaly detection capabilities.
- •
SIGNET []: Proposes a self-interpretable graph-level anomaly detection framework that infers anomaly scores while providing subgraph explanations. By maximizing mutual information of multi-view subgraphs, it achieves both detection and interpretation of anomalies.
- •
GLADC []: Incorporates graph-level adversarial contrastive learning to identify anomalies. Through the creation of adversarial examples and learning robust representations, this method effectively distinguishes between normal and abnormal graphs.
- •
CVTGAD []: Introduces a simplified transformer with cross-view attention for unsupervised graph-level anomaly detection. It overcomes the limited receptive field of GNNs by using a transformer-based module to capture relationships between nodes and graphs from both intra-graph and inter-graph perspectives.

⚫ Graph neural network-based GLOD methods.

- •
GOOD-D []: Performs perturbation-free graph data augmentation and utilizes hierarchical contrastive learning on the generated graphs for graph-level OOD detection. By leveraging multiple levels of contrastive learning, GOOD-D enhances the representation learning process, making it robust in distinguishing between in-distribution (ID) and out-of-distribution (OOD) graphs.
- •
GraphDE []: Models the generative process of the graph to characterize distribution shifts. Using variational inference, GraphDE infers the environment from which a graph sample is drawn. This generative approach effectively identifies ID and OOD graphs by detecting shifts in the underlying data distribution.

## Appendix C Supplemental Information of UB-GOLD

### C.1 Metrics

In UB-GOLD, we consider three metrics for evaluation. Their definitions are given as follows.

- •
AUROC: Fundamental for both GLOD and GLAD, AUROC measures a model’s ability to distinguish between normal and anomalous or OOD instances across various threshold levels. A higher AUROC value indicates better performance in correctly classifying positives (anomalies or OOD instances) and negatives (normal instances), making it crucial for evaluating the overall effectiveness of detection algorithms.
- •
AUPRC: Gains importance in imbalanced datasets, common in AD where anomalies are rare, and in GLOD where OOD instances are infrequent. This metric focuses on precision (the accuracy of positive predictions) and recall (the model’s ability to detect all positive cases), providing a clear measure of performance in scenarios where positive cases are critical and more challenging to detect.
- •
FPR95: Evaluates the number of false positives accepted when the model correctly identifies 95% of true positives. This metric is particularly useful in settings where missing an anomaly or an OOD instance can lead to significant consequences, emphasizing the need for models that maintain high sensitivity without sacrificing specificity.

**Table 4: Hyper-parameter search space of all implemented methods.**
| Algorithm | Hyper-parameter | Search Space |
| --- | --- | --- |
| General Settings | hidden size<br>dropout<br>layers<br>learning rate | 16, 32, 64, 128<br>0, 0.1, 0.2, 0.3<br>1, 2, 3, 4<br>1e-1, 1e-2, 1e-3, 1e-4 |
| GOOD-D | str_dim<br>cluster number<br>$\alpha$ | 8, 16, 24, 32<br>2, 3, 4<br>[0, 1.0] |
| GraphDE | dropedge<br>dropnode<br>model type | 0, 0.1, 0.2, 0.3<br>0, 0.1, 0.2, 0.3<br>graphde-v, graphde-a |
| CVTGAD | str_dim<br>cluster number<br>$\alpha$<br>pooling | 8, 16, 24, 32<br>2, 3, 4<br>[0, 1.0]<br>mean, max |
| GLADC | hidden size<br>output size | 32, 64, 128, 256<br>32, 64, 128 |
| GLocalKD | clip<br>nobn<br>nobias | 0.10, 0.15, 0.20<br>True, False<br>True, False |
| OCGTL | hidden size<br>layers | 32, 64, 128<br>2, 3, 4, 5 |
| SIGNET | pooling<br>readout<br>layers | add, max<br>concat, add, last<br>3, 4, 5 |
| OCGIN | aggregation<br>bias | mean, add, max<br>True, False |
| GCL+kernel | tree number<br>sample ratio | 200, 250, 300<br>0.3, 0.4, 0.5, 0.6 |
| KernelGLAD | tree number<br>sample ratio<br>neighbors<br>leaves<br>WL iteration | 200, 250, 300<br>0.3, 0.4, 0.5, 0.6<br>20, 30, 40<br>25, 30, 35<br>3, 4, 5, 6, 7 |

### C.2 Additional Experimental Details

Implementation Details. To ensure a comprehensive evaluation and maintain fairness across a broad spectrum of models, we develop an open-source toolkit named UB-GOLD. This toolkit is built on top of Pytorch 2.01 [], torch_geometric 2.4.0 [] and DGL 2.1.0 [].
We implement graph kernel methods with the DGL library. All other models are unified using the torch_geometric library. GCL and IG are included via the PYGCL library [].

Hardware Specifications.
All our experiments were carried out on a Linux server with an Intel(R) Xeon(R) Gold 5120 2.20GHz CPU, 160GB RAM, and NVIDIA A40 GPU, 48GB RAM.

Hyperparameter Settings.
Table 4 provides a comprehensive list of all hyperparameters used in our random search complete with their search spaces. For the design of the default hyperparameters please refer to our code base in. /benchmark/Source.

Efficiency Analysis (Sec. 4.4).
We evaluate the computational efficiency of GLOD and GLAD methods using default hyperparameter settings. Our assessment focuses on two main aspects:

- •
Time Efficiency: We record the average time each method takes to achieve the best results across all datasets, providing insights into their processing speed.
- •
Resource Usage: We monitor each method’s CPU and GPU consumption (on COLLAB) during experiments, determining their demand for computational resources.

This setup allows us to measure the methods’ efficiency directly and reliably, reflecting their practicality for real-world application.

### C.3 New Experiments Dataset Split

In the experiments for generalizability and robustness, we do not follow the original split but utilize experiment-specific splits for fair comparison. The details are as follows.

Generalizability on Near-OOD and Far-OOD (Sec. 4.2).
To rigorously evaluate the performance of GLOD and GLAD methods in handling near and far OOD conditions, we have implemented a distinct partitioning strategy for this research question.
Unlike the setup in RQ1, here we ensure that the number of samples in the Near OOD, Far OOD, and ID Test groups are precisely equal, detailed in Table 5. This balanced configuration is designed to provide a fair comparison across different degrees of OOD scenarios, and it includes two specific setups:

**Table 5: Dataset statistics including ID Train, ID Test, Near OOD, and Far OOD counts. Additionally, Unseen Class (UC), Unseen Dataset (UD), Near Size (NS), and Far Size (FS).**
| Scenario Type | Data | ID Train | ID/Near/Far Test | Near OOD | Far OOD |
| --- | --- | --- | --- | --- | --- |
| Class-based Anomaly | AIDS | 1280 | 80 | AIDS(UC) | DHFR(UD) |
| BZR | 69 | 17 | BZR(UC) | COX2(UD) |  |
| ENZYMES | 400 | 20 | ENZYMES(UC) | PROTEINS(UD) |  |
| Size-based | GOOD-ZINC-Size | 1000 | 500 | ZINC(NS) | ZINC(FS) |

- •
Intra-inter dataset setting: We utilize the class-based anomaly partitioning method to set the ID Train and ID Test, along with the Near OOD Test. The Far OOD Test employs datasets from the Inter-Dataset Shift category, representing more significant deviations.
- •
Size-based distance setting: We maintain the same ID Train and ID Test groupings of Intra-Dataset Shift. However, for the Near OOD and Far OOD tests, we categorize the samples based on differing graph sizes, with smaller sizes representing Near OOD and larger sizes for Far OOD.

These settings are designed to rigorously test the capability of GLOD and GLAD methods to recognize and differentiate between subtle and substantial distribution shifts, thereby assessing their effectiveness in realistic and challenging environments.

Robustness under Training Set Perturbation (Sec. 4.3).
In this study, we investigate the robustness of GLOD and GLAD methods against the contamination of the ID training set with OOD samples. In Table 6, we begin by partitioning the OOD test dataset, randomly selecting 30% of its samples to be mixed into the ID training set. The remaining 70% of the OOD test dataset is kept intact for performance evaluation. This procedure is repeated to create four distinct experimental groups where 0%, 10%, 20%, and 30% of the originally selected OOD samples are added to the ID training dataset. These modifications allow us to systematically explore how progressively increasing the proportion of OOD samples within the ID training set affects the methods’ ability to identify and differentiate true OOD instances during testing accurately.

**Table 6: Dataset statistics including ID Train, ID Train with different perturbation levels, ID Test, and OOD Test counts.**
| Data | ID Train | ID Train (10%) | ID Train (20%) | ID Train (30%) | ID Test | OOD Test |
| --- | --- | --- | --- | --- | --- | --- |
| BZR | 69 | 76 | 82 | 89 | 17 | 44 |
| PROTEINS | 360 | 374 | 387 | 400 | 90 | 93 |
| COX2 | 81 | 89 | 96 | 103 | 21 | 51 |

## Appendix D Reproducibility

Ensuring the reproducibility of experimental results is a core principle of UB-GOLD. Below, we outline the measures we have taken to achieve this:

Accessibility.
All datasets, algorithm implementations, and experimental configurations are freely accessible through our open-source project at [https://github.com/UB-GOLD/UB-GOLD](https://github.com/UB-GOLD/UB-GOLD). No special requests or permissions are needed to access the resources.

Datasets.
Our datasets are publicly available and include TUDataset, OGB, TOX21, DrugOOD, and GOOD. Among them, TUDataset [], OGB [], and TOX21 [] are licensed under the MIT License. DrugOOD [] is licensed under the GNU General Public License 3.0. GOOD [] is licensed under GPL-3.0.

All these datasets are permitted by their authors for academic use and contain no personally identifiable information or offensive content.

Documentation and Usage.
We provide comprehensive documentation to facilitate easy use of our library. The code includes thorough comments to enhance readability. Users can reproduce experimental results by following the examples, which outline how to run the code with specific data, methods, and GPU configuration arguments.

License.
UB-GOLD is distributed under the MIT license, ensuring wide usability and adaptability.

Code Maintenance.
We are dedicated to regularly updating our codebase, addressing user feedback, and incorporating community contributions. We also enforce strict version control to maintain reproducibility throughout the code maintenance process.

With these measures, UB-GOLD aims to foster transparency, accessibility, and collaboration within the research community.

**Table 7: Near and Far OOD performance comparison.**
|  |  | AIDS | BZR | ENZYMES | ZINC-Size |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Type | AUROC $\uparrow$ | AUPR $\uparrow$ | FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$ | FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$ | FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$ | FPR95 $\downarrow$ |
| IG-iF | Far OOD | 46.50 | 48.88 | 93.75 | 50.80 | 53.88 | 94.12 | 51.87 | 56.31 | 92.00 | 48.95 | 50.19 | 96.00 |
|  | Near OOD | 45.92 | 47.75 | 93.75 | 49.58 | 54.43 | 91.76 | 50.15 | 54.76 | 89.00 | 50.39 | 50.90 | 94.72 |
| IG-SVM | Far OOD | 47.17 | 48.57 | 95.75 | 50.17 | 51.98 | 95.29 | 53.88 | 55.27 | 85.00 | 48.66 | 50.02 | 95.64 |
|  | Near OOD | 45.01 | 46.42 | 95.50 | 50.83 | 55.36 | 96.47 | 53.85 | 55.51 | 85.00 | 50.14 | 50.46 | 94.92 |
| GCL-iF | Far OOD | 49.73 | 50.60 | 94.00 | 38.55 | 46.59 | 100.00 | 42.85 | 50.73 | 93.00 | 48.95 | 49.71 | 95.32 |
|  | Near OOD | 53.53 | 53.86 | 94.50 | 47.23 | 54.29 | 98.82 | 50.65 | 52.68 | 88.00 | 50.51 | 50.00 | 94.00 |
| GCL-SVM | Far OOD | 50.26 | 50.60 | 93.00 | 42.56 | 49.85 | 97.65 | 44.82 | 50.47 | 91.00 | 49.43 | 49.90 | 95.20 |
|  | Near OOD | 53.52 | 53.41 | 93.50 | 47.47 | 51.52 | 95.29 | 48.20 | 53.17 | 90.00 | 50.55 | 50.36 | 93.76 |
| OCGIN | Far OOD | 99.88 | 99.88 | 0.75 | 87.20 | 83.44 | 37.65 | 63.45 | 72.12 | 93.00 | 41.97 | 45.39 | 97.00 |
|  | Near OOD | 96.01 | 95.95 | 13.25 | 66.16 | 59.78 | 50.59 | 58.35 | 63.39 | 89.00 | 36.96 | 45.98 | 99.56 |
| GLocalKD | Far OOD | 100.00 | 100.00 | 0.00 | 83.39 | 75.63 | 35.29 | 53.70 | 67.07 | 85.00 | 30.07 | 34.50 | 95.40 |
|  | Near OOD | 99.89 | 99.90 | 0.00 | 65.81 | 58.54 | 51.76 | 40.45 | 54.30 | 85.00 | 13.14 | 31.09 | 99.92 |
| OCGTL | Far OOD | 100.00 | 100.00 | 0.00 | 84.64 | 77.25 | 28.24 | 80.10 | 77.04 | 41.00 | 84.40 | 70.09 | 23.44 |
|  | Near OOD | 99.61 | 99.66 | 0.00 | 65.19 | 58.08 | 48.24 | 74.20 | 69.04 | 51.00 | 99.58 | 99.32 | 1.00 |
| SIGNET | Far OOD | 98.00 | 93.07 | 2.00 | 95.50 | 90.38 | 11.76 | 67.48 | 67.05 | 86.00 | 80.48 | 79.23 | 66.96 |
|  | Near OOD | 97.16 | 91.02 | 2.50 | 88.93 | 83.93 | 85.88 | 71.60 | 71.27 | 68.00 | 72.90 | 71.41 | 82.64 |
| GLADC | Far OOD | 100.00 | 100.00 | 0.00 | 79.58 | 69.52 | 41.18 | 43.00 | 57.80 | 90.00 | 35.55 | 40.40 | 98.12 |
|  | Near OOD | 98.74 | 98.60 | 3.75 | 64.36 | 56.84 | 52.94 | 41.25 | 43.32 | 85.00 | 11.63 | 32.34 | 100.00 |
| CVTGOD | Far OOD | 99.94 | 99.94 | 0.25 | 93.91 | 91.30 | 17.65 | 70.20 | 76.29 | 87.00 | 44.57 | 51.56 | 95.44 |
|  | Near OOD | 94.93 | 95.01 | 24.00 | 80.48 | 72.37 | 63.53 | 53.05 | 58.40 | 88.00 | 20.68 | 38.52 | 99.84 |
| GOOD-D | Far OOD | 98.95 | 98.93 | 5.75 | 99.31 | 99.37 | 9.41 | 71.60 | 77.06 | 83.00 | 67.73 | 67.68 | 88.00 |
|  | Near OOD | 87.80 | 88.60 | 43.25 | 87.54 | 86.49 | 69.41 | 58.10 | 62.40 | 88.00 | 55.85 | 56.94 | 94.36 |
| GraphDE | Far OOD | 80.25 | 90.13 | 100.00 | 51.18 | 50.91 | 96.47 | 58.75 | 64.84 | 100.00 | 59.01 | 65.91 | 98.80 |
|  | Near OOD | 50.09 | 71.87 | 100.00 | 49.41 | 50.61 | 97.65 | 40.30 | 47.18 | 100.00 | 56.61 | 57.25 | 98.00 |

Figure: Figure 6: Comparison of the ranking on 35 datasets in terms of three metrics.
Refer to caption: x7.png

## Appendix E Additional Experimental Results

In this section, we present additional experimental results, providing comprehensive insights into the performance of our models across different datasets and metrics.

Main experiments.
Table 8 presents the complete results of the main experiment. Each model is executed 5 times with varying random seeds, and the mean scores along with standard deviations are reported. “Avg. AUROC”, “Avg. FPR95”, “Avg. AUPRC”, and “Avg. Rank” indicate the average AUROC, FPR95, AUPRC, and rank across all datasets, respectively. We observe that the End-to-End approach stands out in AUROC, achieving the best overall results. However, in the metrics of AUPRC and FPR95, SSL-D in the 2-step approach also performs well, showing competitive results.

Ranking experiments.
Fig. 6 shows the ranking of models across 35 datasets in terms of their performance. It provides a visual representation of the comparative ranking, allowing for an easy assessment of how different models rank against each other across a large number of datasets.

Near and far OOD performance.
We provide the full results of Near-OOD and Far-OOD evaluations for three metrics across four datasets, as shown in Table 7. This table allows for a detailed comparison of Near and Far OOD performance, showcasing how our models perform under different out-of-distribution scenarios. Further confirming our findings in Observations❺ and ❻.

**Table 8: Comparison in terms of AUPRC (top), AUROC (middle), and FPR95 (bottom). The best three results are highlighted using 1st, 2nd, and 3rd. Avg. AUROC”, Avg. FPR95”, “Avg. AUPRC” and “Avg. Rank” indicate the average AUROC, FPR95, AUPRC, and rank across all datasets.**
|  | GK-D (2-step) | SSL-D (2-step) | GNN-based GLAD (E2E) | GNN-based GLOD (E2E) |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | PK-SVM | PK-iF | WL-SVM | WL-iF | IG-SVM | IG-iF | GCL-SVM | GCL-iF | OCGIN | GLocalKD | OCGTL | SIGNET | GLADC | CVTGAD | GOOD-D | GraphDE |
| p53 | $49.17_{\pm 0.46}$ | $54.05_{\pm 0.25}$ | $57.69_{\pm 0.57}$ | $54.40_{\pm 0.41}$ | $68.11_{\pm 0.07}$ | $60.85_{\pm 0.68}$ | $68.61_{\pm 0.14}$ | $64.60_{\pm 0.30}$ | $68.35_{\pm 1.20}$ | $65.43_{\pm 0.29}$ | $67.58_{\pm 0.50}$ | $68.10_{\pm 0.15}$ | $65.82_{\pm 0.21}$ | $69.40_{\pm 0.28}$ | $67.82_{\pm 0.37}$ | $62.59_{\pm 0.56}$ |
| HSE | $60.72_{\pm 0.43}$ | $56.49_{\pm 0.66}$ | $63.27_{\pm 0.26}$ | $52.98_{\pm 0.53}$ | $60.33_{\pm 0.59}$ | $22.77_{\pm 0.63}$ | $67.40_{\pm 0.16}$ | $63.95_{\pm 0.39}$ | $71.42_{\pm 0.10}$ | $60.21_{\pm 0.12}$ | $63.36_{\pm 0.69}$ | $64.56_{\pm 0.60}$ | $61.37_{\pm 0.48}$ | $70.52_{\pm 0.30}$ | $68.71_{\pm 0.52}$ | $62.47_{\pm 0.27}$ |
| MMP | $51.03_{\pm 0.44}$ | $49.95_{\pm 1.38}$ | $55.50_{\pm 0.67}$ | $51.98_{\pm 0.39}$ | $57.72_{\pm 0.58}$ | $52.58_{\pm 0.29}$ | $69.91_{\pm 0.13}$ | $71.31_{\pm 0.41}$ | $69.37_{\pm 0.12}$ | $68.12_{\pm 0.23}$ | $67.51_{\pm 0.27}$ | $71.23_{\pm 0.18}$ | $70.03_{\pm 0.46}$ | $70.58_{\pm 0.22}$ | $71.41_{\pm 0.51}$ | $60.12_{\pm 0.34}$ |
| PPAR | $53.74_{\pm 0.21}$ | $48.42_{\pm 0.16}$ | $57.76_{\pm 0.33}$ | $49.45_{\pm 1.12}$ | $61.78_{\pm 0.66}$ | $63.22_{\pm 0.22}$ | $68.37_{\pm 0.44}$ | $69.88_{\pm 0.28}$ | $67.75_{\pm 0.34}$ | $65.29_{\pm 0.58}$ | $66.43_{\pm 0.67}$ | $68.88_{\pm 0.42}$ | $69.43_{\pm 0.36}$ | $68.83_{\pm 0.10}$ | $68.21_{\pm 0.19}$ | $66.31_{\pm 0.30}$ |
| COLLAB | $49.72_{\pm 0.60}$ | $51.38_{\pm 0.20}$ | $54.62_{\pm 1.28}$ | $51.41_{\pm 0.39}$ | $36.47_{\pm 0.42}$ | $38.18_{\pm 0.34}$ | $44.91_{\pm 0.56}$ | $45.44_{\pm 0.83}$ | $60.58_{\pm 0.27}$ | $51.85_{\pm 0.18}$ | $48.13_{\pm 0.41}$ | $72.45_{\pm 0.11}$ | $54.32_{\pm 0.37}$ | $71.01_{\pm 0.58}$ | $69.34_{\pm 0.55}$ | $46.77_{\pm 0.48}$ |
| IMDB-B | $51.75_{\pm 0.30}$ | $52.83_{\pm 0.51}$ | $52.98_{\pm 0.69}$ | $51.79_{\pm 0.32}$ | $40.89_{\pm 0.48}$ | $45.64_{\pm 0.67}$ | $68.00_{\pm 0.22}$ | $63.88_{\pm 0.15}$ | $61.47_{\pm 0.18}$ | $53.31_{\pm 0.53}$ | $65.27_{\pm 0.24}$ | $70.12_{\pm 0.61}$ | $65.94_{\pm 0.26}$ | $69.82_{\pm 0.13}$ | $66.68_{\pm 0.14}$ | $59.25_{\pm 0.39}$ |
| REDDIT-B | $48.36_{\pm 0.67}$ | $46.19_{\pm 0.21}$ | $49.50_{\pm 0.54}$ | $49.84_{\pm 0.11}$ | $60.32_{\pm 0.32}$ | $52.51_{\pm 0.62}$ | $84.49_{\pm 0.20}$ | $82.64_{\pm 0.42}$ | $82.10_{\pm 0.37}$ | $80.32_{\pm 0.10}$ | $89.92_{\pm 0.12}$ | $85.24_{\pm 0.45}$ | $78.87_{\pm 0.56}$ | $87.43_{\pm 0.60}$ | $89.43_{\pm 0.22}$ | $63.42_{\pm 0.30}$ |
| ENZYMES | $52.45_{\pm 0.29}$ | $49.82_{\pm 0.67}$ | $53.75_{\pm 0.34}$ | $51.03_{\pm 0.42}$ | $60.97_{\pm 0.19}$ | $53.94_{\pm 0.66}$ | $62.73_{\pm 0.21}$ | $63.09_{\pm 0.26}$ | $62.44_{\pm 0.38}$ | $61.75_{\pm 0.10}$ | $63.59_{\pm 0.11}$ | $63.12_{\pm 0.52}$ | $63.44_{\pm 0.30}$ | $68.56_{\pm 0.43}$ | $64.58_{\pm 0.57}$ | $52.10_{\pm 0.65}$ |
| PROTEINS | $49.43_{\pm 0.69}$ | $61.24_{\pm 0.34}$ | $53.85_{\pm 0.26}$ | $65.75_{\pm 0.35}$ | $61.15_{\pm 0.11}$ | $52.78_{\pm 0.58}$ | $72.61_{\pm 0.64}$ | $72.60_{\pm 0.20}$ | $76.46_{\pm 0.13}$ | $77.29_{\pm 0.41}$ | $72.89_{\pm 0.57}$ | $75.86_{\pm 0.30}$ | $77.43_{\pm 0.19}$ | $76.49_{\pm 0.29}$ | $76.02_{\pm 0.17}$ | $68.81_{\pm 0.33}$ |
| DD | $47.69_{\pm 0.24}$ | $75.29_{\pm 0.46}$ | $47.98_{\pm 0.32}$ | $70.49_{\pm 0.28}$ | $70.33_{\pm 0.13}$ | $42.67_{\pm 0.59}$ | $76.43_{\pm 0.45}$ | $65.41_{\pm 0.60}$ | $79.08_{\pm 0.19}$ | $80.76_{\pm 0.50}$ | $77.76_{\pm 0.48}$ | $74.53_{\pm 0.11}$ | $76.54_{\pm 0.25}$ | $78.84_{\pm 0.40}$ | $79.91_{\pm 0.65}$ | $60.49_{\pm 0.17}$ |
| BZR | $46.67_{\pm 0.52}$ | $59.08_{\pm 0.29}$ | $51.16_{\pm 0.36}$ | $51.71_{\pm 0.45}$ | $41.50_{\pm 0.24}$ | $45.42_{\pm 0.34}$ | $68.93_{\pm 0.63}$ | $67.81_{\pm 0.32}$ | $69.13_{\pm 0.13}$ | $68.55_{\pm 0.15}$ | $51.89_{\pm 0.46}$ | $80.79_{\pm 0.38}$ | $68.23_{\pm 0.31}$ | $77.69_{\pm 0.28}$ | $73.28_{\pm 0.10}$ | $65.94_{\pm 0.12}$ |
| AIDS | $50.93_{\pm 0.19}$ | $52.01_{\pm 0.53}$ | $52.56_{\pm 0.41}$ | $61.42_{\pm 0.50}$ | $87.20_{\pm 0.18}$ | $97.96_{\pm 0.36}$ | $95.44_{\pm 0.65}$ | $98.80_{\pm 0.32}$ | $96.89_{\pm 0.20}$ | $96.93_{\pm 0.34}$ | $99.36_{\pm 0.67}$ | $97.60_{\pm 0.28}$ | $98.02_{\pm 0.23}$ | $99.21_{\pm 0.27}$ | $97.10_{\pm 0.22}$ | $70.82_{\pm 0.57}$ |
| COX2 | $52.15_{\pm 0.16}$ | $52.48_{\pm 0.38}$ | $53.34_{\pm 0.27}$ | $49.56_{\pm 0.11}$ | $49.11_{\pm 0.26}$ | $48.61_{\pm 0.60}$ | $59.68_{\pm 0.41}$ | $59.38_{\pm 0.69}$ | $57.81_{\pm 0.50}$ | $58.93_{\pm 0.47}$ | $59.81_{\pm 0.30}$ | $72.35_{\pm 0.58}$ | $64.13_{\pm 0.23}$ | $64.36_{\pm 0.16}$ | $63.19_{\pm 0.31}$ | $54.73_{\pm 0.15}$ |
| NCI1 | $51.39_{\pm 0.19}$ | $50.22_{\pm 0.12}$ | $54.18_{\pm 0.67}$ | $50.41_{\pm 0.31}$ | $45.11_{\pm 0.33}$ | $61.88_{\pm 0.23}$ | $43.33_{\pm 0.25}$ | $46.44_{\pm 0.60}$ | $69.46_{\pm 0.36}$ | $65.29_{\pm 0.21}$ | $75.75_{\pm 0.47}$ | $74.32_{\pm 0.34}$ | $68.32_{\pm 0.22}$ | $69.13_{\pm 0.58}$ | $61.58_{\pm 0.40}$ | $58.74_{\pm 0.18}$ |
| DHFR | $48.31_{\pm 0.47}$ | $52.79_{\pm 0.35}$ | $50.30_{\pm 0.31}$ | $51.64_{\pm 0.22}$ | $45.58_{\pm 0.21}$ | $63.15_{\pm 0.24}$ | $58.21_{\pm 0.66}$ | $57.01_{\pm 0.39}$ | $61.09_{\pm 0.27}$ | $61.79_{\pm 0.54}$ | $59.82_{\pm 0.44}$ | $72.87_{\pm 0.28}$ | $61.25_{\pm 0.19}$ | $63.23_{\pm 0.38}$ | $64.48_{\pm 0.13}$ | $53.23_{\pm 0.10}$ |
| IM→IB | $49.80_{\pm 0.33}$ | $51.23_{\pm 0.65}$ | $53.45_{\pm 0.29}$ | $53.03_{\pm 0.14}$ | $56.26_{\pm 0.25}$ | $51.32_{\pm 0.16}$ | $74.45_{\pm 0.19}$ | $78.62_{\pm 0.57}$ | $80.98_{\pm 0.31}$ | $81.25_{\pm 0.68}$ | $66.73_{\pm 0.52}$ | $71.10_{\pm 0.63}$ | $78.28_{\pm 0.23}$ | $80.23_{\pm 0.34}$ | $80.94_{\pm 0.17}$ | $52.67_{\pm 0.12}$ |
| EN→PR | $52.53_{\pm 0.21}$ | $53.36_{\pm 0.31}$ | $53.92_{\pm 0.58}$ | $51.90_{\pm 0.49}$ | $46.01_{\pm 0.42}$ | $33.52_{\pm 0.22}$ | $59.76_{\pm 0.16}$ | $63.23_{\pm 0.40}$ | $61.77_{\pm 0.35}$ | $59.36_{\pm 0.11}$ | $67.18_{\pm 0.61}$ | $62.42_{\pm 0.30}$ | $56.95_{\pm 0.24}$ | $64.31_{\pm 0.56}$ | $63.84_{\pm 0.28}$ | $54.48_{\pm 0.36}$ |
| AI→DH | $51.18_{\pm 0.13}$ | $51.69_{\pm 0.66}$ | $52.28_{\pm 0.21}$ | $50.95_{\pm 0.48}$ | $44.33_{\pm 0.29}$ | $63.27_{\pm 0.47}$ | $97.11_{\pm 0.52}$ | $98.48_{\pm 0.40}$ | $95.68_{\pm 0.26}$ | $94.33_{\pm 0.65}$ | $98.95_{\pm 0.61}$ | $96.82_{\pm 0.37}$ | $95.42_{\pm 0.17}$ | $99.10_{\pm 0.12}$ | $99.27_{\pm 0.18}$ | $94.58_{\pm 0.63}$ |
| BZ→CO | $43.34_{\pm 0.58}$ | $52.43_{\pm 0.14}$ | $49.76_{\pm 0.47}$ | $52.16_{\pm 0.54}$ | $64.29_{\pm 0.20}$ | $64.65_{\pm 0.31}$ | $78.98_{\pm 0.44}$ | $76.01_{\pm 0.62}$ | $87.27_{\pm 0.32}$ | $80.55_{\pm 0.69}$ | $81.86_{\pm 0.29}$ | $89.11_{\pm 0.37}$ | $83.21_{\pm 0.66}$ | $96.32_{\pm 0.17}$ | $95.16_{\pm 0.49}$ | $65.26_{\pm 0.24}$ |
| ES→MU | $52.99_{\pm 0.25}$ | $52.63_{\pm 0.13}$ | $52.13_{\pm 0.20}$ | $52.28_{\pm 0.42}$ | $58.12_{\pm 0.26}$ | $51.57_{\pm 0.65}$ | $78.66_{\pm 0.17}$ | $79.66_{\pm 0.32}$ | $86.70_{\pm 0.36}$ | $90.55_{\pm 0.47}$ | $88.32_{\pm 0.44}$ | $91.43_{\pm 0.69}$ | $89.30_{\pm 0.55}$ | $92.41_{\pm 0.14}$ | $91.98_{\pm 0.39}$ | $75.65_{\pm 0.12}$ |
| TO→SI | $53.73_{\pm 0.34}$ | $51.87_{\pm 0.67}$ | $53.50_{\pm 0.18}$ | $52.25_{\pm 0.64}$ | $64.32_{\pm 0.22}$ | $66.53_{\pm 0.60}$ | $66.85_{\pm 0.38}$ | $64.85_{\pm 0.57}$ | $67.29_{\pm 0.29}$ | $69.80_{\pm 0.50}$ | $68.91_{\pm 0.52}$ | $66.72_{\pm 0.33}$ | $72.51_{\pm 0.20}$ | $68.24_{\pm 0.47}$ | $66.70_{\pm 0.13}$ | $72.34_{\pm 0.15}$ |
| BB→BA | $54.15_{\pm 0.61}$ | $53.11_{\pm 0.48}$ | $54.62_{\pm 0.55}$ | $53.48_{\pm 0.12}$ | $63.27_{\pm 0.39}$ | $32.37_{\pm 0.30}$ | $69.18_{\pm 0.22}$ | $67.33_{\pm 0.37}$ | $78.83_{\pm 0.46}$ | $77.69_{\pm 0.14}$ | $78.93_{\pm 0.60}$ | $89.88_{\pm 0.25}$ | $79.07_{\pm 0.17}$ | $80.17_{\pm 0.44}$ | $81.44_{\pm 0.11}$ | $55.69_{\pm 0.67}$ |
| PT→MU | $51.52_{\pm 0.52}$ | $55.87_{\pm 0.19}$ | $54.03_{\pm 0.28}$ | $52.12_{\pm 0.40}$ | $55.88_{\pm 0.33}$ | $53.78_{\pm 0.15}$ | $78.10_{\pm 0.58}$ | $79.74_{\pm 0.65}$ | $79.27_{\pm 0.18}$ | $77.54_{\pm 0.66}$ | $62.51_{\pm 0.50}$ | $84.63_{\pm 0.24}$ | $80.12_{\pm 0.27}$ | $79.44_{\pm 0.36}$ | $82.05_{\pm 0.47}$ | $58.28_{\pm 0.61}$ |
| FS→TC | $50.06_{\pm 0.55}$ | $54.76_{\pm 0.25}$ | $51.98_{\pm 0.53}$ | $53.24_{\pm 0.64}$ | $44.98_{\pm 0.27}$ | $49.57_{\pm 0.66}$ | $67.05_{\pm 0.22}$ | $66.01_{\pm 0.30}$ | $66.98_{\pm 0.13}$ | $68.92_{\pm 0.36}$ | $64.38_{\pm 0.18}$ | $78.12_{\pm 0.60}$ | $67.32_{\pm 0.29}$ | $69.89_{\pm 0.41}$ | $71.58_{\pm 0.50}$ | $60.12_{\pm 0.38}$ |
| CL→LI | $50.85_{\pm 0.47}$ | $51.74_{\pm 0.55}$ | $52.66_{\pm 0.30}$ | $51.54_{\pm 0.18}$ | $55.62_{\pm 0.45}$ | $56.45_{\pm 0.11}$ | $59.65_{\pm 0.33}$ | $54.17_{\pm 0.48}$ | $61.21_{\pm 0.27}$ | $58.31_{\pm 0.50}$ | $59.30_{\pm 0.37}$ | $72.15_{\pm 0.69}$ | $63.42_{\pm 0.21}$ | $70.21_{\pm 0.14}$ | $69.28_{\pm 0.29}$ | $50.79_{\pm 0.66}$ |
| HIV-Size | $48.94_{\pm 0.33}$ | $49.96_{\pm 0.44}$ | $66.11_{\pm 0.56}$ | $45.10_{\pm 0.12}$ | $31.39_{\pm 0.25}$ | $32.67_{\pm 0.37}$ | $26.73_{\pm 0.50}$ | $35.80_{\pm 0.29}$ | $38.55_{\pm 0.15}$ | $42.94_{\pm 0.40}$ | $96.34_{\pm 0.67}$ | $91.86_{\pm 0.23}$ | $47.56_{\pm 0.65}$ | $56.23_{\pm 0.26}$ | $74.12_{\pm 0.39}$ | $68.31_{\pm 0.30}$ |
| ZINC-Size | $48.66_{\pm 0.27}$ | $50.12_{\pm 0.35}$ | $50.58_{\pm 0.18}$ | $48.96_{\pm 0.12}$ | $53.07_{\pm 0.50}$ | $53.47_{\pm 0.66}$ | $52.61_{\pm 0.54}$ | $53.46_{\pm 0.29}$ | $54.22_{\pm 0.34}$ | $54.21_{\pm 0.41}$ | $59.41_{\pm 0.56}$ | $57.29_{\pm 0.45}$ | $52.53_{\pm 0.30}$ | $58.78_{\pm 0.64}$ | $68.43_{\pm 0.69}$ | $55.63_{\pm 0.24}$ |
| HIV-Scaffold | $49.36_{\pm 0.65}$ | $48.43_{\pm 0.42}$ | $44.72_{\pm 0.11}$ | $54.57_{\pm 0.67}$ | $58.78_{\pm 0.13}$ | $58.74_{\pm 0.20}$ | $61.00_{\pm 0.62}$ | $59.26_{\pm 0.40}$ | $56.82_{\pm 0.35}$ | $54.38_{\pm 0.30}$ | $58.05_{\pm 0.24}$ | $70.93_{\pm 0.36}$ | $63.23_{\pm 0.22}$ | $59.49_{\pm 0.17}$ | $62.28_{\pm 0.33}$ | $53.48_{\pm 0.69}$ |
| ZINC-Scaffold | $51.12_{\pm 0.50}$ | $46.66_{\pm 0.26}$ | $51.17_{\pm 0.34}$ | $53.28_{\pm 0.65}$ | $54.77_{\pm 0.14}$ | $55.17_{\pm 0.38}$ | $54.04_{\pm 0.22}$ | $55.80_{\pm 0.21}$ | $51.87_{\pm 0.57}$ | $50.12_{\pm 0.53}$ | $56.79_{\pm 0.16}$ | $59.24_{\pm 0.40}$ | $55.79_{\pm 0.47}$ | $55.73_{\pm 0.67}$ | $56.39_{\pm 0.24}$ | $55.62_{\pm 0.45}$ |
| IC50-Size | $67.08_{\pm 0.23}$ | $59.35_{\pm 0.17}$ | $90.76_{\pm 0.11}$ | $52.49_{\pm 0.37}$ | $32.61_{\pm 0.32}$ | $36.15_{\pm 0.22}$ | $24.44_{\pm 0.64}$ | $30.96_{\pm 0.14}$ | $42.58_{\pm 0.41}$ | $41.29_{\pm 0.52}$ | $97.36_{\pm 0.13}$ | $81.43_{\pm 0.45}$ | $39.63_{\pm 0.30}$ | $50.37_{\pm 0.56}$ | $50.71_{\pm 0.29}$ | $45.24_{\pm 0.67}$ |
| EC50-Size | $70.50_{\pm 0.33}$ | $59.33_{\pm 0.21}$ | $92.29_{\pm 0.34}$ | $49.36_{\pm 0.19}$ | $29.71_{\pm 0.61}$ | $32.60_{\pm 0.26}$ | $22.83_{\pm 0.30}$ | $27.68_{\pm 0.15}$ | $41.37_{\pm 0.12}$ | $39.42_{\pm 0.40}$ | $97.74_{\pm 0.27}$ | $77.12_{\pm 0.36}$ | $39.23_{\pm 0.24}$ | $55.84_{\pm 0.69}$ | $56.58_{\pm 0.42}$ | $59.49_{\pm 0.18}$ |
| IC50-Scaffold | $66.33_{\pm 0.49}$ | $60.95_{\pm 0.55}$ | $88.49_{\pm 0.12}$ | $58.56_{\pm 0.30}$ | $36.96_{\pm 0.28}$ | $38.67_{\pm 0.66}$ | $34.60_{\pm 0.21}$ | $38.82_{\pm 0.33}$ | $44.56_{\pm 0.22}$ | $42.97_{\pm 0.57}$ | $96.00_{\pm 0.60}$ | $77.58_{\pm 0.47}$ | $38.43_{\pm 0.10}$ | $57.96_{\pm 0.19}$ | $56.62_{\pm 0.31}$ | $59.68_{\pm 0.40}$ |
| EC50-Scaffold | $64.95_{\pm 0.21}$ | $62.78_{\pm 0.38}$ | $86.03_{\pm 0.18}$ | $55.27_{\pm 0.63}$ | $27.88_{\pm 0.34}$ | $30.36_{\pm 0.60}$ | $31.28_{\pm 0.23}$ | $32.35_{\pm 0.41}$ | $51.92_{\pm 0.19}$ | $48.43_{\pm 0.50}$ | $94.18_{\pm 0.14}$ | $74.65_{\pm 0.64}$ | $40.98_{\pm 0.30}$ | $66.52_{\pm 0.11}$ | $58.29_{\pm 0.56}$ | $54.24_{\pm 0.29}$ |
| IC50-Assay | $54.47_{\pm 0.46}$ | $53.13_{\pm 0.22}$ | $59.18_{\pm 0.30}$ | $52.11_{\pm 0.28}$ | $51.23_{\pm 0.40}$ | $51.42_{\pm 0.13}$ | $51.15_{\pm 0.17}$ | $51.93_{\pm 0.24}$ | $52.61_{\pm 0.52}$ | $49.87_{\pm 0.48}$ | $68.78_{\pm 0.37}$ | $65.99_{\pm 0.57}$ | $52.12_{\pm 0.26}$ | $54.25_{\pm 0.45}$ | $53.74_{\pm 0.50}$ | $57.11_{\pm 0.55}$ |
| EC50-Assay | $49.08_{\pm 0.41}$ | $46.66_{\pm 0.65}$ | $48.43_{\pm 0.23}$ | $45.32_{\pm 0.19}$ | $47.80_{\pm 0.44}$ | $47.81_{\pm 0.66}$ | $47.80_{\pm 0.55}$ | $46.02_{\pm 0.52}$ | $56.11_{\pm 0.21}$ | $52.44_{\pm 0.28}$ | $69.31_{\pm 0.31}$ | $82.97_{\pm 0.30}$ | $54.34_{\pm 0.13}$ | $66.71_{\pm 0.40}$ | $65.57_{\pm 0.17}$ | $58.96_{\pm 0.11}$ |
| Avg. AUROC | 52.69 | 53.67 | 57.56 | 52.91 | 52.11 | 50.35 | 61.29 | 61.50 | 66.00 | 64.29 | 73.14 | 75.86 | 65.50 | 71.07 | 71.05 | 60.38 |
| Avg. Rank | 12.09 | 11.94 | 10.46 | 11.77 | 12.06 | 11.65 | 8.14 | 8.31 | 6.69 | 8.23 | 4.91 | 3.34 | 7.14 | 4.51 | 4.54 | 10.14 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1) How do the main findings of the paper contribute to future research on GLAD and/or GLOD?

2) What is the motivation for unifying GLAD and GLOD?

3) How can the findings in the paper be impacted by the dominance of molecular graphs in the benchmark?

4) How are hyperparameters set in the experiments?
soundness: 3
strengths: - The paper is well-written and easy to follow
- The benchmark proposed in the paper contains 35 datasets
- The datasets and implementations are shared as open-source
summary: This experimental paper proposes unifying two graph machine learning tasks, graph anomaly detection (GLAD) and graph out-of-distribution detection (GLOD) in the unsupervised setting. The paper applies 35 datasets with different properties (anomaly/out-of-distribution scenarios) and applications. Moreover, the evaluation considered 18 methods proposed for anomaly detection and OOD detection that are evaluated in terms of accuracy, generalization, and efficiency. Some of the major findings in the paper are: (1) GNN-based methods achieve the best results on average even though there is no single best method, (2) the methods often struggle with near-OOD scenarios compared with far-OOD ones. and (3) some of end-to-end (GNN-based) methods are more efficient and accurate than the alternatives.
weaknesses: - The main findings of the paper are mostly expected
- It is not clear why the same methods should solve anomaly detection and out-of-distribution detection
- Most of the datasets are from the same domain

Detailed comments:

These are the main findings quoted from the paper:

1) "The SOTA GLAD/GLOD methods show excellent performance on both tasks"
2) "No universally superior method"
3) "Inconsistent performance in terms of different metrics"
4) "End-to-end methods show consistent superiority over two-step methods"
5) "Near-OOD samples are harder to detect compared to far-OOD samples"
6) "Poor generalizability of several GLAD/GLOD methods in specialized scenarios"
7) "Performance degradation with increasing contamination ratio"
8) "The sensitivity of different methods/datasets can be diverse"
9) "Certain end-to-end methods outperform two-step methods in terms of both performance and computational costs"

The only findings that one could find unexpected are 1 and maybe 9. Finding 3 is expected because of class imbalance. I am not diminishing the effort that the authors have put into running these experiments but the value of an experimental paper is what can be learned from the results. It is unclear how much can be learned from these results.

Regarding the unification of the tasks, I would like to see a more clear formalization than definition 1. One could argue that an anomaly can still be in-distribution (as points can be simply ranked in terms of anomaly scores). On the other hand, an OOD point might not be an anomaly (as there might be many similar points in the test dataset). The setting considered in the experiments seems to assume that the test data points are unseen (besides being unlabelled) but if every point can be observed, the difference between an anomaly and an OOD point should be clear. This setting needs better motivation.

From Table 1, only 8/35 datasets are not molecules. This can bias the findings towards methods that perform well on molecules, which is something not discussed in the paper. 

Minor comments:
- How are the hyperparameters of the methods set without a validation set? I was not able to understand this from Appendix D.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: Please check the weak points.
soundness: 4
strengths: 1. this manuscript is well written and easy to follow;
2. I believe this is the first work that attempts to unify and benchmark GLAD and GLOD tasks, the findings could be broadly interesting to the graph data mining community;
3. the authors conducted extensive experiments and open-sourced their code base (which I believe is easy to extend), making contributions to future research;
summary: This manuscript presents a unified benchmark called UB-GOLD, which aims to unify and benchmark Graph Level Anomaly Detection (GLAD) and Graph lavel OOD detection (GLOD). Specifically, the authors compare the performance of 18 GLAD/GLOD methods on 35 datasets. The performance is investigated in four different dimensions: effectiveness (using three common accuracy metrics), generalisability, robustness, and efficiency (time and memory usage). Based on extensive experiments, the authors provide insightful observations and discuss possible future directions.
weaknesses: ## Major Comments

### 1. Unification and Benchmarking of GLAD and GLOD Tasks
As the first work attempting to unify (please indicate the related work if this is not true) and benchmark GLAD (Graph-Level Anomaly Detection) and GLOD (Graph-Level Out-of-Distribution) detection tasks, I believe the authors should dedicate more space and effort to this unification process. Specifically:

- **1.1) Formal Definitions**: It would be beneficial for the authors to formally (with math symbols) define GLAD and GLOD before introducing the concept of "Unsupervised generalized graph-level OOD detection." Providing these definitions upfront would establish clarity and context for readers unfamiliar with these tasks.

- **1.2) Relationship Between GLAD and GLOD**: The authors should discuss the relationship between GLAD and GLOD in more depth, as there seems to be room for clarification on whether they perceive OOD detection as a broader concept than anomaly detection. In my view, anomaly detection is a broader concept, as abnormal instances may either (1) exist within the distribution but in low-density regions or (2) represent out-of-distribution instances. Including a specific section comparing and contrasting GLAD and GLOD, with formal definitions and discussions of their conceptual overlap and distinctions, would enhance the unification effort and clarify this relationship.

---

### 2. Experimental Details and Completeness
There are some missing details regarding the experiments and results. Adding these would make the manuscript more self-contained, especially as it aims to serve as a benchmark paper.

- **2.1) Definition and Generation of Anomalies/OOD Samples**: To make the manuscript self-contained, please provide details in the appendix on how the anomalies or OOD samples are defined or generated in each dataset. Clear explanations will help readers replicate and understand your experiments.

- **2.2) Completeness of Results in Figures**: In Figure 4, results are only provided for 15 out of 19 methods, with similar omissions in Figures 5 and 6. If these omissions are due to space limitations, including the full results in the appendix would help ensure completeness. Please also add a brief discussion explaining why certain methods were omitted from the main figures, if applicable.

- **2.3) Metrics and Additional Results**: In Figures 4 and 5, please specify which metrics are being used. Additionally, including complete results for all metrics in the appendix, along with brief analyses, would make the benchmark more comprehensive and allow for a better understanding of the methods' performances.

---

### 3. Clarification of Experimental Settings and Terminology

- **3.1) Hyperparameter Search**: In the hyperparameter search section, the authors conduct a random search to optimize hyperparameters based on **test set performance**. This approach raises questions regarding practical applicability. Could you clarify if a validation set was used instead of the test set for hyperparameter tuning, or if there was a specific reason for using the test set? Also, discussing the implications of this choice on the generalizability of your results would provide valuable context.

- **3.2) Use of the Term "Generalisability"**: The term "generalisability" is typically associated with a model’s performance on unseen data, while the manuscript uses it to describe differences in OOD detection for near- and far-OOD samples. Alternative terms, such as "OOD detection range" or "OOD sensitivity spectrum," may better capture the intended meaning. Clarifying or refining this terminology would enhance reader understanding.

---

## Minor Comments and Suggestions

- **4.1) Page 4, Line 084: Typo**: Please correct "28 GLAD and GLOD methods" to "18 GLAD and GLOD methods."

- **4.2) Anomaly Ratio in Table 1**: In Table 1, adding the anomaly ratio for each dataset would make the data overview more informative.

- **4.3) Visualization of CPU and GPU Results in Figure 6**: In Figure 6, consider superimposing the CPU and GPU bars (distinguished by color) to make the visual comparison clearer.

- **4.4) OOD Judge Score Distribution Analysis**: On Page 5, you mention an "OOD Judge Score Distribution Analysis," but this concept was not referenced later in the main text, though it appears in the appendix. Including a reference in the main text would clarify its relevance.

- **4.5) "Well-trained GNNs" on Page 5**: On Page 5, the term "well-trained GNNs" is used. If this refers to pre-trained GNNs, please provide a brief explanation for clarity.

---

## Conclusion
Overall, I appreciate the authors’ efforts in developing a unified framework and benchmark for GLAD and GLOD tasks. Addressing these points would improve the manuscript’s clarity, depth, and completeness, making it a more robust resource for the community. And I will consider increasing my rating if they (or some of them) are well addressed.

---

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What is the difference between near-OOD samples and anomalies if both come from the same dataset?
2. If some OOD samples or anomalies are integrated into the training set as contamination, are they still retained in the test set during inference?
soundness: 3
strengths: 1. A large amount of datasets are employed to establish the benchmark to fairly compare the mutiple GLAD and GLOD emthods under a unified experimental setting.
2. In addition to the performance, the paper also investigates more characteristics of methods including generalizability, robustness, and efficiency.
3. The comprehensive benchmark and the multiple observations are provided in the experimental analysis to inspire future work.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
4. There is no clear distinction between the training set and test set in unsupervised GLAD. Leveraging unsupervised graph-level anomaly detection methods for OOD detection is a good point.
summary: This paper unifies the unsupervised  graph level ood and anomaly detection as generalized graph level ood detection problem and conduct a comprehensive and unified benchmark. This paper provide some remarkable observation through the comprehensive comparison and analyses.
weaknesses: 1. In GLAD, anomalies can be considered as out-of-distribution (OOD) test samples for type 1 and type 2, but further clarification is needed in the paper on how the in-distribution (ID) test samples are constructed.
2. In an unsupervised setting, does this mean the test set can also be used as the training set? The in-distribution samples in the test set could potentially help improve performance.
3. From the several observations, we found there are large overlaps between OOD samples,  in addition to the experimental explanation, more specific analysis is needed on this point for particular datasets.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: ①Performance of SSL-D and GK-D on Graph Structure-Based Datasets: In datasets like IC50-Size, where out-of-distribution samples primarily differ from in-distribution ones in graph structure, SSL-D and GK-D methods show poor performance. Could you provide insights into why these methods struggle with such datasets? Is it related to the specific way these methods handle graph structure?

②Handling Near-OOD Samples: You mention that near-OOD samples are more difficult to detect than far-OOD samples. Could you elaborate on what specific characteristics of near-OOD samples make them harder to distinguish? Do you have any thoughts on how future work could address this issue?
soundness: 4
strengths: ①The paper makes an original contribution by unifying GLAD and GLOD into a single benchmark, UB-GOLD. This creative combination highlights the conceptual overlap between the two tasks, simplifying evaluation in both areas.

②The paper's experiments evaluate 18 methods on 35 datasets. It offers a multi-dimensional analysis. The breadth of datasets used ensures reliable, real-world applicability, while the open-source codebase.

③UB-GOLD offers insights into detecting near-OOD samples and noisy training data. These findings current limitations in existing methods. What's more, this work focus on generalizability, robustness, and efficiency ensures the benchmark’s practical relevance across different application domains, making it a useful resource in the field.
summary: The paper introduces UB-GOLD (Unified Benchmark for unsupervised Graph-level OOD and anomaly Detection), a comprehensive benchmark that unifies two related tasks in graph machine learning: graph-level anomaly detection (GLAD) and graph-level out-of-distribution detection (GLOD). UB-GOLD bridges this gap by providing a unified evaluation across 18 representative methods and 35 datasets, covering various scenarios in anomaly and OOD detection. The benchmark offers a multi-dimensional analysis of methods, assessing their effectiveness, generalizability, robustness, and efficiency, while also providing an open-source codebase to facilitate reproducible research and encourage further exploration.
weaknesses: ①While the paper provides extensive comparisons across methods, it could benefit from a more in-depth discussion of why certain methods fail in specific scenarios. For instance, understanding the root causes of poor performance under noisy training data or near-OOD conditions.

②Consider adding a more concise summary section that highlights the most important findings from your multi-dimensional analyses. This could be in the form of a bulleted list or a short paragraph at the end of each results subsection to help readers quickly digest key insights.

③In datasets like IC50-Size, the out-of-distribution samples differ primarily in graph structure from the in-distribution ones. It’s unexpected that SSL-D and GK-D methods perform poorly on these datasets. Could the authors offer an explanation for these results?

Meta-review: This paper proposes an unified benchmark for graph-level anomaly detection / out of distribution detection (UB-GOLD).  The authors observe overlaps between the disparate communities for GLAD and GLOD and propose this benchmark to unify evaluation and benchmarking across 35 datasets and several detection scenarios.  

Reviewers were unilaterally positive about this work and consistently rated it positively.  Nonetheless, there were a few limitations raised:

- Some remaining lack of clarity from reviewers around why GLAD and GLOD are conceptually overlapping problems, related some confusing missing definitions (1svq, 2xER)

- Confusions around validity of using test-set performance for hyperparameter evaluation and the train/test setup in general (1svq, 2xER, yW1o)

- More depth of discussion around the difficulty of near-OOD samples and impact of noise during training (cQQu, yW1o)

I encourage the authors to take these considerations into the revision.

Decision: Accept (Poster)