Unifying Unsupervised Graph-Level Anomaly Detection and Out-of-Distribution Detection: A Benchmark
Authors: Yili Wang
Yixin Liu
Xu Shen
Chenyu Li
Rui Miao
Kaize Ding
Ying Wang
Shirui Pan
Xin Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Graph out-of-distribution detection; graph anomaly detection; benchmark
Primary area: datasets and benchmarks

Abstract: To build safe and reliable graph machine learning systems, unsupervised graph-level anomaly detection (GLAD) and unsupervised graph-level out-of-distribution (OOD) detection (GLOD) have received significant attention in recent years. Though these two lines of research share the same objective, they have been studied independently in the community due to distinct evaluation setups, creating a gap that hinders the application and evaluation of methods from one to the other. To bridge the gap, in this work, we present a Unified Benchmark for unsupervised Graph-level OOD and anomaly Detection (UB-GOLD), a comprehensive evaluation framework that unifies GLAD and GLOD under the concept of generalized graph-level OOD detection. Our benchmark encompasses 35 datasets spanning four practical anomaly and OOD detection scenarios, facilitating the comparison of 18 representative GLAD/GLOD methods. We conduct multi-dimensional analyses to explore the effectiveness, generalizability, robustness, and efficiency of existing methods, shedding light on their strengths and limitations. Furthermore, we provide an open-source codebase of UB-GOLD to foster reproducible research and outline potential directions for future investigations based on our insights.

Published as a conference paper at ICLR 2025
UNIFYING UNSUPERVISED GRAPH-LEVEL ANOMALY DETECTION AND DUT-OF-DISTRIBUTION DETECTION: A BENCHMARK
Yili Wang" Yixin Liu Xu Shen Chenyu Li Kaize Ding', Rui Miao', Ying Wang'. Shirui Pan Xin Wang"+
Jilin University, Griffith University, NNottwwestern University
(wangy121, shenxu23, chenyul23, ruimiao20; @mails jlu. edu, cn, syixin.liu, s ..pan/8griffith.co au, kaize. saizze.cingenortweettes edu,
[wangying2010, xinwang) @ flj.llu.d
ABSTRACT
To build safe and reliable graph machine learning systems, unsupervised graph- level anomaly detection (GLAD) and unsupervised graph-level out-of-distribttion (OOD) detection (GLOD) have received significant attention in recent years. Though those two lines research indeed share the same objective, they have been studied independently in the community due to distinct evaluation setups, creating a gap that hinders the application and evaluation of methods from one to the other. To bridge the gap, in this work, we present Unified Benchmark for unsupervised Graph-level OOD and anomaly Detection (UB-GOLD), a com- prehensive evaluation framework that unifies GLAD and GLOD under the cept of generalized graph-level OOD detection. Our benchmark encompasses 35 datasets spanning four practical anomaly and OOD detection scenarios, fa- cilitating the comparison of 18 representative GLAD/GLOD methods. We con- duct multiidiimensoona analyses explore the effectiveness, OOD sensitivity spectrum, robustness, and efficiency of existing methods, shedding light on their strengths and limitations. Furthermore, we provide an open-source codebase (https: //gi thub. com HTtm//BBGGOO///OO GOLDDUB--DOOD of UB-GOLD to foster re- producible research and outline potential directions for future investigations based our insights.
INTRODUCTION
With the ubiquity of graph data, graph machine learning has been widely adopted in various scientific and industrial fields, ranging from bioinformatics to social networks (Xia et al., 2021 Wu et al.
2020
Zhang et al. 2024b Wang et 2024b Zhang et al. 2024a Wang al. 2022] uan et al. 2024 Miao et al. 2024 Liu et al. [224]] Ding et al.) 2024a). As one of the representative graph learning tasks, graph-level anomaly detection (GLAD) has been widely studied to identify abnormal graphs that show significant dissimilarity from the majority of graphs in a collection (Ma et al. 2022 Zhang al. 2022)). GLAD task crucial in various real-world applications, such as toxic molecule recognition and pathogenic brain mechanism discovery (Jiang al. 2021 Liu et al.). Due to the high cost of data labeling, existing GLAD studies generally follow an unsupervised paradigm, eliminating
the requirement of labeled anomaly samples for model training (Ma al. 2022 Qiu al. et al. 20236 Pan et al. 2023).
2022 Liu
In the meantime, another line of research graph-level out-of- detection (GLOD) has
drawn increasing attention in the research community lately (Liu et al.) 2023a Li et al. 2022 Wu
al. 2024). GLOD aims to identify whether each graph sample in test is in-distribution (ID), meaning it comes from the same distribution as the training data, or out-of-distributton (OOD),
indicating comes from different distributions. Considering the universality of distribution shift
in open-world data, GLOD plays an important role in real-world high-stakes applications such as
drug discovery and cyber-attack detection (Shen et al., 20224] Ji et al.) al.]]2022 [023] JJu al. 2024 Ding
et al. 2024b Li et al. 2024). Though a few post-hoc GLOD methods (Guo et al. 2023 Wang et al.
"Equal Contribution.
1 "Correponndigg Authors.

---

Published as a conference paper at ICLR 2025
Generalized Graph-Level Dut-of-Diitributioo Detection
Graph-Level Anomaly Detection (GLAD)
Graph-Level Out-of-Distribution Detection (GLOD)
Intrinsic Anomaly (Type I)
Class-based Anomaly
Inter- Inter-Dataset Shift (Type III)
Intra-Dataset Shift
(Type II)
(Type IV)
M
Class
ID
Non-toric malecules
ID
Daiaset
Scaflold 1015/1016
ID
Class
ID
CHa
Toxic molecules
OOD
Class
OOD
Dataset
OOD
Scaffold 29007/29008
OOD
Figure 1: Diagram of generalized OOD detection scenarios supported by UB-GOLD.
2024a) that rely well-trained graph classifier have been proposed, most of the existing GLOD methods (Liu et al. 2023a Li et al. 2022 Shen et al., (224) are developed in an unsupervised fashion, unlabeled ID data, and then predicts score for each test sample to indicate its ID/OOD status. In essence, unsupervised GLOD and unsupervised GLAD share the same goal, suggesting that research in one area could potentially be applied to solve problems in the other. Recent survey and benchmark papers (Yang et al., 2022] 2021 also identify anomaly detection and OOD detection as two branches under the concept of "generalized OOD detection", which further highlights the close conceptual connection between GLOD and GLAD
where specific OOD detection model is trained
tasks.
Despite the inherent connection between the problems of unsupervised GLAD and GLOD, those two research sub-areas have been studied independently in the literature and have distinct evaluation setups. To fill the gap, in this paper, we develop a Unified Benchmark for unsupervised Graph-level OOD and anomalyy Detection (UB-GOLD for short). As shown in Fig. [ we unify GL.AD and GLOD as "generalized graph-level OOD detection problem". and consider two GLOD scenarios and two GLAD scenarios for comprehensive evaluation. For GLAD task that aims to detect graph samples with semantic abnormality, we explore scenarios with intrinsic anomaly and class-based anomaly. Specifically, the first type of dataset inherently includes semantic anomalies, while the second type treats samples from one class (usually the minority class) as anomalies. For GLOD task that emphasizes the distribution shift between ID and OOD samples, we consider two scenarios with different distribution shifts, i.e., inter-dataset shift and intra-dataset shift. Specifically, the former simulates distribution shifts by drawing ID and OOD samples from different datasets within the same domain (Liu et al.) 2023a [Sehwag et al., 2021), while the latter refers to datasets with intrinsic distribution shilts regarding graph sizes or molecular scaffolds (Gui et al. (022). Based on 35 datasets belonging to 4 types, we establish comprehensive benchmark to fairly compare 18 GLAD and GLOD methods under a unified experimental setting, exploring the effectiveness of state-of-the-art (SOTA) approaches across diverse scenarios and domains.
Apart from comparing performance, we further investigate the characteristics of GLAD/GLOD methods in terms three dimensions: OOD sensitivity spectrum, robustness, and efficiency, through extensive experiments. For OOD sensitivity spectrum, test a GLAD/GLOD model on OOD data with varying degrees of distribution shift, investigating each method's s ability to identify both near- OOD and far-OOD samples. For robustness, the training data to observe the impact of noisy ID data on the performance of existing methods. For efficiency, we perform efficiency evaluations for representative GLAD/GLOD approaches, focusing on time and space complexity. At the end of this paper, we discuss the future directions of this emerging research direction. To sum up, the main contribution of this paper is three-fold:
introduce varying proportions of OOD samples into
Comprehensive benchmark. We introduce UB-GOLD, the first comprehensive and unified benchmark for GLAD and GLOD. UB-GOLD compares 18 representative GLAD/GLOD methods across 35 datasets from four practical anomaly and OOD detection scenarios.
Multi-dimensional analysis. To explore their ability and limitations, we conduct a systematic analysis of existing methods from multiple dimensions, encompassing effectiveness on different datasets, OOD sensitivity spectrum to near-OOD and far-OOD, robustness against unreliable training data, and efficiency in terms of time and memory usage.
Unified Codebase and future directions. To facilitate future research and quick implementation, we provide an open-source codebase for UB-GOLD. We also outline potential directions based on our findings to inspire future research.
2

---

Published as a conference paper at ICLR 2025
Key findings. Through comprehensive comparison and analyses, we have the following remarkable observations: 1) The SOTA GLAD/GLOD methods show excellent performance across tasks; 2) Near- OOD samples are harder to detect compared to far- far-OOD samples; 3) Most methods are vulnerable to the perturbations of training sets; and 4) Certain end-to-end methods outperform two-step methods in terms of both performance and computational costs.
RELATED WORK AND PROBLEM DEFINITION
In this section, we begin by briefly reviewing related work on GLAD and GLOD. Next, we define the generalized unsupervised graph-level OOD detection problem, formulating GLAD and GLOD into unified learning task.
Graph-level anomaly detection (GLAD). GLAD aims to identify abnormal graphs that show significant dissimilarity from the majority of graphs in a collection. simple solution for GLAD is to graph kernels (Vishwanathan al., 2010: Neumann et al. 2016) to extract graph-level features and utilize shallow anomaly detectors (e.g. OCSVM (Amer et al. 2013) and iForest (Liu
et al. )008) to identify anomalies. Recently GNN-based end-to-end methods have demonstrated
significant performance improvements in GLAD (Ma et al. 2022 /Zhoo & Akoglu 2023). Among
them, some approaches assume that the anomaly labels of graphs are available for model training, and hence formulate GLAD as supervised classification problem (Zhang et al. [2022] Ma et al. 2023
Dong et al., 2024). Nevertheless, their heavy reliance on labeled anomalies makes it challenging to apply them to real-world scenarios where annotated anomalies are scarce or unavailable. To
overcome this shortage, the majority of GNN-based GLAD methods focus on an unsupervised learning paradigm, learning the GLAD model only from normal data (Ma et al. 2022 Qiu et al.
2022 Liu et al. 2023b Zhao & Akoglu 2023 Luo et al. 2022 Li al. 2023). Recent research (Liu
ct al. 2023a Li et al. 2022) indicates that GLAD methods also show great potential in handling
GLOD problem, prompting us to unify these two fields into a generalized research problem and conduct comprehensive benchmarking study.
Graph-level out-of-distribution detection (GLOD), GLOD refers to the problem of identifying whether a graph sample in test set is ID OOD, i.e., whether it originates from the same distribution as the training data or from a different one (Li et al. 2022). A line of studies developed post-hoc OOD detectors that identify OOD samples by additional fine-tuned detectors on top of well-trained
GNN classifiers (Guo et al. 2023 Wang et al. 2024a). However, these methods require annotated ID data to train the backbone GNNs, which limits their applicability in scenarios where labeled data is unavailable. In contrast, another line of research proposes training an OOD-specific GNN model using only ID data, without relying on any labels or OOD data (Shen et al., 2024: [iu et al. 2023a
Li et al.) 2022). To learn discriminative patterns of ID data, they employ unsupervised learning techniques such as contrastive learning (Liu et al. 2023a and diffusion model (Shen et al.. 2024).
Problem definition. Inspired by generalized OOD detection framework in computer vision (Yag et al. 202T: Gui et al.] (022), and due to the highly similar objectives of GLAD and GL.D (for detailed discussion, see Appendix D.I), unify GLAD and GLOD into high-level topic termed generalized graph-level OOD detection. The new learning task aims to distinguish generalized OOD samples (i.c., OOD or abnormal graphs) from generalized ID samples (i.c.. ID normal graphs). In this paper, we focus on the unsupervised scenario, considering its universality. The research problem is formulated as:
Definition 1. [Unsupervised generalized graph-level OOD detection] We define the training dataset as Drrain = (G1, G..), where each sample G; is graph drawn from specific distribution pprn. We define the testing dataset as Drest = Din. U Doutt where Din contains graphs sampled from pin and Dous consists of graphs drawn from an OOD distribution poou,, Given Ptrain and Dreet, the learning objective is to train a detection model f-)) on Derain and then the model can predict whether each sample G' E Diest belongs to p or poutt practice, f()) is a scoring function, where larger OOD score s' = f(G" indicates a higher probability that G' is from pout (i.e., abnormal samples).
(0 (OOD), if f(") ST.
(G {1 (ID), if f(G') T.
The theoretical objective of the scoring function f(') is to maximize the separation between the distributions of ID/normal graphs and OOD/anomalous graphs:
max WDj f(G') E f(G'..
(2)
3

---

Published as a conference paper at ICLR 2025
Table 1: Statistics of datasets used in UB-GOLD
Type
Fall
Abbrrssition
Delinttinn
#ID Train
+IID
400D
To2l__63
BSE p5s
Molecules
Inkereni Ansmaaly
8063
241
25
0.34
['pe
JHSE
Molecules
fnterent Anomaly
Intitsi: Auomatyy
Molecules
Interent Ancealy
Anseualy
COLLAB
Social Nawwkk
Ussaza Chase
1920
480
530
17.81
IMD9 BINARY
IMDB J REDDIT-B
Social Networks
Usseet Clisses
400
100
100
16.07
REDDIT-BINARY
Social Nawarka
Unaaza Chasos
800
200
200
1687
ENZYMES
Protes
Unseer (Tasses
400
100
20
3.85
Oyee (D)
PROTUNS
Protess
Uneern (Tasees
160
901
135
12.61
Chosssseal Anonaly
DD
Protcise
Uasaza Chasec
590
47
130
22.30
BZK
Molecules
Usseet Clisses
C9
17
64
42.07
AIDS
Mokseslee
Unaaza Chasos
1280
520
60
4.76
COX2
Molecules
Unseea CTisses
21
73
41.71
NCT1
Molsciles
Unsers (Tasses
411
411
16
DHIFR
Molecales
IMD8 MLL IMDB BINARY
Social Networks
Donieets
EXZYMES PROTEINS
Proiciss
Emeen Dutaccis
540
60
9.09
AIDS -DHFR
A3 (1-DH
Molocales
Umeen Dutaciis
1800
200
200
9.09
WZR .CON?
NZ 400
Moleciies
Lnven Donnetss
364
41
41
9.19
(Type MI))
-MIL
ES -M-
Mokezalex
Enuan Datancte
1015
113
115
9.11
TOX21 -SDER
TO-15
Mokscales
BRBP -BACE
+BA
Molecules
FTC -MUTAG
PT PT-MU
Molocales
Umeen Dutaccis
309
35
35
9.23
FRI0ISOL -
FS -TE:
Moleciles
Lnven Doraetts
477
05
9.19
CLINTOX -LJPO
CL- -LI
Molecules
Uneen Duzustis
1329
145
148
9.11
GOOD-HIV-Son
Siee
Molocales
GOOD
SEV-S:alfoll
Molecaes
Scullild
25.00
G000-2IC NC-Saatll
ZINC-Scorliadd
Mokesiles
Scatiold
1000
500
900
35.00
Type IV)
ICS0-Siae
Mokszalex
Dru0OD--EC
EC50-Sixe
Molocales
25.00
Drogt3 NCa0t-catiool
BC50 Scuffold
Staffold
Molecaees
Scullsld
IC50 Avery
Mokesiles
Photrin Varget
500
900
35.00
Dru,OOD EC50 Aasdy
BC50 Aslay
Molecules
Piotcio Target
1000
500
300
25.00
This unified objective integrates the detection principles of GLAD and GLOD, emphasizing their shared goal of distinguishing graphs that deviate from a given distribution.
3 BENCHMARK DESIGN
In this section, we provide a comprehensive overview of UB-GOLD, covering datasets (Sec. 3.1 algorithms (Sec. 3.2), and evaluation metrics & implementation details (Sec. 3.3
3.1 BENCHMARK DATASETS
UB-GOLD includes 35 datasets in four types, each of which corresponds either GLAD GLOD scenario categorized in Fig Table providee an overview of the benchmark datasets. For GLAD task, we consider two types of datasets with intrinsic anomaly and class-based anomaly to test the models' performance of detecting anomalous graphs. For GLOD task, we consider datasets with inter-dataset shift and intra-dataset shift that assess the models' ability distinguish between ID and OOD samples. For more detailed information on the datasets, please see Appendix B
Type I: Datasets with intrinsic anomaly. These datasets contain natural anomalies within chemical compounds, testing the robustness of GLAD methods. We use four datasets (i.e., Tox21 _MM, Tox21_p53, and 1e-21_PPAR-gaae
from the Tox21 I challenge (Abdelaziz et al. 2016 involving molccules with unexpected biological activities.
Type II: Datasets with blass- anomaly. these datasets, certain classes are designated as anomalies. We use 11 datasets (c.g., COLLAB, IMDB- BINARY, and ENZYMES) from the TU benchmark (Morris et al. 2020) where minority or distinct class samples are treated as anomalies.
Type III: Datasets with inter-dataset shift. We synthetic a benchmark dataset with inter-dataset shift by considering samples from one real-world dataset as II) and samples from another real- world dataset as OOD. For example, in "IMDB-MULTI >IMD-BINNAY"" graphs from IMDB-MULTI are ID, and graphs from MNDB-BINARY are OOD. The datasets belonging to the same domain and having close distribution shifts form a pair (Liu et al. 2023a).
Type IV: Datasets with intra-dataset shift. Designed to assess GLOD methods under various types of aatra-dataset shifts, these datasets are from graph OOD benchmarks, including GOOD (Gui et al., 2022) and DrugOOD (Ji et al. (023). Specifically, datasets with three kinds of domain shifts, i.e., assay shift, scaffold shift, and size shift, are considered.
4

---

Published as a conference paper at ICLR 2025
Table 2: Categorization all benchmark algorithms UB-GOLD.
Method Type
Category
Models
Graph kemel with detector (GK -D)
PK-SVM, PK-iE WL-SVM, WI.iF
Two-Step
Self-supervised leuminng with detector (SSL-D)
IG-SVM. IG-IF, GCL-SVM, GCL-iF
Graph neural network--bseed GLAD
OCGIN, GLADC, GLocalKD, OCGTL, SIGNET, CVTGAD
End-to-Ent to---d
Graph neural network-hased GLOOD>
GOOD-D, GrapDEE, AAGOD, GOODAT
3.2 BENCHMARK ALGORITHMS
In UB-GOLD, we consider four groups of GLADIGLOD methods for a comprehensive evaluation: 1) two-step methods, including graph kernel with detector (GK-D) and self-supervised learning (SSL) with detector (SSL-D), and 2) End-to-Er -i---t-dddddd methods, including GNN-based GLAD and GLOD methods. For the end-to-end methods, we further divided them into 4 categories from the technical perspective, including one-class classification-basse based, and well-trained GNN-based. All methods are unsupervised, aligning with the primary focus of UB-GOLD to provide comprehensive and fair evaluation framework. For a detailed summary of the models and further details, please refer to Table 2 and Appendix C
graph reconstruction-base
contrastive learning-
Graph kernel with detector. These methods follow a two-step process: obtaining graph em- beddings using graph kernels and applying outlier detectors in the embedding space. For kernel methods, we consider Weisfeiler-Leman subtree kernel (WL) (Li et al. 2016) and propagation kernel (PK) (Neumann et al. 2016). For detectors, employ isolation forest (iFF) (Liu al. 2008
and one-class SVM (OCSVM, M. SVM for short) (Amer et al. 2013).
SSL with detector. These approaches also follow a two-step process but use SSL methods to obtain graph embeddings. We consider two SSL methods, GraphCL. (GCL for short) You et al. 2020) and InfoGraph (IG for short) (Sun al. 2020) to generate embeddings and use (Liu et al. 2008 and SVM (Amer et al. 2013) detectors.
GNN-based GLAD methods. We consider SOTA methods for GLAD, including OCGIN (Zhao & Akoglu 2023), GLocalKD (Ma et al. 2022). OCGTL (Qiu et al. 2022), SIGNET (Liu et al. 2023b), GLADC (uoo et al., 2022). and CVTGAD (Li et al. 2023). These methods use different
techniques, such as deep one-class classification, contrastive learning, and knowledge distillation, detect anomalies in graph data.
GNN-based GLOD methods. We involve four representatives of unsupervised GLOD methods, i.c.. GOOD-D (Liu et al.] (203a)), GraphDE (Li et al. 2022), AAGOD Guo et al. 2023 and GOODAT (Wang et |l.]| (2024)) for comparison. GOOD-D contrastive learning-basse baased approach, while GraphDE is a generative model-based approach. Both AAGOD and GOODAT operate on well-trained [ AAGOD follows data-centric approach, and GOODAT focuses on test-time OOD detection.
UB-GOLD: A Unified Benchmark Library for Unsupervised GLAD and GLOD. We offer our developed benchmark library UB-GOLD, as illustrated in Fig. [ Specifically, the benchmark starts with a Benchmark Datasets module, which includes four types of datasets: intrinsic anomaly, class-based anomaly, inter-dataset shift, and intra-dataset shift. Fach dataset type is designed to cover both GLAD and GLOD scenarios, ensuring broad applicability across different graph-level anomaly and OOD detection tasks. The Benchmark Tasks module prepares the settings (c.g.. datasets splits and pre-processing) for different benchmarking tasks of anomaly and OOD detection. supports three main tasks: general anomaly/OOD detection, near-far OOD detection, and perturbation- based detection, enabling models to handle different levels of OOD difficulty and assess robustness to data perturbations. This step ensures the data is well-prepared for downstream tasks. Next, in the Method module, UB-GOLD includes both two-step methods and end-to-end methods. Specifically, we categorize end-to-em GLAD and GLOD methods into four technical groups: one-class classification, graph reconstruction, well-trained GNNs, and contrastive learning. Finally, the Evaluation module ensures comprehensive and fair assessment, offering Comprchensive Metrics, Robustness under
In this context, Swell-trained GNNs" refers to graph neural networks that have been pre-trained using self-supervised or task-specific pretraining strategy, incorporating domain knowledge to effectively learn graph representations. These models are then fine-tuned for the specific GL.D GLOD tasks.
5

---

Published as a conference paper at ICLR 2025
Evaluation
Comprchensive Metrics
Robusiness under Perturbation
Generalization on Near- OOD and Far-OOD
0OD Judge Seare Distributions
UB-GOLD
Benchmark Tasks
Two-Step Methods
Graph Kernel with Detector (GK-D)
Self-Supervised Learning Datector (SSL-D)
Anomaly(O0n Detection
Near-Far OOD Detection
Weisfeller-Leman
Subtree Kernel
Propagation Kernel
GraphCL
InfoGraph
Pertorbation-hsse Detrction
Isolation Forest
Defector
One-Class SVM
End-to-End End Methods
Benchmark Datasets
One-Class Classification
Graph Reconstruction
Well-Trained GNNs
Anomaly
OCGIN
GLADC
AAGOD
OCGTL
GraphDE
Class-based Anomaly
Inter-Dataser Shift
Contrastive learning
GLocalKD
GOOD-D
CVTGOD
SIGNET
Intra-Datasel
Figure 2: An overview UB-GOLD.
Perturbation, Generalization Near-OOD and Far-OOD, and OOD Judge Score Distribution Analysis to measure how well models distinguish between ID and OOD samples.
3.3 EVALUATION METRICS AND IMPLEMENTATION DETAILS
Evaluation metrics. For comprehensive comparison, UB-GOLD utilizes three commonly used metrics for anomaly/OOD detection (Zhang al. 2023), i.e., AUROC, AUPRC, and FPR95. Higher AUROC and AUPRC values indicate better performance, while lower FPR95 values are preferable.
Data split. In our target scenarios (i.e., unsupervised GLAD/GLOD) all the samples in the training set are normal/ID, while the anomaly/OOD samples only occur in the testing set. such an unsupervised case, the validation set with anomaly/OOD samples is usually unavailable during the training phase. Thus, following the implementation OpenOOD (Zhang et al. (023), we divide the datasets into training and testing sets, without using validation set. Specifically, we adopted the splits from (Liu et al. 2023a) and (Li et al. 2022), applying them to the benchmark datasets. Detailed splits are provided in Table
Hyperparameter search. To obtain the performance upper bounds various methods on GLAD/GLOD tasks, conduct random search find the optimal hyperparameters w.r.t. their performance on the testing set. The search space is detailed in Table The random search is conducted 20 times for maximum of one day per method per dataset to ensure fairness.
For more details related to the experimental setup in UB-GOLD, please refer to Appendix D
4 EXPERIMENTAL RESULTS & DISCUSSION
In this section, we introduce the experimental setup and discuss the experimental results in UB-
GOLD benchmark. Specifically, we aim to answer the following research questions: RQ1 (Effectiveness): How do different GLAD and GLOD methods perform under various anomaly scenarios and distribution shifts? (Sec.4. 4.1 RQ2 (OOD sensitivity spectrum): How effective
GLAD and GLOD methods in detecting mear-OOO and far-OOD graph samples? (Sec. 4.2 RQ3
(Robustness): How does the inclusion of OOD samples in the ID training affect the robustness of
GLAD and GLOD methods? (Sec. 4.3 RQ4 (Efficiency): Are the GLAD and GLOD methods
efficient in terms of time and memory usage? (Sec. 4.4

---

Published as a conference paper at ICLR 2025
Table 3: Comparison in terms of AUROC. The best three results are highlighted by 1st 2nd and
3rd "Avg. AUROC" and "Avg. Rank" indicate the average AUROC and rank across all datasets.
GK-3 torerapi
SC-00 (neeskop
Chi-hacd GLAD (cd44-4ad)
CNN-auol GLOD (cs-to-od)
1057
03.32
00:09
03.27
03
00.33
23:33
07.40
00.05
71.43
66.21
$3.30
04.56
01.37
70.55
06.71
62.02
$4.63
61.90
11:31
09:37
06:12
(7:51
70133
70.48
PRAB
53.54
17.35
13:33
(4.31
06.31
66.31
$7.63
65:45
51:35
13.55
$3.00
21.39
1030
13.64
18.00
03:88
41-47
$3.31
05.37
76.13
00.94
09:33
06.05
98:35
13.43
65.46
ENGYYNESS
13:45
19:52
$5.70
11.00
60.9
62.73
63.00
62.44
61.75
$3.90
65.11
62.04
68.56
01.55
$3.00
$3.33
$3.35
12.81
12.00
30
13:25
13.41
18331
62.00
71155
50076
77/20
78.56
17:34
34.00
71.52
21:11
31.97
04.32
12.01
$5.35
13.59
64
19:00
96.99
16035
17.00
1006
19.21
13.52
$5.52
43.39
41.05
19:38
31:91
91.81
NC31
30.22
48.11
63.33
65.29
76.52
08.32
66.13
01.55
38.55
42.54
45.90
48.31
33.79
50,00
71.24
40.39
19.21
27.00
$1.99
01.79
91.52
01:55
60:23
06-46
$3.25
03.93
61:32
12:03
01:35
$3.45
53.59
96.55
11:13
21.45
19.00
$1.35
06.75
71.00
76.55
50.33
07.02
$2.13
37.66
53.55
54
$3.33
11
08.18
16.07
06.35
41.00
61.03
43:65
03:04
57.63
13.15
14
13
51.53
13:6
7916
86:00
10.5
65:30
35.65
$3:03
13.11
36.00
65
13.37
15
7655
290
13:99
32.00
13.55
15
32.59
34.36
31.8
53.34
66.55
03.57
0/30
06.00
96.95
6005
50.05
01:22
71.36
01:02
67.03
$1.54
02/6
11.51
55.15
5666
54.17
41.11
5631
90.30
70:31
66:29
53.79
53.59
03.53
JEV-SN
01:34
03.34
06.11
45:33
31338
12.07
26-33
35.00
31.55
-3.6
26.34
01:66
-7.56
56:23
76.12
08:11
41:33
23.11
01.35
48:45
46.72
$4.57
16.15
11.44
01.0
18:36
$0.83
56.6
51.5
13.03
(3:3
16.45
03.35
53:48
63.15
55.35
07.28
54:35
66.75
13.01
16.1
26:44
26.96
(2.58
-1.3
07.36
01-3
10.33
06.71
15.3
63.57
31.0
15.35
02.55
13.45
36.55
33.00
5480
1832
$3.96
4239
36,33
50.45
57.5
37.55
42:33
15.34
98113
9418
33.53
$2.31
51233
53.45
51.15
$3.9
$2.11
60.99
52.12
14.23
$3.76
$7.10
45344
43.34
43.35
43.01
43.32
41.00
41.31
4720
46.0
56.1
32.44
91.31
02.97
$4.34
66.11
65.97
35.96
63.42
$5.39
Ang AUTDD
32.33
33.97
37.36
12.51
35.11
33.33
61.39
61.30
96.01
06.27
75.18
13.36
00.30
11.0
71/03
01.35
0434
11311
These research questions are designed to comprehensively evaluate the performance, OOD sensitivity spectrum, robustness, and efficiency of GLAD and GLOD methods. By examining these aspects, aim to provide a thorough understanding of their strengths and limitations across different scenarios. For more detailed experimental descriptions and settings, please refer to Appendix F
4.1 PERFORMANCE COMPARISON (RQ1)
Experiment design. We conduct a comprehensive comparison of the detection performance in terms of AUROC, AUPR, and FPR95 of 18 benchmark algorithms on 35 benchmark datasets. For each method and dataset, we conduct 5 runs of experiments and report the average performance.
Experimental results. Table 3 shows the performance comparison in terms of AUROC, and Fig. provides box plots to overview the model performance across 35 datasets for all metrics. In Appendix F) detailed performance with standard deviation is demonstrated in Table R and the box plots of the ranking of each method are demonstrated in Fig. 7 We have the following observations.
Observation O: The SOTA GLAD/GLOD methods show excellent performance on both tasks. The results in TableBJhgghlig the excellent performance of the SOTA GLAD/GLOD methods on both detection tasks. Specifically, the GLAD methods, SIGNET, CVTGAD, and OCGTL, demonstrate outstanding performance in OOD detection tasks, achieving competitive results in several datasets. Meanwhile, the GLOD method GOOD-D, while not attaining the best performance in anomaly detection tasks, still performs commendably with an average ranking of 4. 14, placing it among the top performers. This observation highlights the intercommunity between GLAD and GLOD tasks, emphasizing the need to apply powerful methods designed for one task to the other.
Observation @: No universally superior method. Table demonstrates that no single method consistently outperforms others on more than 12 (out of 35) datasets. Even the top-ranked method, SIGNET, can exhibit sub-optimal performance on several datasets belonging to different types, such as HSE, DD, ES- -MU, and ZINC-Size. In Fig. 3 although SIGNET has higher median and a compact interquartile range, the top 25% of its performance still lower than that of some other models. Similar unstable performance can also be found in other competitive methods, such as OCGTL, CVTGAD, and GOOD-D. This finding illustrates the diversity of our benchmark datasets, underscoring the challenge of identifying "universal method" that performs well across all datasets. This is particularly evident in the AUPRC metric. For instance, on HSE, SIGNET achieves an AUROC of 64.56 vs. 71.42 (SOTA). On DD, it scores 74.53 vs. 80.76 (SOTA). In the inter-dataset shift ES ES, it achieves 91.43 vs. 99.64 (SOTA), and in the intra-dataset shift ZINC-Size, it scores 57.29 vs. 68.43 (SOTA).
7

---

Published as a conference paper at ICLR 2025
(a) AUROC?
100
50
K-3M KCF AL.VM
ILFF
10
SCL oc.
ocuN suxako ooun siGkEl GLADC cVTORD 9090-36-ahhu A4U00 GOODAI
(b) AUPRC1
50
SC.-GMM
aikec
cOdDAT
(c) FPR95
100
50
36-00 n6- WI-90 INLe 15.00
164F
SC Sow GC:4F
6OGN SLank cocn SIGET GIADC OVTICAD 500D-DGmenn ANGOD GOODAT
Figure 3: Comparison of the detection performance on 35 datasets in terms of three metrics.
Observation 6: Inconsistent performance in terms of different metries, From Fig. 3 andd Table 8 we can observe that some methods performing well on certain metrics may show unstable performance on other metrics. Specifically, CVTGOD and GOOD-D consistently achieve high AUC values on most datasets, but show more variability in other metries, particularly their sub-optimal AUPRC and FPR95 performance. This indicates that methods performing well in overall discrimination (high AUROC) may struggle with precision-ree recall trade-offs (low AUPRC) and maintaining low false positive rates at high true positive rates (low FPR95). This finding highlights the importance of comprehensive evaluation using multiple metrics.
Observation O: End-to-end methods show consistent superiority over two-step methods. While two-step methods show notable performance in certain scenarios, end-to-end methods consistently outperform them. Specifically, in Table the average rankings of most end-to-end methods are below 10, while all two-step methods have rankings above 10. This consistent superiority highlights the advantages of integrated and unified learning approaches segmented and two-step processes.
4.2 OOD SENSITIVITY SPECTRUM ON NEAR-OOD AND FAR-OOD (RQ2)
Experiment design. To evaluate the OOD sensitivity spectrum of GLAD/GLOD methods in handling and far OOD samples, consider two different settings define near-OOD and far-OOD: (A) intra-inter dataset setting and (B) size-based distance setting. setting A, we define the intra-dataset samples with different class labels as the near-OOD, while samples from another dataset are considered far-OOD. In setting B. the size of graphs serves as the measure to divide near and far for fair comparison. Please refer Appendix D.4
OOD, We redefine the size of training/testing for more detailed description of experimental settings.
Experimental results. The performance of GLAD and GLOD methods in distinguishing between ID and near-OOD/far-OOD samples is demonstrated in Fig. and Table where Subligures (a)-(c) are setting A and (d) is in setting B. We have the following key observations.
Observation 6: Near-OOD samples are harder to detect compared to far-OOD samples. From Fig. 4 evident that end-to-end models generally perform better detecting far-OOD samples than near-OOD samples. This trend is consistent across various datasets. For instance, in the AIDS dataset, models like GOOD-D, GraphDF, and GOODAT achieve highly better AUROC values for far-OOD conditions than for near-OOD, Similar patterns are observed in the other datasets, where most models display better performance in far-OOD scenarios. This consistent performance discrepancy underscores the challenge of detecting near-OOD samples that closely resemble the ID data, highlighting the need for enhanced model sensitivity to subtle deviations.
Observation @: Poor OOD sensitivity spectrum of several GLAD/GLOD methods in specialized scenarios. Despite the overall effectiveness GLADIGLOD methods, their performance has notable limitations, particularly in specialized scenarios. Firstly, in setting A, GLOD approaches (i.e., GOOD-
8

---

Published as a conference paper at ICLR 2025
AIDS
(b) BZR
(c) ENZYMES
(d) ZINC
100
80
AUROC
80
60
50
80
50
40
20
30
40
30
Figure 4: Near and Far OOD performance comparison in terms of AUROC.
Peturtetin 0%
Perturtarion on 10%
Pertursation 20%
Perturtorinn von 10%
(a) PROTEINS
(b) BZR
COX2
80
75
75
AUROC
70
65
55
60
acoi
alacaro
acorz
grower
GLASE
cyreap
conto
Gramtr
A630p
acc2aT
60
acaw
alecako
occo
SIGNET
GLANC
avreco
canp.
Granne ARcco
cOCDAT
acole Gloane
ocort
sunne
GLADG cyrava
apcza Gramioy Addpp
GooDar
Figure 5: Performance of models under different perturbation levels in terms of AUROC.
D and GraphDE) exhibit significant performance gaps between near-OOD and far-OOD conditions. This suggests that when OOD samples are similar to ID samples, the detection capability of GLOD methods is significantly compromised. This limitation is even more pronounced in setting B, where GLAD methods significantly underperform two-step methods. In these scenarios, GLAD methods struggle with size-based deviations, resulting in substantial performance gaps.
4.3 ROBUSTNESS UNDER TRAINING SET PERTURBATION (RQ3)
Experiment design. In this study, we investigate the robustness of GLAD/GLOD methods against the perturbation of the normal/ID training set by contaminating anomaly/OOD samples. Specifically, we set the perturbation ratios as 0%, 10 % 20%, and 30% to explore the impacts of different perturbation strengths. For more details, please refer to Appendix [D.4
Experimental results. The performance of each GLAD/GLOD method under different strengths of perturbation of the training dataset is demonstrated in Fig. which brings the below observations.
Observation G: Performance degradation with increasing contamination ratio. From Fig. 5 obvious that the performance of GLAD/GLOD methods generally deteriorates the proportion of OOD samples in the ID training set increases. This trend is evident across multiple datasets and methods. For example, in the PROTEINS, BZR, and COX2 datasets, we observe that models such as GLocalKD, OCGTL, SIGNET, GOOD-D, and others show a consistent decrease in AUROC values as the proportion of OOD contamination increases from 0% 30%. This decline performance suggests that the presence of anomaly/OOD samples in the training data introduces noise and confounds the models, making it increasingly difficult for them distinguish between generalized ID and OOD samples during testing.
The sensitivity of different methods/datasets can be diverse. Although increas- ing perturbation strength affects most methods, some methods exhibit notable robustness to the perturbation. Specifically, GLADC shows minimal performance degradation across all datasets, while CVTGOD demonstrates impressive stability in the BZR. Conversely, several methods are highly sensitive to perturbations some cases. For example, OCGTL's performance on the PROTEINS declines sharply as the level of contamination increases. The unstable performance highlights the need for improving the robustness of such methods to ensure reliable performance in real-world scenarios where data contamination is common.
Observation 3:
4.4 EFFICIENCY ANALYSIS (RQ4)
Experiment design. We evaluate the computational efficiency of GLAD/GLOD methods using default hyperparameter settings. Our assessment focuses on two main aspects: time efficiency and memory usage. See Appendix DD3 for detailed description.

---

Published as a conference paper at ICLR 2025
Experimental results. We present the computational efficiency of all methods in terms of time and memory usage in Fig (egf6(a)aam and Fig 6(b) respectively. We highlight the below observation.
Observation @: Certain end-to-end meth- ods outperform two-step methods in terms of both performance and computational costs. Observation ! demonstrates that end-to-end methods usually outperform the two-step meth- ods. In addition to their superior performance, most end-to-end methods exhibit comparable time efficiency and significantly better memory
(a) Different
Nol of Dificent Mode's
12.
Time tsl
Figure 6: Time and memory usage comparison.
efficiency. As shown Fig- ((a) end-to-end methods achieve optimal results much faster
than two-step methods, except for GOOD-D and GraphDE. In contrast, the iF detector in two-step methods significantly increases the time required to reach optimal performance, leading to sub- stantially higher time costs compared to most existing methods. illustrated in Fig. [6()] end-to-end methods show much lower CPU and GPU consumption than two-step methods. Although graph kernel methods do not utilize GPU resources, their high CPU consumption is critical consideration. To up, the majority of end-to-end methods may be preferable for GLAD/GLON tasks due to their advantages in effectiveness and efficiency.
terms of memory usage, as
5 CONCLUSION AND FUTURE DIRECTIONS
In this paper, we introduce a unified benchmark, UB-GOLD, for graph-level anomaly detection (GLAD) and graph-level -iit---disttibution detection (GLOD), which comprehensively compare the performance of 18 GLAD/GLOD methods on 35 datasets across various scenarios and domains. Based on UB-GOLD, we conduct extensive experiments to analyze the effectiveness, OOD sensitivity spectrum, robustness, and efficiency of these methods. Additionally, we provide a visualization and analysis of the Judge Score Distribution in the Appendix [3 Observation 1 highlights the intercommunity between GLAD and GLOD tasks. It reveals that many approaches show comparable performance across both tasks, suggesting significant overlap in the underlying principles and techniques used for detecting anomalies and OOD samples. Several insightful observations summarized from the results, providing an in-depth understanding of existing methods and inspiration for future work. Our experiments reveal that SOTA methods for GLAD and GLOD exhibit superior performance across both tasks, underscoring the strong interconnection between these two forefront research domains.
Despite the promising results of existing approaches, several critical challenges and research directions remain worthy of future investigation:
Universal approaches for diverse datasets. Observation @ suggests the existing GLADD/LOOD methods do not consistently perform well across diverse datasets. In this case, it is promising opportunity for producing novel approaches that can generally work well on diverse datasets. Such universal approaches should be aware of various structural and attribute characteristics of graph data from diverse domains and be sensitive to different types of OOD and anomaly samples.
Awareness of near-OOD samples. Observations G and @ indicate that most existing methods struggle to detect near-OOD samples. Thus, future research is expected to discover more advanced solutions to effectively differentiate between near-OOD and ID samples. Considering the inacces- sibility of OOD samples during model training, accurately capturing the patterns of ID data and establishing reliable decision boundaries can be the key to achieving the goal.
Robust approaches against unclean training data. Observations t and @ expose that most meth- ods are vulnerable to perturbation (i.e., data contaminated by OOD samples) of training datasets. Considering the difficulty of acquiring clean training data in several real-world applications, future approaches are expected be more robust against noisy training data.
Limitation. Since UB-GOLD mainly focuses on unsupervised GLADIGLOD tasks, the methods that require anomaly labels (e.g. i-GAD (Zhang et al.) 2022)), and domain-specine pre-trained models (e.g., PGR-MOOD (Shen et al. (0244) are not included for a fair comparison. As long-term evolving project, UB-GOLD will include these methods in the codebase for quick implementation.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
This research is dedicated solely to scientific inquiry, without involving human subjects, animals, or materials that may pose environmental concerns. As such, we do not anticipate any ethical risks or conflicts of interest. We are committed to upholding the highest standards of scientific integrity and ethics to ensure the accuracy and credibility of our findings.
ACKNOWLEDGMENTS
This work was supported by a grant from the National Natural Science Foundation of China under grants (No.62372211, 62272191), the Foundation of the National Key Research and Development of China 0w.2222IZD11250000 the International Science and Technology Cooperation Program of Jilin Province (No. 2,.2022340006GGH No. 20240402067GH), and the Science and Technology Development Program Jilin Province (No. 20220201 Y. Liu and S. Pan were partially supported by Australian Research Council (ARC) under grant DP240101547.
REFERENCES
Ahmed Abdelaziz, Hilde
Karl- Werner Schramm, and Igor V Tetko, Consensus modeling for hts assays using in silico descriptors calculates the best balanced accuracy in tox21 challenge. Frontiers in Environmental Science, 4:2, 2016.
Han Altae-Tran, Bharath Ramsundar, Aneesh S Pappu, and Vijay Pande. Low data drug discovery with one-shot learning. ACS central science, 344)2283,, 2017.
Mennatallah Amer, Markus Goldstein, and Slim Abdennadher. Enhancing one-class support vector machines for unsupervised anomaly detection. In Proceedings of the ACM SIGKDD workshop on outlier detection and description, pp. 8-15, 2013.
John Delancy. Esol: estimating aqueous solubility directly from molecular structure. Journal of chemical information and computer sciences, 44(3):1000- 1005, 2004.
Kaize Ding, Yixin Liu, Chuxu Zhang, and Jianling Wang. Data-efficient graph learning: Problems, progress, and prospects. Al Magazine, 45(4):549 560, 2024a.
Kaize Ding, Xiaoxiao Ma, Yixin Liu, and Shirui Pan. Divide and denoise: Empowering simple models for robust semi-supervised node classification against label noise. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 574-584 2024b.
Paul D Dobson and Andrew Doig. Distinguishing enzyme structures from non-enzymes without alignments. Journal of molecular biology, 330(4))771-783 2003.
Xiangyu Dong, Xingyi Zhang. and Sibo Wang. Rayleigh quotient graph neural networks for graph-level anomaly detection. In International Conference on Learning Representations, 2024.
Matthias Fey and Jan E. Leussen. Fast graph representation learning with PyTorch Geometric. In /CLR Workshop on Representation Learning on Gruphs and Manifolds, 2019.
Kaitlyn M Gayvert, Neel S Madhukar, and Olivier Elemento. A Latvenriven approach to predicting successes and failures of clinical trials. Cell chemical biology, 23(10):1294 1301, 2016.
Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. Good: A graph out-t--diittibution benchmark. Advances Neural Information Processing Systems, 1--2059-2073 2022.
Yuxin Guo, Cheng Yang, Yuluo Chen, Jixi Liu, Chuan Shi, and Junping Du. A data-centric framework endow graph neural networks with out-t---ditiibutti detection ability. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 638-648, 2023.
Christoph Helma, Ross D. King, Stefan Kramer, and Ashwin Srinivasan. The predictive toxicology challenge 2000-2001. Bioinformatics, 17(1):107-108 2001.
Weihua Hu. Matthias Fey. Marinka Zitnik, Yuxiao Dong. Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning graphs. Advances in neural information processing systems, 33:22118 18-21133. 2020.
11

---

Published as a conference paper at ICLR 2025
Yuanfeng Ji, Lu Zhang, Jiaxiang Wu, Bingzhe Wu, Lanqing Li, Long-Kat Huang, Tingyang Xu, Yu Rong, Jie Ren, Ding Xue, et al. Drugood: outoof-distributinn dataset curator and benchmark for ai-aided drug discovery-a a focus on affinity prediction problems with noise annotations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 8023 023-8031,220 2023.
Jian Jiang, Rui Wang, and Guo- Wei Wei. Ggl-tox: geometric graph learning for toxicity prediction. Journal of chemical information and modeling, 61(4):1611-1700 2021.
Wei Ju, Siyu Yi, Yifan Wang, Zhiping Xiao, Zhengyang Mao, Hourun Li, Yiyang Gu, Yifang Qin, Nan Yin, Senzhang Wang, et al. A survey of graph neural networks in real world: Imbalance, noise, privacy and ood challenges. arXiv preprint arXiv:2403.04400 2024.
Xin Juan, Kaixiong Zhou, Ninghao Liu, Tianlong Chen, and Xin Wang. Molecular data programming: Towards molecule pseudo-labeling with systematic weak supervision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle WA, USA, June 16-22, 2024, pp. 308 318, IEEE, 2024, doi: 10. N0.1109CCVPP57733200000000
URL /tttps://dol..o
10. C1110//////223333/3 2024 00037
Jindong Li, Qianli Xing, Qi Wang, and Yi Chang. Cvtgad: Simplified transformer with cross-view attention for unsupervised graph-level anomaly detection. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 185-200. Springer, 2023.
Shiyuan Li, Yixin Liu, Qingfeng Chen, Geoffrey I Webb, and Shirui Pan. Noise-resilient unsupervised graph representation learning via multi-hop feature quality estimation. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 1255-1265, 2024.
Wenchao Hassen Saidi, Huascar Sanchez, Martin Schäf, and Pascal Schweitzer. Detecting similar programs via the weisfeiler-leman graph kernel. In Software Reuse: Bridging with Social-Awareesss: 15th International Conference, ICSR 2016, Limassol, Cyprus, June 5-7, 2016, Proceedings 15, pp. 315-330. Springer, 2016.
Zenan Li, Qitian Wu, Fan Nie, and Junchi Yan. Graphde: generative framework for debiased learning and of-distribution detection on graphs. Advances in Neural Information Processing Systems, 35:30277-30290. 2022.
Fei Tony Liu, Kai Ming Ting, and Zhi- Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference on data mining, pp. 413-422. IEFF, 2008
Kay Liu, Yingtong Dou, Yue Zhao, Xueying Ding. Xiyang Hu, Ruitong Zhang, Kaize Ding, Canyu Chen, Hao Peng, Kai Shu, et al. Bond: Benchmarking unsupervised outlier node detection on static attributed graphs Advances in Neurul Information Processing Systems, 35:27021-220335. 2022.
Yixin Liu. Shiyuan Li, Yu Zheng, Qingfeng Chen, Chengqi Zhang, and Shirui Pan. ARC: A generalist graph anomaly detector with in-context learning. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Uirich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems.
Yixin Liu, Kaize Ding, Huan Liu, and Shirui Pan. Good-d: On unsupervised graph oi-isttribtttin detection. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pp. 339-347, 2023a.
Yixin Liu, Kaize Ding, Qinghua Lu, Fuyi Li, Leo Yu Zhang, and Shirui Pan. Towards self- interrrrtrre graph-level anomaly detection, Advances in Neural Information Processing Systems, 36, 2023b.
Yixin Liu, Thalaiyasingam Ajanthan, Hisham Husain, and Vu Nguyen. Self-supervision improves diffusion models for tabular data imputation. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 1513 1522, 2024.
Xuexiong Luo, Jia Wu, Jian Yang, Shan Xue, Hao Peng, Chuan Zhou, Hongyang Chen, Zhao Li, and Quan Z Sheng. Deep graph level anomaly detection with contrastive learning. Scientific Reports, 12(1):19867, 2022.
Rongrong Ma, Guansong Pang, Ling Chen, and Anton van den Hengel. Deep graph-level anomaly detection by glocal knowledge distillation. In Proceedings of the fifteenth ACM international conference on web search and data mining, pp. 704-714, 714, 2022.
Xiaoxiao Ma. Jia Wu. Jian Yang, and Quan Z Sheng. Towards graph-level anomaly detection via deep evolutionary mapping. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1631-1642, 2023.
12

---

Published as a conference paper at ICLR 2025
Ines Filipa Martins, Ana L Teixeira, Luis Pinheiro, and Andre O Falcao. bayesian approach to in silico blood-brain barrier penetration modeling, Journal of chemical information and modeling, 52(6):1686-1699 2012.
Rui Miao, Kaixiong Zhou, Yili Wang, Ninghao Liu, Ying Wang. and Xin Wang. Rethinking independent cross-entropy loss for graph-structured data. In Forty-first International Conference on Machine Learning, JCMI. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https:://ppnnteeeee
net/fel settforummiddzzge c9mQQN
David L Mobley and J Peter Guthric. Freesolv; a database of experimental and calculated hydration free energies, with input files, Journal of computer-aided molecular design, 28:711-728 2014.
Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. In ICML 2020 Workshop Graph Representation Learning and Beyond (GRI+ 2020), 2020. URL. www www.graphlee.. ea rning.io
Marion Neumann, Roman Gamett, Christian Bauckhage, and Kristian Kersting. Propagation kernels: efficient graph kernels from propagated information. Machine learning, 102:209 245, 2016.
Junjun Pan, Yixin Liu, Yizhen Zheng. and Shirui Pan. Prem: simple yet effective approach for node-level graph anomaly detection. In 2023 IEEE International Conference on Data Mining (ICDM), pp. 1253-1258, 2023. doi: 10. 10.11190DDM8522..000 0TAM:M/882222222.00.250 2023,00157.
Adam Paszke. Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Chen Qiu. Marius Kloft, Stephan Mandt, and Maja Rudolph. Raising the bar in graph-leve anomaly detection. In Lud De Raedt (ed.). Proceedings of the Thirty-Fire International Joint Conference on Artificial Intelligence, IJCAI-22. pp. 1199-2203. International Joint Conferences on Artificial Intelligence Organization, 7 2022.
doi: 10.2 10.24963/jjai. 2022/305. htt.pcss:://0i org/ 10. 24 724933/15 j ca1 2022/3022 305
Main Track.
Ann M Richard, Richard Judson, Keith A Houck, Christopher M Grulke, Patra Volarath, Inthirany Thillainadara- jah, Chihac Yang, James Rathman, Matthew T Martin, John F Wambaugh, al. Toxcast chemical landscape: paving the road to 21st century toxicology. Chemical research in toxicology, 29(8): 1225 1251, 2016.
Sebastian G Rohrer and Knut Baumann. Maximum unbiased validation (muv) data sets for virtual screening based on pubchem bioactivity data. Journal of chemical information and modeling, 49(2):169-18) 184, 2009.
Ida Schomburg, Antje Chang, Christian Ebeling, Marion Gremse, Christian Heldt, Gregor Huhn, and Dietmar Schomburg. Brenda, the enzyme database: updates and major new developments. Nucleic acids research, 32 (suDpL]):D433 D433, 2004.
Vikash Sehwag, Mung Chiang, and Prateek Mittal. SSD: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2021.
Xu Shen, Yili Wang, Kaixiong Zhou, Shirui Pan, and Xin Wang. Optimizing ood detection in molecular graphs: A novel approach with diffusion models. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, 2024.
Govindan Suhramanian, Bharath Ramsundar, Vijay Pande, and Rajiah Aldrin Denny. Computational madeling of 8-sccretase 1 (hace-1) 1) inhibitors using ligand hased approaches. Journal of chemical information and modelingg 56(10):19361144) 2016.
Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In International Conference on Learning Representations, 2020.
Jianheng Tang, Fengrui Hua, Ziqi Gao, Peilin Zhao, and Jia Li. Gadhench: Revisiting and henchmarking supervised graph anomaly detection. Advances in Neural Information Processing Systems, 36:29628-2966 2023.
S Vichy N Vishwanathan, Nicol N Schraudolph, Risi Kondor, and Karsten M Borgwardt. Graph kernels. The Journal of Machine Learning Research, 11:1201-1242, 2010.
Nikil Wale, Ian A Watson, and George Karypis. Comparison of descriptor spaces for chemical compound retrieval and classificttion. Knowledge and Information Systems, 14:347-375, 2008
13

---

Published as a conference paper at ICLR 2025
Luzhi Wang, Dongxiao He, He Zhang, Yixin Liu, Wenjie Wang, Shirui Pan, Di Jin, and Tat-Seng Chua. Goodat: Towards test-time graph out-of-distribution detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 1553,, 2024a.
Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma. Lingfan Yu. Yu Gai, et al. Deep graph library: graph-centrie, highly-performant package for graph neural networks. arXiv preprint arXiv: / 005,,, 2019.
Yili Wang, Kaixiong Zhou, Rui Miao, Ninghao Liu, and Xin Wang. Adagel: Adaptive subgraph contrastive learning to generalize large-scale graph training. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 2046-2055, 2022.
Yili Wang. Kaixiong Zhou, Ninghao Liu, Ying Wang, and Xin Wang. Efficient sharpness- aware minimization for molecular graph transformer models. In The Twelfth International Conference Learning Representations. ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.nel, 2024b. URL hnttps://openreyes net. / 0o0
Bang Wu, He Zhang, Xiangwen Yang, Shuo Wang. Minhui Xue, Shirui Pan, and Xingliang Yuan. Graphguard: Detecting and counteracting training data misuse in graph neural networks. In NDSS. The Internet Society, 2024.
Zhenqin Wu, Bharath Ramsundar, Evan Feinherg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9 (2::513-530, 2018.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey graph neural networks. IEEE transactions on neural networks and learning systems, 32(1 1:4224, 2020.
Feng Xia, Ke Sun, Shuo Yu, Abdul Aziz, Liangtian Wan, Shirui Pan, and Huan Liu. Graph learning: A survey.
IEEE Trunsactions on Artificial Intelligence, 2(2):109-127, 2021.
Pinar Yanardag and SVN Vishwanathan. Deep graph kemels.. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1365-1374, 2015.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey.
arXiv preprint arXiv:2110. 1134,, 2021.
Jingkang Yang. Pengyun Wang. Dejian 7ou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang. Guangyao Chen, Bo Li, Yiyou Sun, al. Openood: Benchmarking generalized out-of-distribution detection. Advances Neural Information Processing Systems, 35:32598-333 32661,, 2022.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances neural information processing systems, 33:5812-5823, 2020.
Ge Zhang, Zhenyu Yang, Jia Wu, Jian Yang, Shan Xue, Hao Peng, Jianlin Su, Chuan Zhou, Quan Z Sheng, Leman Akoglu, et al. diserrmiriiiieeevve graph neural network for imbalanced graph-level anomaly detection. Advances in Neural Information Processing Systems, 35:24144-24157, 2022.
He Zhang, Bang Wu, Xingliang Yuan, Shirui Pan, Hanghang Tong, and Jian Pei. Trustworthy graph neural networks: Aspects, methods, and trends. Proc. IEEE, 11((2):97-19, 2024a.
He Zhang, Xingliang Yuan, and Shirui Pan. Unraveling privacy risks individual fairness in graph neural networks. In ICDE, pp. 1712-1725. IEEE, 2024b.
Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Yixuan Li, Ziwei Liu, Yiran Chen, and Hai Li. Openood v1.5: Enhanced benchmark for out-of-distribution detection. arXiv preprint urXiv:2306.09300 2023.
Lingxiao Zhao and Leman Akoglu. On using classification datasets to evaluate graph outlier detection: Peculiar observations and new insights. Big Data, 11(3):151-18,, 2023.
Yanqiao Zhu, Yichen Xu, Qiang Liu, and Shu Wu. An Empirical Study of Graph Contrastive Learning, arXiv.org, September 2021.
14

---

Published as a conference paper at ICLR 2025
A WHY UB-GOLD FOCUSES ON UNSUPERVISED SCENARIOS
Node-level anomaly detection (AD) and out-of-distribution (OOD) detection can performed in both supervised (Tagg et al. 2023) and unsupervised (Liu et al. 2022) settings due to their transductive nature. In node-level tasks, the goal is to detect anomalous or OOD nodes within a single graph. Since the entire graph is observed, labeled data is often accessible, and supervised methods can be effectively applied. Supervised approaches leverage these labels to guide the model in identifying anomalies, while unsupervised methods detect deviations from normal node behavior without labels. Both approaches can coexist, as node-level tasks benefit from the complete visibility of the graph during training.
In contrast, graph-level AD/OOD detection poses different challenges, making supervised or semi- supervised approaches less meaningful. Unlike node-level tasks, graph-level detection deals with unseen, independent graphs at test time, where labeled anomaly or OOD samples are rarely available in real-world scenarios. Relying on labeled data in supervised or semi-supervised settings shifts the focus to classification rather than detecting unexpected anomalies. This undermines the core challenge of generalizing across unseen graphs. Furthermore, graph-level tasks share closer resemblance to computer vision (CV) problems, where the primary challenge is detecting shifts in distribution without labeled anomalies. Our inspiration stems from the success of benchmarks like OpenOOD (Zhang et al., 2023) in CV, where unsupervised methods have proven to be more effective at handling distribution shifts in the absence of labeled OOD data.
By focusing solely unsupervised methods in UB-GOLD
ensure that the benchmark reflects the true nature of graph-level AD/OOD detection in real-world applications. Unsupervised approaches labeled data, allowing models to learn from the normal distribution
eliminate the dependency and detect deviations more naturally. This aligns with practical conditions, where labeled OOD or anomaly samples are rarely available. Additionally, an unsupervised framework promotes more robust generalization across diverse datasets, pushing the boundaries of AD/OOD detection in the graph domain, similar to the advancements driven by OpenOOD in computer vision.
B DETAILED DESCRIPTION OF DATASETS IN UB-GOLD
The description of datasets in UB-GOLD is given as follows. To ensure the reliability of our benchmark and avoid any data leakage, we carefully curated all dataset splits to guarantee no overlap between ID and OOD samples, or between training and test sets. Each dataset was meticulously prepared to maintain a clear separation, ensuring that OOD samples are entirely distinct from ID data. This strict partitioning is consistently applied across all experimental setups, providing robust and unbiased evaluation of the methods.
Type Datasets with intrinsic anomaly
Datasets under this type include real anomalies inherently present in the data, defined based on the biological or chemical properties of the samples. These anomalies are characterized by rare or atypical patterns that deviate from the majority of the data. Such intrinsic anomalies are valuable for evaluating the model's ability to detect deviations from the norm, a common challenge in real-world data analysis.
HSE (Abdelaziz et al., 2016:: Anomalies in this dataset are samples exhibiting unusual stress response elements that deviate significantly from typical cellular stress responses. These may include rare mutations or outlier stress markers.
MMP (bddllaiiz et al.) 2016): Anomalous samples are identified as molecular structures with abnormal expressions matrix metalloproteinases, often linked to extreme biological behaviors such as uncontrolled cancer metastasis.
p53 Abddeaaiiz et al. 2016): The anomalies are instances with abnormal p53 protein behav- ior or structural deviations, which may indicate dysfunction in apoptosis or tumor suppression mechanisms.
Tox21 Challenge Data.
15

---

Published as a conference paper at ICLR 2025
--aRmaa (Abdelaziz et al., 2016): Anomalies molecular samples exhibiting inregular activation or suppression of PPAR-gaa gamma, which is unusual typical metabolic and inflammatory processes.
Type II: Datasets with class-based anomaly
This type includes six datasets (PROTEINS, ENZYMES, AIDS, DHFR, BZR, COX2) where the graphs are attributed, meaning that each node contains descriptive features. The remaining datasets plain graphs without node-level attributes. All datasets with class-based anomalies are derived from graph classification benchmarks, and for anomaly detection tasks, the minority class is treated anomalies. These datasets particularly useful for evaluating models' ability to identify rare classes or outliers within a given graph dataset.
COLLAB (Yanardag Vishwanathan 2015): Scientific collaboration networks, representing authors and their co-authored papers in various scientific fields.
IMDB-BINARY Yanardag & Vishwanathan 2015): Movie collaboration networks, where nodes represent actors and edges denote their co-appearance in films. This dataset helps in analyzing the collaboration patterns within the film industry.
REDDIT-BINARY (Yanardag & Vishwanathan, 2015): Discussion threads from Reddit, each graph representing thread with nodes as users and edges interactions. It captures the structure online discussions and their dynamics.
ENZYMES (Schomburg et al., 2004):: Protein tertiary structures, each graph represents an enzyme with nodes as secondary structure elements and edges as spatial adjacencies, crucial for biochemical and functional studies.
Comprehensive protein structures and interaction data, where nodes represent amino acids and edges denote interactions, providing insights into protein functions and interactions.
PROTEINS (Dobson & Doig, 2003):
DD (Vishwanathan et al., 2010): Protein-protein interaction networks, capturing the intricate relationships between proteins within biological systems, essential for understanding cellular functions.
BZR (Wu et al. 2018): Benzodiazepine receptor ligands, with nodes representing atoms and edges representing bonds, used for studying molecular interactions with benzodiazepine receptors.
AIDS (Morris et al. 2020): Chemical compounds screened for antiviral activity against HIV, where each graph represents a molecule, useful in drug discovery and antiviral research.
COX2 (Morris et al.) 2020) Cyclooxygenase-2 inhibitors, where nodes and edges represent molecular structures, important for studying anti-inflammatory drug properties.
NCT1 (Wale et al., 2008): Chemical compounds screened for anti-cancer activity, each graph representing molecule, used for evaluating potential anti-cancer drugs.
DHFR (Morris et al. 2020): Dihydrofolate reductase inhibitors, with graphs representing molecu- lar structures, focusing on compounds inhibiting the DHFR enzyme, vital for cancer and bacterial infection treatments.
Type II:: Datasets with inter-dataset shift.
Inter-dataset shift datasets designed by drawing ID and OOD samples from different but related datasets. This design approach allows us to simulate real-world scenarios where data distributions for both ID and OOD samples distinct, yet share some underlying similarities.
BBBP MMrtinns et al. 2012): BBBP is the Blood-brain barrier penetration (BBBP) dataset includes binary labels for over 2000 compounds their permeability properties.
BACE (Subramanian et al. 2016): BACE provides series of human 3-secretases well as their binary label, and all data are experimental values reported the scientific literature over the last decade.
CLINTOX (Gayvert et al. 2016 The ClinTox dataset compares drugs that have been approved by the FDA with drugs that have failed in clinical trials for toxicity reasons.
TUDataset.
16

---

Published as a conference paper at ICLR 2025
LIPO (Wu et al., 2018): The full name of LIPO is Lipophilicity, which is important feature of drug molecules that affects both membrane permeability and solubility.
FREESOLV (Mobley & Guthrie, 2014); The Free Solvation database (FreeSolv) provides experimental and calculated free energies of hydration of small molecules in water.
TOXCAST (Richard et al., 2016); ToxCast providing toxicology data for library of compounds based on high-throughput screening and includes qualitative results of over 600 experiments on 8615 compounds.
ESOL (Delaney, 2004): ESOL is small dataset containing water solubility data for 1128 compounds with the goal of estimating solubility from chemical structures.
MUV (Rohrer Baumann 2009): The Maximum Unbiased Validation (MUV) group is another benchmark dataset selected from PubChem BioAssay by applying modified nearest neighbor analysis
TOX21 (Wu et al. 2018): The dataset contains qualitative toxicity measurements of 8014 com- pounds against 12 different targets, including nuclear receptors and stress response pathways.
SIDER (Altae-Tras et al., 2017): The Side Effect Resource (SIDER) is database of marketed drugs and adverse drug reactions which measured for 1427 approved drugs.
PTC-MR (Helma et al. 200T): PTC dataset labels compounds based on their carcinogenicity where MR Indicates that their rodent is a male rat.
Type IV: Datasets with intra-dataset shift.
Intra-dataset shift datasets simulate OOD conditions within a single dataset by defining shifts based on specific attributes such as size, scaffold, or assay. Unlike inter-dataset shifts, which involve drawing ID and OOD samples from different datasets, intra-dataset shifts introduce variations within the same dataset, creating challenges for models that must adapt to these changes without external data sources. Intra-dataset shifts are particularly useful for testing how well a model can handle structural or feature-based changes within a consistent data domain.
GOOD (Gui et al. 2022): A systematic benchmark specifically tailored for the graph OOD problem. We utilize two molecular datasets for OOD detection tasks: (1) GOOD-HIV a small- scale real-world molecular dataset which aim to predict whether this molecule can inhibit HIV replication. (2) GOOD-ZINC a real-world molecular property regression dataset from ZINC database and aims at predicting molccular solubility. Each dataset comprises two ID-OOD splitting strategies (scaffold and size), resulting in total of 4 distinct datasets.
DrugOOD (Ji et al. 2023): This OOD benchmark is designed for Al-aided drug discovery. includes three ID-OOD splitting strategies: assay, scaffold, and size. These strategies are applied to two measurements (IC50 and EC50), resulting in six datasets. Each dataset comprises a binary classification task aimed at predicting drug target binding affinity.
C DETAILED DESCRIPTION OF ALGORITHMS IN UB-GOLD
The description of benchmarking algorithms in UB-GOLD is demonstrated follows.
Graph kernels.
This kernel generates graph embed- dings by iteratively refining node labels based on subtree patterns. It effectively captures structural similarities within graphs, making it powerful tool for embedding generation.
Weisfeiler-Leman Subtree Kernel (WL) (Li et al. 2016):
Propagation Kernel (PK) (Neumann et al., 2016): This method propagates labels through the graph structure, resulting embeddings that reflect the graph's 's topology. It captures relational information within the graph, providing a robust basis for subsequent outlier detection.
Self-supervised learning methods.
GraphCL (GCL) (You et al., 2020): GCL leverages augmentations to learn robust graph-level representations. By contrasting different views of the graph, the model learns capture
17

---

Published as a conference paper at ICLR 2025
essential graph structures and properties, making it highly effective for various graph-based based tasks. This method is known for its strong performance in unsupervised learning settings.
InfoGraph (IG) (Sun et al. 2020): IG maximizes the mutual information between local and global graph representations to achieve unsupervised and semi-supervised graph-level representation learning. By capturing meaningful information across different levels of the graph, IG effectively
learns comprehensive graph emheddings that are useful for wide range of downstream tasks.
Outlier Detectors.
Isolation Forest (iF) (Liu et al., 2008): This algorithm isolates observations by constructing an ensemble of trees, each built by randomly selecting features and split values. An anomaly score is calculated based on the path length from the root to the leaf node, with shorter paths indicating anomalies.
One-Class SVM (OCSVM) (Amer al., 2013): OCSVM aims to separate the data from the origin using hyperplane in high-dimensional space. Points that lie far from the hyperplane are considered outliers.
Graph neural network-based GLAD methods.
OCGIN (Zhao & Akoglu 2023): Utilizes a GIN encoder optimized with Support Vector Data Description (SVDD) objective to identify anomalies within the graph structure. This method employs an end-to-end GNN model with one-class classification for effective anomaly detection.
GLocalKD (Ma et al. ) Jointly learns two GNNs and performs graph-level and node-level random knowledge distillation between their learned representations. By leveraging both local and global knowledge, this approach enhances the detection of anomalies.
OCGTL (Qiu al. 2022): : Extends deep one-class classification self-supervised detection
approach using neural transformations and graph transformation learning regularization. This technique improves the model's unsupervised anomaly detection capabilities.
SIGNET (Liu et al., 2023b); Proposes a self- inferrrrttrreabble graph-level anomaly detection frame- work that infers anomaly scores while providing subgraph explanations. By maximizing mutual information multi-view subgraphs, it achieves both detection and interpretation of anomalies.
Incorporates graph-level adversarial contrastive learning to identify anomalies. Through the creation of adversarial examples and learning robust representations, this method effectively distinguishes between normal and abnormal graphs.
GLADC (Luo et al. 2022);
CVTGAD (Li et al., 2023): Introduces a simplified transformer with cross-view attention for unsupervised graph-level anomaly detection. It overcomes the limited receptive field of GNNs by using transformer- brasdd module to capture relationships between nodes and graphs from both intra-graph and inter-graph perspectives.
Graph neural network-based GLOD methods.
GOOD-D (Liu et al. 2023a): Performs perturbation-free graph data augmentation and utilizes hierarchical contrastive learning on the generated graphs for graph-level OOD detection. By leveraging multiple levels of contrastive learning, GOOD-D enhances the representation learning process, making robust in distinguishing between in-distribution (ID) and out-of-distribution (OOD) graphs.
GraphDE (Li al., 2022): Models the generative process of the graph to characterize distribution shifts. Using variational inference, GraphDE infers the environment from which a graph sample is drawn. This generative approach effectively identifies ID and OOD graphs by detecting shifts in the underlying data distribution.
AAGOD (Guo et al. 2023): A data-centric framework for graph OOD detection that operates on well-trained GNNs without retraining. AAGOD uses an Adaptive Amplifier to modify input graphs, highlighting key patterns for OOD detection. Through its Learnabl Amplifier Generator (LAG) and Regularized Learning Strategy (RLS), significantly improves detection performance and efficiency.
18

---

Published as a conference paper at ICLR 2025
GOODAT (Wang et al. 2024a): A test-time graph OOD detection method designed to operate well-trained GNNs without requiring training data or altering the GNN architecture. GOODAT em- ploys graph masker and the Graph Information Bottleneck (GIB) principle to extract informative subgraphs. Utilizing GiB-hoosted loss functions effectively distinguishes between ID and OOD) graphs, achieving strong performance across diverse datasets.
D SUPPLEMENTAL INFORMATION OF UB-GOLD
D.I DEFINITION OF GLAD AND GLOD
GLAD definition. For GLAD, task is to detect anomalous graphs within a given distribution P The objective is to assign anomaly scores S(G) such that anomalous graphs are distinguishable from normal graphs based on threshold T:
0 (Anomalous), SfG) S T
gG; T)
1 (Normal),
S(G) T.
The theoretical objective for the anomaly scoring function S(G) is to maximize the separation between the distributions of normal and anomalous graphs:
max P. S(G) - R. S(G),
where peormal and panomalous represent the idealized distributions of normal and anomalous graphs. While this objective cannot be directly optimized during training due to the lack of access to test distributions, provides guiding prineiple for designing S(G). Practical approaches typically rely on approximations or surrogate objectives based training data, such as unsupervised or self-supervised learning frameworks, to approximate the separation of score distributions. When the separation is achieved, threshold can effectively classify graphs as normal (1) anomalous (0), aligning with the GLOD framework.
GLOD definition. For GLOD, the task is to detect graphs in test dataset that originate from different distribution pout compared to the training distribution per. The goal is to assign OOD scores J(G) such that OOD graphs have significantly lower scores than ID graphs, enabling threshold-based classification.
(0 (OOD), JfG) S T, (G
[1 (ID),
if J(G) > T.
(5)
The theoretical objective for the OOD scoring function J(G) maximize the separation between the distributions of ID) and OOD graphs:
max EL Ea-pu (G),
(6)
where ppin and pout represent the idealized distributions of ID and OOD graphs, respectively. Although these distributions are not directly accessible during training (since OOD samples are not present in the training phase), they provide theoretical goal for designing (G). In practice, approximations are used based on the available training data to model the OOD detection function. When distributions of ID and OOD scores are well-separated, threshold 7 can effectively classify graphs as ID (1) or OOD (0).
D.2 METRICS
In UB-GOLD, we consider three metrics for evaluation. Their definitions are given as follows.
AUROC: Fundamental for both GLOD and GLAD, AUROC measures model's ability to dis- tinguish between normal and anomalous or OOD instances across various threshold levels. A higher AUROC value indicates better performance in correctly classifying positives (anomalies or OOD instances) and negatives (normal instances), making crucial for evaluating the overall effectiveness of detection algorithms.
19

---

Published as a conference paper at ICLR 2025
Table 4: Hyper-parameter search space of all implemented methods.
Algorithm
Hyper-parameter
Search Space
hidden size
16, 32, 64, 128
dropout
0, 0.1,, 0.2, 0.3 1,
General Settings
layers
learning rate
le-1, le-2, le-3, le-4
str_ str_dim
8, 16, 24, 32
GOOD-D
cluster number
2,
[0, 1.0]
dropedge
0, 0.2, 0.3 0, 0.1, 0.2, 0.3 graphde-x, graphde-a
GraphDE
dropnode
model type
str str_dim dim cluster number a
8, 24, 24, 32 2, 4 [0. 1.0] mean, max
CVTGAD
pooling
hidden size
32, 64, 128, 256 32, 64, 128
GLADC
output size
clip
0.10, 0.15, 0.20 True, False True, False
GLocalKD
nobn
nobias
hidden size
32, 64, 128
OCGTI
layers
2, 4.
pooling readout
add, max concat, add, last 4,
SIGNET
layers
aggregation bias
mean, add, max True, False 10. 50, 100
OCGIN
AAGOD
A
GOODAT
O B
0.1, 0.3, 0.5, 0.7,0.9 0.03, 0.05, 0.07
tree number
200, 250, 300
GCL+kernel
sample ratio
0.3, 0.4, 0.5, 0.6
number
200, 250, 300
sample ratio
0.3, 0.4, 0.5, 0.6
KerneIGLAD
neighbors
20. 30, 40
leaves
25, 35
WL iteration
3, 6,
AUPRC:
Gains importance in imbalanced datasets, common in AD where anomalies are rare, and GLOD where OOD instances are infrequent. This metric focuses on precision (the accuracy of positive predictions) and recall (the model's 's ability to detect all positive cases), providing clear measure of performance scenarios where positive cases are critical and more challenging to detect.
FPR95: Evaluates the number of false positives accepted when the model correctly identifies 95% of true positives. This metric is particularly useful in settings where missing an anomaly or an OOD instance can lead significant consequences, emphasizing the need for models that maintain high sensitivity without sacrificing specificity.
20

---

Published as a conference paper at ICLR 2025
Table 5: Dataset statistics including ID Train, ID Test, Near OOD, and Far OOD counts. Additionally,
Unseen Class (UC), Unseen Dataset (UD), Near Size (NS), and Far Size (FS).
Scenario Type
Data
ID Train
ID/Near/Far Test
Near OOD
Far OOD
ATDS
1280
80
ATDS(UC)
(UF
Class-based Anomaly
BZR
69
17
BZR(UC)
COX2(UD)
ENZYMES
400
20
ENZYMES(UC)
PROTEINS(UD)
Size-based
GOOD-ZINC Size
1000
500
ZINC(NS)
ZINCOFS)
Table 6: Dataset statistics including ID Train, ID Train with different perturbation levels, ID Test, and
OOD Test counts.
Data
ID Train
ID Train (10%)
ID Train (20%)
ID Train (30%)
ID Test
OOD Test
BZR
89
PROTEINS
360
374
387
400
90
93
COX2
81
89
96
103
21
51
D.3 ADDITIONAL EXPERIMENTAL DETAILS
Implementation Details. To ensure comprehensive evaluation and maintain fairness across broad spectrum of models, we develop an open-source toolkit named UB-GOLD. This toolkit is built on top of Pytorch 2.01 (Paszke et al. 2019), torch arrh__geometric 2.4.0 (Fey & Lenssen 2019) and DGL 2.1.0 (Wang et al., 2019). We implement graph kernel methods with the DGL library. All other models are unified using the torch_geometne library. GCL and IG are included via the PYGCL library (Zhu et al. 2021).
Hardware Specifications. All our experiments were carried out on a Linux server with an Intel(R) Xeon(R) Gold 5120 2.20GHz CPU, 160GB RAM, and NVIDIA A40 GPU, 48GB RAM.
Hyperparameter Settings. Table 4 provides comprehensive list of all hyperparameters used in random search complete with their search spaces. For the design of the default hyperparameters please refer to our code base in. /benchmark/Source.
Efficiency Analysis (See. 4.4). We evaluate the computational efficiency of GLOD and GLAD methods using default hyperparameter settings. Our assessment focuses on two main aspects:
Time Efficiency: record the average time each method takes to achieve the best results across all datasets, providing insights into their processing speed.
Resource Usage: We monitor each method's CPU and GPU consumption (on COLLAB) during experiments, determining their demand for computational resources.
This setup allows measure the methods' efficiency directly and reliably, reflecting their practi- cality for real-world application.
D.4 NEW EXPERIMENTS DATASET SPLIT
In the experiments for OOD sensitivity spectrum and robustness, do not follow the original split but utilize -xperiment-specffic splits for fair comparison. The details are as follows.
OOD sensitivity spectrum Near-OOD and Far-OOD (Sec. 4.2) To rigorously evaluate the performance GLOD and GLAD methods in handling near and far OOD conditions, we have implemented distinct partitioning strategy for this research question. Unlike the setup in RQ1, here we ensure that the number of samples in the Near OOD, Far OOD, and ID Test groups are precisely equal, detailed in Table 5 This balanced configuration is designed provide fair comparison across different degrees of OOD scenarios, and it includes two specific setups:
Intra-inter dataset setting: We utilize the class-based anomaly partitioning method to set the ID Train and ID Test, along with the Near OOD Test. The Far OOD Test employs datasets from the Shift category, representing more significant deviations.
21

---

Published as a conference paper at ICLR 2025
Size-based distance setting: We maintain the same ID Train and ID Test groupings of Intra SttaaDataset Shift. However, for the Near OOD and Far OOD tests, we categorize the samples based on differing graph sizes, with smaller sizes representing Near OOD and larger sizes for Far OOD.
These settings are designed to rigorously test the capability of GLOD and GLAD methods to recognize and differentiate between subtle and substantial distribution shifts, thereby assessing their effectiveness in realistic and challenging environments.
Robustness under Training Set Perturbation (Sec. 4.3). In this study, we investigate the robustness of GLOD and GLAD methods against the contamination of the ID training set with OOD samples. In Table 6 to be mixed into the ID training set. The remaining 70% of the OOD test dataset is kept intact for performance evaluation. This procedure is repeated to create four distinct experimental groups where 0%, 10%. 20%, and 30% of the originally selected OOD samples are added to the ID training dataset. These modifications allow us to systematically explore how progressively increasing the proportion of OOD samples within the ID training set affects the methods' ability to identify and differentiate true OOD instances during testing accurately.
we begin by partitioning the OOD test dataset, randomly selecting 30% of its samples
E REPRODUCIBILITY
Ensuring the reproducibility of experimental results is a core principle of UB-GOLD. Below, we outline the measures we have taken to achieve this:
Accessibility. All datasets, algorithm implementations, and experimental configurations are freely accessible through open-source project at httpps: / /i thub. Stw.bbb.cooomO---GG///D------ No special requests or permissions are needed to access the resources.
Datasets. Our datasets are publicly available and include TUDataset, OGB, TOX21, DrugOOD, and GOOD, Among them, TUDataset (Morris et al. 2020), OGB (Hu et al. 2020), and TOX21 (Abdelaziz al. 2016) are licensed under the MIT License. DrugOOD (Ji et al.) (023) licensed under the
GNU General Public License 3.0. GOOD (Gui et al. 2022) is licensed under GPL-3.0.
All these datasets are permitted by their authors for academic use and contain no personally identifiable information or offensive content.
Documentation and Usage. We provide comprehensive documentation to facilitate easy use of our library. The code includes thorough comments to enhance readability. Users can reproduce experimental results by following the examples, which outline how to run the code with specific data, methods, and GPU configuration arguments.
License. UB-GOLD is distributed under the MIT license, ensuring wide usability and adaptability.
Code Maintenance. We are dedicated to regularly updating our codebase, addressing user feedback, and incorporating community contributions. We also enforce strict version control to maintain reproducibility throughout the code maintenance process.
With these measures, UB-GOLD aims to foster transparency, accessibility, and collaboration within the research community.
F ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSES
F.1 ADDITIONAL EXPERIMENTAL RESULTS
In this section, we present additional experimental results, providing comprehensive insights into the performance of our models across different datasets and metrics.
Main experiments. Table g presents the complete results of the main experiment. Each model is executed 5 times with varying random seeds, and the mean scores along with standard deviations are reported. "Avg. AUROC". "Avg. FPR95", "Avg. AUPRC", and "Avg. Rank" indicate the average AUROC, FPR95, AUPRC, and rank across all datasets, respectively. We observe that the End-to-Eae todEnd approach stands out in AUROC, achieving the best overall results. However, in the metrics of AUPRC and FPR95, SSL-D in the two-step approach also performs well, showing competitive results.
22

---

Published as a conference paper at ICLR 2025
Table 7: Near and Far OOD performance comparison.
AIDS
BZR
ENZYMES
ZINC-Size
Model
Type
AUROCT
AUPR 7
F9R95
AUROC
AUPR 1
FFR995
AUROCT
AUFR 1
FPR95
AUROCT
ALPR
FFR995
IG-IF
Far 000
16.50
91:75
50.80
53.88
90.12
$1.87
56.31
92.00
03.95
50.19
96.00
Near 000
45.92
47.75
93:75
69.58
56.43
91.76
50.15
54.25
89.00
50.39
50.90
94.72
IG-SVM
Far 000
42.17
95.75
50.17
51.98
95.29
53.88
55.27
50.02
Near 000
45.01
46,42
95.50
50.83
55.36
96.47
53.85
55.51
85.00
50.14
50.46
94.92
GCL-IP
Far 00D
19:73
91.00
38.55
46.39
100.00
42.85
50.23
93.00
68.95
49.71
Near 000
53.53
53.85
91.50
47.25
36.29
98.82
50.65
52.05
63.00
50.51
50.00
94.00
GCL-SVM
Far 00D
30.26
91.00
42,55
97.65
44.82
50.67
91.00
69:43
49.90
Near 000
53.52
53.41
93.50
47.47
51.52
95.29
48.20
53.17
90.00
50.55
50.36
93.76
OCGIN
Far 000
99.58
99.88
0.75
67,20
83.44
37.65
63.45
72,12
93.00
41.97
45.39
97.00
Near 000
96.01
95.95
13:25
66,
59.78
50.59
58.35
63.39
89.00
36.96
45.58
99,55
GLocalDD
Far 000
100.00
100,00
0.00
83.39
7563
15.29
53.70
67.07
85.00
30.07
34.50
95.40
Near 00D
99.89
99.90
0.00
65.81
58.54
51.76
40.45
$4.30
85.00
13.14
31,09
99.90
OCGTL
Far 000
100.00
100.00
0.00
64.64
77:25
26.24
80.10
77.04
41.00
99.58
99.32
Near 000
99.61
99.65
0.00
65.19
58.08
46.34
74.00
89.00
51.00
64.6 40
70.09
23.44
SIGNET
Far 00D
98.00
93.07
2,00
95,50
90.38
11.96
67.48
67.05
85.00
60.48
79.23
66.96
Near 00D
97.16
91,02
2.50
88.93
83.93
85.88
71.60
71.27
68.00
72.90
71.41
82.64
GLADC
Far 000
100.00
100.00
0.00
79.58
69.52
41.18
43.00
57.80
90.00
35.55
40.40
98.12
Near 000
98.74
98.60
3.75
64.36
56.84
52.94
41.25
43.32
85.00
11.63
32:34
100.00
CVTGOD
Far 000
99.94
95
0.25
93.91
91.30
17.65
70.20
76.29
87.00
64.57
51.56
95.44
Near 000
94.93
95.01
24.00
80.45
72.37
63.53
53.05
58.
88.00
20.68
38.52
Nea
87.50
43.25
87.54
36.49
58.
88.00
55.85
GruphDE:
Near
100.00
97.65
100.00
56.61
57:35
57.92
54.23
Near
60.08
55.33
65:97
71.25
23.50
45.96
88.00
46.77
$2.08
PS SVM
PEIF
WUSVN
WLIF
BiF
GCLSVM OLL
OCOI OLIGKDD OLGIL
sicker GLADC OVTEAD GOOD Graploe AKLOU GOODAT
(b) AUPRC Ranki
125
10.0
PK.SUM
PiiI
WLL-OM
Wi-r
15H7
GCLGVM GCL-F
OCON GLOaKD OCOT.
SIGKET GADC cotAA G0OD GrantDE MdOD GOODAT
(c) FPR95 Rank
15.
5-
WLSVV
Wilit
G00D
Figure 7: Comparison of the ranking 35 datasets in terms of three metrics.
Ranking experiments. Fig. 7 shows the ranking of models across 35 datasets in terms of their performance. It provides a visual representation of the comparative ranking, allowing for an easy assessment of how different models rank against each other across large number of datasets.
Near and far OOD performance. We provide the full results of Near-OOD and Far-OOD evaluations for three metrics across four datasets, shown in Table This table allows for detailed comparison of Near and Far OOD performance, showcasing how our models perform under different out-of- distribution scenarios. Further confirming findings in Observations and @
Robustness under Training Set Perturbation. While Observations and are based on AUROC, also provide AUPRC and FPR95 results in this section. However, it is important to note that since AUPRC and FPR95 are computed based on the model's performance on AUROC, the conclusions drawn from AUROC do not necessarily extend to these other metrics. As shown in Fig and Fig 10
23

---

Published as a conference paper at ICLR 2025
Feturfarins
Feeturtorion 10%
Ferturboriin 20%
Perturbarlen 10%
(a) PROTEINS
(b) BZR
CDX2
80
70
70
65
AUROC
60
60-
55
50
50
GEl GELga ocol
auagen o0a07 clare
Good Adtos
GoODAT
10
le detue
dct ges
ate geon
cytero
apona Gravioe 44000 GooDAT
12
au Baaee
GLAne
Gooto AAGOOO
GOOSDAT
Figure 8: Performance of models under different perturbation levels in terms of AUROC.
Poturbation
Pottu
10%
Forturbation
Peturbation 30%
(a] PROTEINS
(b) BZR
[c) COX2
90
80
AUPRC
75-
50
mivi to azuup Gct occae
aegtt Senee
cLice GooD
cyroup
A4o00 GOoDAT
70
iost to ove octue
Gct goon
arogge
stonyy alene cyrane
GooD
Gr Racdo
aoDDAT
Host to ccuer
dc Gcon sove
a
GLADC Cyrona coon
ACCO GOODAY
Figure 9: Performance of models under different perturbation levels in terms of AUPRC.
(e) PROTEINS
(b) BZR
(c) COX2
FPR95
90
90
90
80
70
lo-rr So. uP
oct 2e
Gdd corso condo Grapie A40000 GOOENT
er ed oo
sraa Savco Soode of A400 GooDar
del 222
Bore core Sente Soopee
GooDAr
oo
Figure 10: Performance of models under different perturbation levels in terms of FPR95.
the results for AUPRC and FPR95 do not exhibit the same consistent trends, indicating that the behavior of models may vary across different evaluation metrics.
F.2 ADDITIONAL EXPERIMENTAL ANALYSES
AUROC
Sat AUROC Purk
2001
AADO
GOOM
(b) AUTC!
35
+P905
In Rencs
50
MADC
3000
Co
Recritacionn
Figure 11: Scores and ranks for four types of methods.
In this section, as illustrated in Fig. 2 we categorize the GLADIGLOD methods into four main technical groups:
One-Class Classification: OCGIN, OCGTL.
Contrastive Learning: GLocalKD, GOOD-D, CVTGOD, SIGNET.
Graph Reconstruction: GLADC, GraphDE.
24

---

Published as a conference paper at ICLR 2025
Well-Trained GNNs: AAGOD, GOODAT.
we observe that contrastive learning methods generally perform better and exhibit more stable results compared to other methods. This performance difference can be attributed to several key factors:
From Fig 11
Rich Representations in Contrastive Learning: Contrastive learning methods (GLocalKD, GOOD- D. CVTGOD, SIGNET) demonstrate superior and stable performance across metrics, particularly AUROC and AUPRC. This strong performance can be attributed to their ability to maximize mutual information between different augmentations of the same graph (positive pairs) and minimize it for negative pairs. By capturing and preserving the most relevant and rich graph features, these methods produce robust, discriminative representations that generalize well in OOD detection. Their compact interquartile ranges indicate stable performance across datasets.
Sensitivity to Anomalies in One-Class Classification: One-class classification methods (OCGIN, OCGTL) perform well overall, particularly in FPR95. These methods excel by learning a dense, representative boundary around the in-distribution (ID) data, treating everything outside this boundary as OOD. Their strength lies in handling compact, well-defined distributions, making them particularly effective in scenarios where OOD samples deviate significantly from the ID distribution. This is why they show relatively consistent performance across datasets. However, when the OOD samples closely resemble the ID samples, the separation becomes more challenging, leading to some variability in performance on metrics like AUROC and AUPRC.
Limitations of Graph Reconstruction: Graph reconstruction methods (GLADC, GraphDE) show more variability. GLADC performs reasonably well on AUROC but struggles with AUPRC and FPR95. These methods reconstruct the input graph and detect anomalies based on reconstruction errors, but when the reconstruction error does not strongly correlate with OOD instances, their effectiveness decreases. This is especially true for GraphDE, which shows poor performance and significant variability.
Dependency Pre-trained Models in Well-Traines GNNs: Well-trained GNNs (AAGOD, GOODAT) show mixed results, depending on the availability of pre- treitra model parameters. These methods rely heavily the pre-trained GNNs' quality. GOODAT is more stable across datasets, while AAGOD displays higher variability. The methods perform well when high-quality pre-trained parameters are available, but retraining without these parameters leads noticeable performance drop, indicating strong dependency on the initial pre-training phase.
Overall, contrastive learning methods stand out for their ability to capture high mutual information between graph augmentations, resulting in robust and generalizable representations. One-class classification methods perform well, especially in handling distinct OOD cases but show some sensitivity when OOD samples closely resemble ID data. Graph reconstruction methods face challenges in correlating reconstruction errors with OOD detection, leading to inconsistent results. Well-trained GNNs can be effective but are highly dependent on pre-trained models, impacting their consistency across different scenarios. This analysis highlights the importance of selecting methods that align with the characteristics of the task and data.
F.3 JUDGE SCORE DISTRIBUTION ANALYSIS
As illustrated in Fig 12 we visualize the OOD Judge score distributions for eight baseline models across three datasets (AIDS, BZR, Tox21_ HSE, and AI-DH). The X-axis shows the OOD Judge score, and the Y-axis shows the frequency. These plots demonstrate how OOD and ID samples are distributed based on their scores, highlighting the separation between these distributions. This analysis directly assesses the models' ability to distinguish OOD from ID samples, which is more task-relevant than visualizing the embedding space.
Most methods perform well on the AIDS dataset, with clear separation between OOD (red) and ID (blue) distributions. However, methods like GraphDE and SIGENT exhibit noticeable differences in distribution shapes, suggesting variability in decision boundaries and sensitivity to data characteristics.
In contrast, all methods show poor performance the Tox21 HSE dataset, with overlapping OOD and ID distributions. This overlap indicates a failure to effectively distinguish OOD from ID samples, suggesting that the dataset presents significant challenges for OOD detection.
25

---

Published as a conference paper at ICLR 2025
COOR ADD
OLoVRO ABS
OCOTL-ADS
SRAET-ABS
AAG2O- AVSS
1.00 MBA
CVTAN ABS
90c8 ABS
angeiie ABS
OC0R:
AMG0O
Freqxence
oceN Tod3 Hot
SLocKED Thazt Hit
Ocomt To23 Hel
SOVeT Tr22 Hib
A4080 be2t He
Turi
We
806RB Hit
Bopert Tart
acteal Tw21
2
Guscrto-4-4
0C0TL A-00
Grashoc A1-21
3C0
000
luge
000 ludgeadre
000
Figure 12: OOD Judge score distributions.
26

---

Published as a conference paper at ICLR 2025
Table 8: Comparison in terms of AUPRC (top). AUROC (middle), and FPR95 (bottom). The best three results are highlighted using 1st 2nd and 3rd Avg. AUROC", Avg. FPR95", "Avg. AUPRC" and "Avg. Rank" indicate the average AUROC, FPR95, AUPRC, and rank across all
datasets.
CLAD
5009
101
A3-01 -111
14.0mm
G.
11 3o
Safrec
6-5-.
3.8mmm
01
Peg Kak
19A8
1.3
Fin
R1.59M
90.87
Kil
A0000
1968
Sclass
3-4u11
47hes
001140
Blane
PROTEISS
712
G2.eca
Idhm
Kioo
6dan
85 Gthers
0.3omrr
6
10-00
Silbea
P0-001
12
12.
0.20m
2.0wa
E3
Millnn
Sucuen
13
12:88
Ci8-tweECL4D (edrovad)
2647
V1.514
96-4
Kilf
0C-51M
00-4
OCGN
GooalD
Cool.
GLADC
071040
6000
Gralic
ANC00
90000
PETITELS
10/Pzz
(133
18:18
19
40.8pmm
10-41
12
10
14
1813
11.4
14.4
9010
N.3
2544
0662
3.36
1332
1531
0.0
5.4
11.9
27


Reviews:
[{"review_id": "Wiv8f2UmY6", "reviewer": "Reviewer_1svq", "summary": "This experimental paper proposes unifying two graph machine learning tasks, graph anomaly detection (GLAD) and graph out-of-distribution detection (GLOD) in the unsupervised setting. The paper applies 35 datasets with different properties (anomaly/out-of-distribution scenarios) and applications. Moreover, the evaluation considered 18 methods proposed for anomaly detection and OOD detection that are evaluated in terms of accuracy, generalization, and efficiency. Some of the major findings in the paper are: (1) GNN-based methods achieve the best results on average even though there is no single best method, (2) the methods often struggle with near-OOD scenarios compared with far-OOD ones. and (3) some of end-to-end (GNN-based) methods are more efficient and accurate than the alternatives.", "questions": "1) How do the main findings of the paper contribute to future research on GLAD and/or GLOD?\n\n2) What is the motivation for unifying GLAD and GLOD?\n\n3) How can the findings in the paper be impacted by the dominance of molecular graphs in the benchmark?\n\n4) How are hyperparameters set in the experiments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow\n- The benchmark proposed in the paper contains 35 datasets\n- The datasets and implementations are shared as open-source", "weaknesses": "- The main findings of the paper are mostly expected\n- It is not clear why the same methods should solve anomaly detection and out-of-distribution detection\n- Most of the datasets are from the same domain\n\nDetailed comments:\n\nThese are the main findings quoted from the paper:\n\n1) \"The SOTA GLAD/GLOD methods show excellent performance on both tasks\"\n2) \"No universally superior method\"\n3) \"Inconsistent performance in terms of different metrics\"\n4) \"End-to-end methods show consistent superiority over two-step methods\"\n5) \"Near-OOD samples are harder to detect compared to far-OOD samples\"\n6) \"Poor generalizability of several GLAD/GLOD methods in specialized scenarios\"\n7) \"Performance degradation with increasing contamination ratio\"\n8) \"The sensitivity of different methods/datasets can be diverse\"\n9) \"Certain end-to-end methods outperform two-step methods in terms of both performance and computational costs\"\n\nThe only findings that one could find unexpected are 1 and maybe 9. Finding 3 is expected because of class imbalance. I am not diminishing the effort that the authors have put into running these experiments but the value of an experimental paper is what can be learned from the results. It is unclear how much can be learned from these results.\n\nRegarding the unification of the tasks, I would like to see a more clear formalization than definition 1. One could argue that an anomaly can still be in-distribution (as points can be simply ranked in terms of anomaly scores). On the other hand, an OOD point might not be an anomaly (as there might be many similar points in the test dataset). The setting considered in the experiments seems to assume that the test data points are unseen (besides being unlabelled) but if every point can be observed, the difference between an anomaly and an OOD point should be clear. This setting needs better motivation.\n\nFrom Table 1, only 8/35 datasets are not molecules. This can bias the findings towards methods that perform well on molecules, which is something not discussed in the paper. \n\nMinor comments:\n- How are the hyperparameters of the methods set without a validation set? I was not able to understand this from Appendix D.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NWPOcMLA5Z", "reviewer": "Reviewer_2xER", "summary": "This manuscript presents a unified benchmark called UB-GOLD, which aims to unify and benchmark Graph Level Anomaly Detection (GLAD) and Graph lavel OOD detection (GLOD). Specifically, the authors compare the performance of 18 GLAD/GLOD methods on 35 datasets. The performance is investigated in four different dimensions: effectiveness (using three common accuracy metrics), generalisability, robustness, and efficiency (time and memory usage). Based on extensive experiments, the authors provide insightful observations and discuss possible future directions.", "questions": "Please check the weak points.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. this manuscript is well written and easy to follow;\n2. I believe this is the first work that attempts to unify and benchmark GLAD and GLOD tasks, the findings could be broadly interesting to the graph data mining community;\n3. the authors conducted extensive experiments and open-sourced their code base (which I believe is easy to extend), making contributions to future research;", "weaknesses": "## Major Comments\n\n### 1. Unification and Benchmarking of GLAD and GLOD Tasks\nAs the first work attempting to unify (please indicate the related work if this is not true) and benchmark GLAD (Graph-Level Anomaly Detection) and GLOD (Graph-Level Out-of-Distribution) detection tasks, I believe the authors should dedicate more space and effort to this unification process. Specifically:\n\n- **1.1) Formal Definitions**: It would be beneficial for the authors to formally (with math symbols) define GLAD and GLOD before introducing the concept of \"Unsupervised generalized graph-level OOD detection.\" Providing these definitions upfront would establish clarity and context for readers unfamiliar with these tasks.\n\n- **1.2) Relationship Between GLAD and GLOD**: The authors should discuss the relationship between GLAD and GLOD in more depth, as there seems to be room for clarification on whether they perceive OOD detection as a broader concept than anomaly detection. In my view, anomaly detection is a broader concept, as abnormal instances may either (1) exist within the distribution but in low-density regions or (2) represent out-of-distribution instances. Including a specific section comparing and contrasting GLAD and GLOD, with formal definitions and discussions of their conceptual overlap and distinctions, would enhance the unification effort and clarify this relationship.\n\n---\n\n### 2. Experimental Details and Completeness\nThere are some missing details regarding the experiments and results. Adding these would make the manuscript more self-contained, especially as it aims to serve as a benchmark paper.\n\n- **2.1) Definition and Generation of Anomalies/OOD Samples**: To make the manuscript self-contained, please provide details in the appendix on how the anomalies or OOD samples are defined or generated in each dataset. Clear explanations will help readers replicate and understand your experiments.\n\n- **2.2) Completeness of Results in Figures**: In Figure 4, results are only provided for 15 out of 19 methods, with similar omissions in Figures 5 and 6. If these omissions are due to space limitations, including the full results in the appendix would help ensure completeness. Please also add a brief discussion explaining why certain methods were omitted from the main figures, if applicable.\n\n- **2.3) Metrics and Additional Results**: In Figures 4 and 5, please specify which metrics are being used. Additionally, including complete results for all metrics in the appendix, along with brief analyses, would make the benchmark more comprehensive and allow for a better understanding of the methods' performances.\n\n---\n\n### 3. Clarification of Experimental Settings and Terminology\n\n- **3.1) Hyperparameter Search**: In the hyperparameter search section, the authors conduct a random search to optimize hyperparameters based on **test set performance**. This approach raises questions regarding practical applicability. Could you clarify if a validation set was used instead of the test set for hyperparameter tuning, or if there was a specific reason for using the test set? Also, discussing the implications of this choice on the generalizability of your results would provide valuable context.\n\n- **3.2) Use of the Term \"Generalisability\"**: The term \"generalisability\" is typically associated with a model’s performance on unseen data, while the manuscript uses it to describe differences in OOD detection for near- and far-OOD samples. Alternative terms, such as \"OOD detection range\" or \"OOD sensitivity spectrum,\" may better capture the intended meaning. Clarifying or refining this terminology would enhance reader understanding.\n\n---\n\n## Minor Comments and Suggestions\n\n- **4.1) Page 4, Line 084: Typo**: Please correct \"28 GLAD and GLOD methods\" to \"18 GLAD and GLOD methods.\"\n\n- **4.2) Anomaly Ratio in Table 1**: In Table 1, adding the anomaly ratio for each dataset would make the data overview more informative.\n\n- **4.3) Visualization of CPU and GPU Results in Figure 6**: In Figure 6, consider superimposing the CPU and GPU bars (distinguished by color) to make the visual comparison clearer.\n\n- **4.4) OOD Judge Score Distribution Analysis**: On Page 5, you mention an \"OOD Judge Score Distribution Analysis,\" but this concept was not referenced later in the main text, though it appears in the appendix. Including a reference in the main text would clarify its relevance.\n\n- **4.5) \"Well-trained GNNs\" on Page 5**: On Page 5, the term \"well-trained GNNs\" is used. If this refers to pre-trained GNNs, please provide a brief explanation for clarity.\n\n---\n\n## Conclusion\nOverall, I appreciate the authors’ efforts in developing a unified framework and benchmark for GLAD and GLOD tasks. Addressing these points would improve the manuscript’s clarity, depth, and completeness, making it a more robust resource for the community. And I will consider increasing my rating if they (or some of them) are well addressed.\n\n---", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "107gl4cL5D", "reviewer": "Reviewer_yW1o", "summary": "This paper unifies the unsupervised  graph level ood and anomaly detection as generalized graph level ood detection problem and conduct a comprehensive and unified benchmark. This paper provide some remarkable observation through the comprehensive comparison and analyses.", "questions": "1. What is the difference between near-OOD samples and anomalies if both come from the same dataset?\n2. If some OOD samples or anomalies are integrated into the training set as contamination, are they still retained in the test set during inference?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A large amount of datasets are employed to establish the benchmark to fairly compare the mutiple GLAD and GLOD emthods under a unified experimental setting.\n2. In addition to the performance, the paper also investigates more characteristics of methods including generalizability, robustness, and efficiency.\n3. The comprehensive benchmark and the multiple observations are provided in the experimental analysis to inspire future work.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            \n4. There is no clear distinction between the training set and test set in unsupervised GLAD. Leveraging unsupervised graph-level anomaly detection methods for OOD detection is a good point.", "weaknesses": "1. In GLAD, anomalies can be considered as out-of-distribution (OOD) test samples for type 1 and type 2, but further clarification is needed in the paper on how the in-distribution (ID) test samples are constructed.\n2. In an unsupervised setting, does this mean the test set can also be used as the training set? The in-distribution samples in the test set could potentially help improve performance.\n3. From the several observations, we found there are large overlaps between OOD samples,  in addition to the experimental explanation, more specific analysis is needed on this point for particular datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9x3QoiG2a0", "reviewer": "Reviewer_cQQu", "summary": "The paper introduces UB-GOLD (Unified Benchmark for unsupervised Graph-level OOD and anomaly Detection), a comprehensive benchmark that unifies two related tasks in graph machine learning: graph-level anomaly detection (GLAD) and graph-level out-of-distribution detection (GLOD). UB-GOLD bridges this gap by providing a unified evaluation across 18 representative methods and 35 datasets, covering various scenarios in anomaly and OOD detection. The benchmark offers a multi-dimensional analysis of methods, assessing their effectiveness, generalizability, robustness, and efficiency, while also providing an open-source codebase to facilitate reproducible research and encourage further exploration.", "questions": "①Performance of SSL-D and GK-D on Graph Structure-Based Datasets: In datasets like IC50-Size, where out-of-distribution samples primarily differ from in-distribution ones in graph structure, SSL-D and GK-D methods show poor performance. Could you provide insights into why these methods struggle with such datasets? Is it related to the specific way these methods handle graph structure?\n\n②Handling Near-OOD Samples: You mention that near-OOD samples are more difficult to detect than far-OOD samples. Could you elaborate on what specific characteristics of near-OOD samples make them harder to distinguish? Do you have any thoughts on how future work could address this issue?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "①The paper makes an original contribution by unifying GLAD and GLOD into a single benchmark, UB-GOLD. This creative combination highlights the conceptual overlap between the two tasks, simplifying evaluation in both areas.\n\n②The paper's experiments evaluate 18 methods on 35 datasets. It offers a multi-dimensional analysis. The breadth of datasets used ensures reliable, real-world applicability, while the open-source codebase.\n\n③UB-GOLD offers insights into detecting near-OOD samples and noisy training data. These findings current limitations in existing methods. What's more, this work focus on generalizability, robustness, and efficiency ensures the benchmark’s practical relevance across different application domains, making it a useful resource in the field.", "weaknesses": "①While the paper provides extensive comparisons across methods, it could benefit from a more in-depth discussion of why certain methods fail in specific scenarios. For instance, understanding the root causes of poor performance under noisy training data or near-OOD conditions.\n\n②Consider adding a more concise summary section that highlights the most important findings from your multi-dimensional analyses. This could be in the form of a bulleted list or a short paragraph at the end of each results subsection to help readers quickly digest key insights.\n\n③In datasets like IC50-Size, the out-of-distribution samples differ primarily in graph structure from the in-distribution ones. It’s unexpected that SSL-D and GK-D methods perform poorly on these datasets. Could the authors offer an explanation for these results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)