Unifying Unsupervised Graph-Level Anomaly Detection and Out-of-Distribution Detection: A Benchmark

Authors: Yili Wang, Yixin Liu, Xu Shen, Chenyu Li, Rui Miao, Kaize Ding, Ying Wang, Shirui Pan, Xin Wang
Primary area: datasets and benchmarks
Keywords: Graph out-of-distribution detection; graph anomaly detection; benchmark

Abstract:
To build safe and reliable graph machine learning systems, unsupervised graph-level anomaly detection (GLAD) and unsupervised graph-level out-of-distribution (OOD) detection (GLOD) have received significant attention in recent years. Though these two lines of research share the same objective, they have been studied independently in the community due to distinct evaluation setups, creating a gap that hinders the application and evaluation of methods from one to the other. To bridge the gap, in this work, we present a Unified Benchmark for unsupervised Graph-level OOD and anomaly Detection (UB-GOLD), a comprehensive evaluation framework that unifies GLAD and GLOD under the concept of generalized graph-level OOD detection. Our benchmark encompasses 35 datasets spanning four practical anomaly and OOD detection scenarios, facilitating the comparison of 18 representative GLAD/GLOD methods. We conduct multi-dimensional analyses to explore the effectiveness, generalizability, robustness, and efficiency of existing methods, shedding light on their strengths and limitations. Furthermore, we provide an open-source codebase of UB-GOLD to foster reproducible research and outline potential directions for future investigations based on our insights.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1729741622061

thread_type:
official_review

<thread_object>
creator: reviewer_cqqu

<content>
summary:
The paper introduces UB-GOLD (Unified Benchmark for unsupervised Graph-level OOD and anomaly Detection), a comprehensive benchmark that unifies two related tasks in graph machine learning: graph-level anomaly detection (GLAD) and graph-level out-of-distribution detection (GLOD). UB-GOLD bridges this gap by providing a unified evaluation across 18 representative methods and 35 datasets, covering various scenarios in anomaly and OOD detection. The benchmark offers a multi-dimensional analysis of methods, assessing their effectiveness, generalizability, robustness, and efficiency, while also providing an open-source codebase to facilitate reproducible research and encourage further exploration.

strengths:
①The paper makes an original contribution by unifying GLAD and GLOD into a single benchmark, UB-GOLD. This creative combination highlights the conceptual overlap between the two tasks, simplifying evaluation in both areas.

②The paper's experiments evaluate 18 methods on 35 datasets. It offers a multi-dimensional analysis. The breadth of datasets used ensures reliable, real-world applicability, while the open-source codebase.

③UB-GOLD offers insights into detecting near-OOD samples and noisy training data. These findings current limitations in existing methods. What's more, this work focus on generalizability, robustness, and efficiency ensures the benchmark’s practical relevance across different application domains, making it a useful resource in the field.

weaknesses:
①While the paper provides extensive comparisons across methods, it could benefit from a more in-depth discussion of why certain methods fail in specific scenarios. For instance, understanding the root causes of poor performance under noisy training data or near-OOD conditions.

②Consider adding a more concise summary section that highlights the most important findings from your multi-dimensional analyses. This could be in the form of a bulleted list or a short paragraph at the end of each results subsection to help readers quickly digest key insights.

③In datasets like IC50-Size, the out-of-distribution samples differ primarily in graph structure from the in-distribution ones. It’s unexpected that SSL-D and GK-D methods perform poorly on these datasets. Could the authors offer an explanation for these results?

questions:
①Performance of SSL-D and GK-D on Graph Structure-Based Datasets: In datasets like IC50-Size, where out-of-distribution samples primarily differ from in-distribution ones in graph structure, SSL-D and GK-D methods show poor performance. Could you provide insights into why these methods struggle with such datasets? Is it related to the specific way these methods handle graph structure?

②Handling Near-OOD Samples: You mention that near-OOD samples are more difficult to detect than far-OOD samples. Could you elaborate on what specific characteristics of near-OOD samples make them harder to distinguish? Do you have any thoughts on how future work could address this issue?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730176794846

thread_type:
official_review

<thread_object>
creator: reviewer_yw1o

<content>
summary:
This paper unifies the unsupervised  graph level ood and anomaly detection as generalized graph level ood detection problem and conduct a comprehensive and unified benchmark. This paper provide some remarkable observation through the comprehensive comparison and analyses.

strengths:
1. A large amount of datasets are employed to establish the benchmark to fairly compare the mutiple GLAD and GLOD emthods under a unified experimental setting.
2. In addition to the performance, the paper also investigates more characteristics of methods including generalizability, robustness, and efficiency.
3. The comprehensive benchmark and the multiple observations are provided in the experimental analysis to inspire future work.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
4. There is no clear distinction between the training set and test set in unsupervised GLAD. Leveraging unsupervised graph-level anomaly detection methods for OOD detection is a good point.

weaknesses:
1. In GLAD, anomalies can be considered as out-of-distribution (OOD) test samples for type 1 and type 2, but further clarification is needed in the paper on how the in-distribution (ID) test samples are constructed.
2. In an unsupervised setting, does this mean the test set can also be used as the training set? The in-distribution samples in the test set could potentially help improve performance.
3. From the several observations, we found there are large overlaps between OOD samples,  in addition to the experimental explanation, more specific analysis is needed on this point for particular datasets.

questions:
1. What is the difference between near-OOD samples and anomalies if both come from the same dataset?
2. If some OOD samples or anomalies are integrated into the training set as contamination, are they still retained in the test set during inference?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730625807566

thread_type:
official_review

<thread_object>
creator: reviewer_2xer

<content>
summary:
This manuscript presents a unified benchmark called UB-GOLD, which aims to unify and benchmark Graph Level Anomaly Detection (GLAD) and Graph lavel OOD detection (GLOD). Specifically, the authors compare the performance of 18 GLAD/GLOD methods on 35 datasets. The performance is investigated in four different dimensions: effectiveness (using three common accuracy metrics), generalisability, robustness, and efficiency (time and memory usage). Based on extensive experiments, the authors provide insightful observations and discuss possible future directions.

strengths:
1. this manuscript is well written and easy to follow;
2. I believe this is the first work that attempts to unify and benchmark GLAD and GLOD tasks, the findings could be broadly interesting to the graph data mining community;
3. the authors conducted extensive experiments and open-sourced their code base (which I believe is easy to extend), making contributions to future research;

weaknesses:
## Major Comments

### 1. Unification and Benchmarking of GLAD and GLOD Tasks
As the first work attempting to unify (please indicate the related work if this is not true) and benchmark GLAD (Graph-Level Anomaly Detection) and GLOD (Graph-Level Out-of-Distribution) detection tasks, I believe the authors should dedicate more space and effort to this unification process. Specifically:

- **1.1) Formal Definitions**: It would be beneficial for the authors to formally (with math symbols) define GLAD and GLOD before introducing the concept of "Unsupervised generalized graph-level OOD detection." Providing these definitions upfront would establish clarity and context for readers unfamiliar with these tasks.

- **1.2) Relationship Between GLAD and GLOD**: The authors should discuss the relationship between GLAD and GLOD in more depth, as there seems to be room for clarification on whether they perceive OOD detection as a broader concept than anomaly detection. In my view, anomaly detection is a broader concept, as abnormal instances may either (1) exist within the distribution but in low-density regions or (2) represent out-of-distribution instances. Including a specific section comparing and contrasting GLAD and GLOD, with formal definitions and discussions of their conceptual overlap and distinctions, would enhance the unification effort and clarify this relationship.

---

### 2. Experimental Details and Completeness
There are some missing details regarding the experiments and results. Adding these would make the manuscript more self-contained, especially as it aims to serve as a benchmark paper.

- **2.1) Definition and Generation of Anomalies/OOD Samples**: To make the manuscript self-contained, please provide details in the appendix on how the anomalies or OOD samples are defined or generated in each dataset. Clear explanations will help readers replicate and understand your experiments.

- **2.2) Completeness of Results in Figures**: In Figure 4, results are only provided for 15 out of 19 methods, with similar omissions in Figures 5 and 6. If these omissions are due to space limitations, including the full results in the appendix would help ensure completeness. Please also add a brief discussion explaining why certain methods were omitted from the main figures, if applicable.

- **2.3) Metrics and Additional Results**: In Figures 4 and 5, please specify which metrics are being used. Additionally, including complete results for all metrics in the appendix, along with brief analyses, would make the benchmark more comprehensive and allow for a better understanding of the methods' performances.

---

### 3. Clarification of Experimental Settings and Terminology

- **3.1) Hyperparameter Search**: In the hyperparameter search section, the authors conduct a random search to optimize hyperparameters based on **test set performance**. This approach raises questions regarding practical applicability. Could you clarify if a validation set was used instead of the test set for hyperparameter tuning, or if there was a specific reason for using the test set? Also, discussing the implications of this choice on the generalizability of your results would provide valuable context.

- **3.2) Use of the Term "Generalisability"**: The term "generalisability" is typically associated with a model’s performance on unseen data, while the manuscript uses it to describe differences in OOD detection for near- and far-OOD samples. Alternative terms, such as "OOD detection range" or "OOD sensitivity spectrum," may better capture the intended meaning. Clarifying or refining this terminology would enhance reader understanding.

---

## Minor Comments and Suggestions

- **4.1) Page 4, Line 084: Typo**: Please correct "28 GLAD and GLOD methods" to "18 GLAD and GLOD methods."

- **4.2) Anomaly Ratio in Table 1**: In Table 1, adding the anomaly ratio for each dataset would make the data overview more informative.

- **4.3) Visualization of CPU and GPU Results in Figure 6**: In Figure 6, consider superimposing the CPU and GPU bars (distinguished by color) to make the visual comparison clearer.

- **4.4) OOD Judge Score Distribution Analysis**: On Page 5, you mention an "OOD Judge Score Distribution Analysis," but this concept was not referenced later in the main text, though it appears in the appendix. Including a reference in the main text would clarify its relevance.

- **4.5) "Well-trained GNNs" on Page 5**: On Page 5, the term "well-trained GNNs" is used. If this refers to pre-trained GNNs, please provide a brief explanation for clarity.

---

## Conclusion
Overall, I appreciate the authors’ efforts in developing a unified framework and benchmark for GLAD and GLOD tasks. Addressing these points would improve the manuscript’s clarity, depth, and completeness, making it a more robust resource for the community. And I will consider increasing my rating if they (or some of them) are well addressed.

---

questions:
Please check the weak points.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730670289250

thread_type:
official_review

<thread_object>
creator: reviewer_1svq

<content>
summary:
This experimental paper proposes unifying two graph machine learning tasks, graph anomaly detection (GLAD) and graph out-of-distribution detection (GLOD) in the unsupervised setting. The paper applies 35 datasets with different properties (anomaly/out-of-distribution scenarios) and applications. Moreover, the evaluation considered 18 methods proposed for anomaly detection and OOD detection that are evaluated in terms of accuracy, generalization, and efficiency. Some of the major findings in the paper are: (1) GNN-based methods achieve the best results on average even though there is no single best method, (2) the methods often struggle with near-OOD scenarios compared with far-OOD ones. and (3) some of end-to-end (GNN-based) methods are more efficient and accurate than the alternatives.

strengths:
- The paper is well-written and easy to follow
- The benchmark proposed in the paper contains 35 datasets
- The datasets and implementations are shared as open-source

weaknesses:
- The main findings of the paper are mostly expected
- It is not clear why the same methods should solve anomaly detection and out-of-distribution detection
- Most of the datasets are from the same domain

Detailed comments:

These are the main findings quoted from the paper:

1) "The SOTA GLAD/GLOD methods show excellent performance on both tasks"
2) "No universally superior method"
3) "Inconsistent performance in terms of different metrics"
4) "End-to-end methods show consistent superiority over two-step methods"
5) "Near-OOD samples are harder to detect compared to far-OOD samples"
6) "Poor generalizability of several GLAD/GLOD methods in specialized scenarios"
7) "Performance degradation with increasing contamination ratio"
8) "The sensitivity of different methods/datasets can be diverse"
9) "Certain end-to-end methods outperform two-step methods in terms of both performance and computational costs"

The only findings that one could find unexpected are 1 and maybe 9. Finding 3 is expected because of class imbalance. I am not diminishing the effort that the authors have put into running these experiments but the value of an experimental paper is what can be learned from the results. It is unclear how much can be learned from these results.

Regarding the unification of the tasks, I would like to see a more clear formalization than definition 1. One could argue that an anomaly can still be in-distribution (as points can be simply ranked in terms of anomaly scores). On the other hand, an OOD point might not be an anomaly (as there might be many similar points in the test dataset). The setting considered in the experiments seems to assume that the test data points are unseen (besides being unlabelled) but if every point can be observed, the difference between an anomaly and an OOD point should be clear. This setting needs better motivation.

From Table 1, only 8/35 datasets are not molecules. This can bias the findings towards methods that perform well on molecules, which is something not discussed in the paper. 

Minor comments:
- How are the hyperparameters of the methods set without a validation set? I was not able to understand this from Appendix D.

questions:
1) How do the main findings of the paper contribute to future research on GLAD and/or GLOD?

2) What is the motivation for unifying GLAD and GLOD?

3) How can the findings in the paper be impacted by the dominance of molecular graphs in the benchmark?

4) How are hyperparameters set in the experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732158962141

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank all the reviewers for their valuable and constructive feedback. We are pleased that the reviewers recognize the **novelty and originality** of unifying GLAD and GLOD into a single benchmark (Reviewer 2xER, cQQu), the **comprehensive and multi-dimensional evaluation** provided by the UB-GOLD benchmark (Reviewer yW1o, cQQu), the **open-source datasets and codebase** that facilitate reproducibility and encourage future research (Reviewer 2xER, cQQu), and the **clarity and accessibility of the writing** (Reviewer 1svq, 2xER).

To the best of our efforts, we have provided detailed responses to address the concerns raised by each reviewer in the following. Meanwhile, we have carefully revised the paper according to the reviewers’ comments. All feedback has been incorporated into the revised version of the manuscript, and all changes and additions are highlighted in blue for easy reference.

Specifically, the main modifications we made are as follows.

* We provided formal definitions and discuss the relationships of GLAD, GLOD, and the unified unsupervised generalized graph-level OOD detection.
* We added a more detailed explanation of the dataset partitioning in the appendix.
* We replaced the term "generalization" in the near-far OOD detection task description with a more suitable term "OOD sensitivity spectrum".
* We included additional experimental results of metrics for the perturbation task in the appendix.
* We further elaborated on the motivation and contribution of the proposed benchmark UB-GOLD.
* We provided more experimental results in terms of additional metrics in the perturbation task, providing a more complete evaluation for the existing methods.


We hope these revisions address all the reviewers’ concerns and improve the manuscript. Thank you again for your constructive feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes an unified benchmark for graph-level anomaly detection / out of distribution detection (UB-GOLD).  The authors observe overlaps between the disparate communities for GLAD and GLOD and propose this benchmark to unify evaluation and benchmarking across 35 datasets and several detection scenarios.  

Reviewers were unilaterally positive about this work and consistently rated it positively.  Nonetheless, there were a few limitations raised:

- Some remaining lack of clarity from reviewers around why GLAD and GLOD are conceptually overlapping problems, related some confusing missing definitions (1svq, 2xER)

- Confusions around validity of using test-set performance for hyperparameter evaluation and the train/test setup in general (1svq, 2xER, yW1o)

- More depth of discussion around the difficulty of near-OOD samples and impact of noise during training (cQQu, yW1o)

I encourage the authors to take these considerations into the revision.