Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a comprehensive unsupervised node outlier detection benchmark called UNOD that aims to understand graph OD algorithms’ performance better. It examines fourteen OD methods on real-world and synthetic datasets and different types of outliers. Additionally, this paper creates Python Graph Outlier Detection (PyGOD), including more than ten latest graph OD algorithms. 

strengths: This paper has a clear motivation. The benchmark is well documented, and the paper is also well written and easily followed. 

weaknesses: This research makes a novel attempt to measure the performance of various OD methods using taxonomy node outliers. This taxonomy, however, is too intuitive and unrelated to realistic datasets. Thus, despite the fact that this taxonomy can more accurately assess the effectiveness of current methods on synthetic datasets, the result is difficult to apply to real-world scenarios. I suggest that authors connect this part to real-world datasets, for example, by attempting to analyze current real-world datasets using this taxonomy or by connecting Table 4 and the real-world datasets parts of Table 3.

This benchmark places a strong emphasis on efficiency and scalability measurements. The largest dataset used in this study, however, is Flickr, which has just 89K nodes. This number has a gap with numerous real-world scenarios, which may restrict the applicability of efficiency and scalability analysis. I therefore strongly suggest the authors include more substantially large-scale datasets in their experiments, such as Amazon [1] (more than 4M edges), YelpChi [1] (more than 3M edges), Elliptic [2] (more than 203K nodes) and DGraph [3] (more than 3M nodes).

[1]  Yingtong Dou, Zhiwei Liu, Li Sun, Yutong Deng, Hao Peng, and Philip S Yu. Enhancing graph neural network-based fraud detectors against camouflaged fraudsters. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 315–324, 2020.

[2] Mark Weber, Giacomo Domeniconi, Jie Chen, Daniel Karl I Weidele, Claudio Bellei, Tom Robinson, and Charles E Leiserson. Anti-money laundering in bitcoin: Experimenting with graph convolutional networks for financial forensics. arXiv preprint arXiv:1908.02591, 2019.

[3] Xuanwen Huang, Yang Yang, Yang Wang, Chunping Wang, Zhisheng Zhang, Jiarong Xu, and Lei Chen. DGraph: A Large-Scale Financial Dataset for Graph Anomaly Detection. Preprint, 2022.
correctness: The evaluation methods and experiment design is appropriate and performed correctly.
clarity: This paper is well written.
relation to prior work: There are one or two papers missing, but they aren't crucial omissions.
additional feedback: See Weaknesses.
review_title: A unified benchmark for UNOD

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: By defining the problem of “Unsupervised Node Outlier Detection on the Static Attributed Graph”, the authors conducted experiments with 14 detectors on 5 datasets. 
The main contribution is the 14 detectors’ performance reports with taking below factors into account: 
(1)	the choice of detectors from different years (2000, 2007, 2012, 2014, 2016, 2017, 2018, 2019, 2020, 2021, 2022);
(2)	the choice of outlier types including: injected outliers and organic outliers;
(3)	the evaluation metrics: accuracy, runtime, memory usage;

strengths: 
 The paper is good in organization.

 I appreciate the code and datasets used are open

weaknesses: (1)	The problem is defined on the Static Attributed Graph (Problem Definition in section 3.1), while the title claims “Benchmarking Node Outlier Detection on Graphs”.

(2)	Given 14 detectors tested, only 8 of them [1,4,6,7,16,23,32,33] are originally designed for Attributed Graph. The real motivation for comparison to non-Attributed-Graph methods is missing.

(3)	Given 5 datasets, 2 of them contain organic outliers, while 3 of them contain outliers generated by the authors. However, since the datasets play a very vital role in experiments, the limitations of the method of generating outlier should be discussed. The data quality needs more guarantee, instead of merely using random injection. Otherwise, how can it be convincing that the dataset with injected outliers are trustworthy to the community.  

(4)	From the experiment reports, there is no new findings to outlier detection domain listed in [1] published in 2017. 

(5)    The authorship is very confusing. 
correctness: uncertain
(1) limitations of the datasets with generated outliers are missing, leading to uncertain risky to the conclusions drawn.
(2) all datasets are in small scale (<10w nodes)
(3) common parameter settings for different detectors are the same, not sure if this is fair or unfair actually. Why not try also the default settings of each detector? 
clarity: yes
relation to prior work: yes
additional feedback: no
review_title: can be good work after massive improvement

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: A framework is presented for evaluating unsupervised graph anomaly detection methods that flag anomalous nodes in a given attributed graph. Synthetic graphs with injected anomalies and real world graphs with known anomalies are used as datasets. Multiple conclusions are presented based on the performance of the methods on these datasets, given the type of the method (deep, graph, features), what signals it uses (graph, features, etc), as well as type of injected anomalies.  This could be a very valuable resource but there are weaknesses that need to be addressed to make it impactful.
strengths: Paper is very well written and easy to read
The code, framework for datasets seems to be packaged nicely and is easy to use

weaknesses: Small number of real world datasets: only 3 real world dataset with anomalies are included which also do not show the same patterns with the datasets with injected anomalies. Given the evaluation of anomaly detection methods is hard, and results show inconsistency and depend on dataset/domain, it could have been more appropriate to have a larger set of real world datasets included, maybe even some effort into categorizing them. For example, datasets used in ref 57 for evaluation can be included and/or in Kumar, Srijan, et al. "Rev2: Fraudulent user prediction in rating platforms.". 

There is little/no discussion/recommendations on inconsistencies between the performance metrics used, e.g. recall @k in appendix and AUC in the main paper don't agree, ranking of methods changes, and hence conclusions based on that.  


correctness: Paper is correct and sound.
clarity: Very well written and well organized, easy to read.

"Since there is no dedicated algorithm for the isolated structural outlier, we would not discuss it in our benchmark"> what does this mean? there are methods that find outliers in graphs, so it is not clear what this statement is suggesting. 


relation to prior work: Well positioned but still some prior work are missing, e.g. Fraudar, which is understandable given the number of works on this topic.
additional feedback: NA
review_title: Benchmarking graph anomaly detection

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work provides a comprehensive unsupervised node outlier detection benchmark for graphs. In particular, it evaluates 14 methods, ranging from classical to graph neural network models, on 5 different datasets with synthetic or organic outliers.
The paper provides a definition of two different types of outliers (structural and contextual) and injects synthetic outliers into the datasets based on the proposed definitions.
The paper also compares the methods based on the efficiency and scalability of the algorithms by runtime and GPU memory usage on synthetic graphs at different scales.

strengths: -	The considered problem has attracted a lot of attention and is a very important problem with many usecases in different application domains.
-	Different range of methods from general outlier detection methods to graph neural network based methods is evaluated. Different performance metrics have been analyzed. In addition to efficiency, scalability of the methods based on the runtime and GPU memory of different methods is also evaluated.
-	Two different categories of outliers are identified and the power of different methods for detecting each type of outliers have been analyzed.

weaknesses: -	There are some grammar issues throughout the paper.
-	There are some proposed future research directions. Although discussing future direction is insightful, it should not be considered as a contribution of the paper.
-	Nine deep node outlier detection methods have been selected as baselines. However, there is no discussion on the intuition behind selecting these methods among all other available methods. This also applies to the non-existence of enough explanation on the selection criteria of graph-based methods.
-	There is no reference or discussion that could consolidate Definition 2 (Structural Outlier); why densely connected nodes are considered outliers? 
-	The datasets are mostly small or medium size.
-	There is no discussion on the contextual vs. structural outliers on datasets with real outliers (i.e., Weibo & Reddit). Considering the lack of investigation of different types of outliers on datasets with real outlies, what is the motivation behind presenting the two kinds of outliers?
-	The caption of Table 3: … five real-world benchmark datasets…  this is misleading; actually, only two datasets contain actual outliers and the other three have injected outliers that are synthetic.
-	It has been shown that the performance on synthetic outliers does not generalize to organic outliers. It would be nice if more efficient methods for injecting or generating outliers were discussed.
-	Observing that the synthetic outliers are not good indicator of the actual outliers, what is the importance of evaluating the performance of synthetic outliers based on separating different types of outliers?
-	Figure 2: if the sequence of the methods in the legend follows the sequence of the bars, it could improve readability.
-	Section 5 has almost summarizes the previously observed results in related domains such as anomaly detection in other formats like tabular data.

correctness: -	The paper is a benchmarking paper. The evaluations make sense, however it could improve for better quality. Also, the results could be better investigated and explained.
clarity: Totally, the paper is well-written, there are only some grammar issues (e.g., Section 4: research questions are incomplete sentences.)
relation to prior work: The discussion on the previous work is clear. 
additional feedback: The impact of different kinds of outliers on datasets with organic outliers could be investigated. Better methods for injecting outliers to follow the behavior of real outliers could be considered and analyzed. The comparison between classical anomaly detection methods vs. graph-based or deep methods could be more elaborated to better incentivize the usage of one category of the methods (based on the dataset characteristics of types of outliers).
review_title: This work benchmarks a diverse set of unsupervised node outlier detection methods on several datasets for synthetic and organic outliers. The baseline methods span from classical to graph neural network models. The datasets include injected synthetic or organic outliers.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a benchmark for node outlier detection, which contains 2 outlier generators to construct the dataset.13 object detection methods are included in the benchmark for effectiveness and efficiency comparison.
strengths: 1. It is a well-written paper that addresses an important need within the fair comparison of node OD. 
2. The analysis for efficiency and scalability is important but has been ignored in previous literature. This work is a pioneer for these evaluations and I believe it is worthwhile for the fairness community.
3. The datasets and the documents for this benchmark are complete and accessible.
4. Beyond the benchmark contribution, the empirical analysis the authors provide is instructive and highlights the potential direction of node OD.

weaknesses: 1. For structural outlier, it is too strict to construct the outlier subgraphs as cliques, which could make this kind of outlier too easy to be detected. Intuitively, humans can be aware of the outlier subgraph if it is obviously denser than the rest part.

2. The outlier number for each dataset is confusing. As mentioned in Appendix A.1.3, the outlier number for each outlier type is m×n. But in Table 1 of the dataset subsection, the outlier number equals n.

correctness: Not all datasets support the conclusion in Section 4.2.1 about “there exists a trade-off between algorithm stability and potential”. This claim is strong. For instance, in Table 3, the best method on the Weibo dataset performs well for both stability and potential.

The evaluation of the benchmark is appropriate.

For dataset construction, there is a conflict about the outlier number, which needs further clarification.

clarity: The paper is well written and easy to understand.
relation to prior work: The relation to existing work is clearly discussed in the paper.
additional feedback: The best results in the experiments in the Appendix should be highlighted.
review_title: A solid benchmark for node-level outlier detection. Some clarifications are needed.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: 
This work aims to address the lack of a standard and unified benchmark for node outlier detection. To this end, 

* the authors proposed the first comprehensive unsupervised node outlier detection benchmark for graphs (UNOD) and examined fourteen existing outlier detection methods covering both non deep learning, deep learning and GNN based ones. 

* An open sourced library called PyGOD is also included to reproduce many of these OD algorithms and the evaluation pipeline. 

* The authors then classifier node outliers into structural and contextual outliers and methods have uneven performance across these two types of outliers.

* GPU memory consumption and runtime are also considered on top of current performance measures such as ROC-AUC. 

* the authors also points out some future research directions. 
strengths: * It is important to have a unified benchmark for node outlier detection to facilitate fair comparison between methods and understand the progress in the field

* code is open sourced with good documentation 

* a variety of methods are included in the benchmark and code for future research

* the paper includes interesting observations drawn from the benchmark such as synthetic outliers might not generalize to organic outliers and low-order structure information are sufficient for detecting structural outliers. 
weaknesses: * **definition of structural outlier might be too narrow** (only densely connected nodes), I would suggest a more general definition. in static graphs, there exist natural dense hubs such as community leaders and they are not necessarily outliers. No experiments are conducted on more complex structural outliers. Due to this narraw definition, the injection of sturctural outlier is too easy to detect. Such synthetic outliers can be perfectly detected by simply counting the node degree in a single pass of all edges with 0 parameter. one suggested variation would be to add anomalous community nodes: form a community for nodes which doesn't initially have a community structure and have such group of nodes as anomalies. This is more similar to the example claimed in the paper "Structural outlier is prevalent in real-world graphs as members of organized fraud gangs usually interconnect with each other to carry out malicious activities" (see line 118)


* **lack of datasets.** There are currently 5 real world datasets and only 2 of them has true (non-synthetic) anomalies. In general, lack of labelled dataset is a huge challenge in anomaly detection over graphs. However, there are other available labeled datasets in the dynamic graph anomaly detection literature which can be converted to static graphs. For example, some well-known datasets are 
  1. Enron email dataset
  2. DARPA dataset
  3. UCI Message
  4. Digg social network
see [Fast and Accurate Anomaly Detection in Dynamic Graphs with a Two-Pronged Approach (KDD 2019)]( https://arxiv.org/pdf/2011.13085.pdf) and [AddGraph: Anomaly Detection in Dynamic Graph Using Attention-based
Temporal GCN](https://www.ijcai.org/proceedings/2019/0614.pdf) As this is a dataset track paper, it would be important to include more datasets.

* **some contribution claims are weak** (see highlights in Introduction Section). The additional performance measures such as computational time and GPU usage are welcome to be compared between different methods however this is not novel concept and it is in general measured in anomaly detection literature. Minor note: future directions should be reserved in discussion or conclusion section only.


correctness: datasets are constructed / collected in a sound way.

some contribution claims are not really novel and the synthetic structural anomaly definition & experiments can be improved (see weakness section). 


clarity: The paper is well-written and easy to understand

relation to prior work: Relation to prior work has been discussed well. 
additional feedback: I would consider raising my score accordingly if the authors can: 
  * add additional synthetic outliers, improve the definition of the structural outliers. 
  * add more datasets such as the ones I discussed in weakness section. 

[update] The authors have addressed my concern for lack of datasets and added more synthetic structural anomaly definition.
review_title: Important Work that Needs Improvement

Meta-review: metareview: This paper focuses on the task of anomaly or outlier detection in graphs. It provides a formulation of the notion of anomalies that enables synthetic injection of them into existing data, in addition to the consideration of organic anomalies. While reviewers raised certain concerns about the benchmarking procedures, as well as the used data (e.g., the use of synthetic anomalies vs. organic ones) in the initial version, most of these have been addressed during the rebuttal and discussion period, and at this point all reviewers ultimately recommend acceptance. Therefore, I also recommend acceptance of the paper.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept