Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a new financial-related dataset in the field of graph anomaly detection, which is both dynamic and large-scale. This dataset fills the lack of large-scale datasets for dynamic graph anomaly detection to a certain extent. The article also analyzes some features of the data itself and provides researchers with a convenient way to access the dataset and a uniform benchmark test.
strengths: Many current GAD-related works on dynamic graphs use datasets that are often small in size, and sometimes even add some anomalous nodes manually on the normal graphs, which does not facilitate comparison with other related works. This dataset presents a large-scale dynamic graph dataset and provides a convenient way to download and make comparisons with other GAD work. The paper also conducts experiments and analyses for the case that the labels or attributes of all nodes are not guaranteed to be complete in large-scale datasets.
weaknesses: The anomaly node proposed in this paper is a user who is overdue for repayment, and the graph structure is constructed based on the user's emergency contact. In the dataset, the relationship between the graph structure and the final detection result is not very intuitive, and more analysis or proof is needed. Also, for an anomaly detection task, users with overdue payments may not be considered anomalous nodes, but as a separate normal category, because their patterns are relatively independent of their information. 
correctness: The construction process of this dataset is sound. The article gives the credibility analysis of the original data as well as the process and experiments for handling the partial node nature or label loss of the large-scale data. 
clarity: The overall structure and delivery of the paper are well presented. The paper provides a focused, detailed, and well-developed account of the work performed.
relation to prior work: The paper provides a good explanation and analysis of its dynamics and large scale, highlighting the advantages of the paper compared to previous work.
additional feedback: The current dataset includes only the complete large-scale data. It may be helpful for related research if suitable subgraphs of smaller sizes are available.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper provides DGraph, a new large-scale graph anomaly detection dataset containing temporal information. The graph is naturally constructed in a real-world financial scenario. The authors also provide some observations and experiment results on the graph. Finally, a few insights into the graph and potential research directions on the data are pointed out by the authors.
strengths: The paper provides a large-scale real-world graph anomaly detection dataset, which is exactly what researchers and practitioners need in this problem. It can significantly benefit the graph anomaly detection and fraud detection community.

The authors apply both general graph methods and graph anomaly detection methods on DGraph, and provide interesting findings and research directions based on the experiment results.
weaknesses: In Section 4, although the authors provide some observation of the graph, the analysis is limited. The authors only describe the results but fail to explain the reason and provide some insights into the problem, e.g., why the average out-degree of normal users is higher than fraudsters'? Does it match the domain knowledge?

In experiments, arbitrarily converting DGraph into an undirected graph may affect the results. As the authors discuss in section 4, in-edges and out-edges distribute differently. Retaining the directions of edges may help improve the correctness of the experiments.
correctness: The majority of the claims in the paper are correct, but some contain factual flaws. In Table 1, the Amazon dataset actually should have 3305 background nodes instead of 0.

In line 82, the authors state that more than half of the current GAD datasets are not suited for anomalous node detection due to the network structure. Most of the datasets are released with GAD methods. Network structure is important for detecting anomalies in those datasets.
clarity: Frequent minor mistakes in the paper, but the mistakes won't significantly affect the readers' understanding of the paper.

- line 19: missing reference

- line 59, 111: missing dot

- line 101: 137.3 billion RMB

- line 147: duplicate

- fig. 3 (d): missing annotation

- line 322: generally
relation to prior work: DGraph is an unprecedented large-scale real-world financial graph. The methods used to construct the graph and label the nodes are very solid. The graph has a large portion of missing values and background nodes, which contain abundant information.
additional feedback: There are a few places I do not understand. Could you please answer the following question during the discussion?

- In line 223, how to reach 0.801 in the f1 score with a random guess?

- In Fig 3 (d), why 2-hop homophily (XBX/XTX) is compared with 1-hop's (XX)?
review_title: A very good dataset, but the paper needs some improvements

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: In this paper, the authors propose a large dynamic graph dataset, named DGraph for anomaly detection. Furthermore, they provide a comprehensive observation of DGraph revealing the characteristics of anomaly nodes. Finally, they conduct experiments of GAD (Graph Anomaly Detection) baselines to show the effectiveness of DGraph on this task.
strengths: - The process of constructing the dataset is elaborated detailedly. The authors provide detailed analysis of anomaly nodes based on the statistics of DGraph, which helps the community profile anomaly nodes in GAD tasks.
- The authors provide amenable website and leaderboard for DGraph, which makes it easy for researchers to use in the future research projects. (All results in this paper could be updated to the leaderboard.)
weaknesses: - The experiment of GAD baselines is insufficient. More GAD baselines could be added to the experiment. I list some of them as follows. (More related baselines could be found at https://github.com/pygod-team/pygod)

Ding K, Li J, Bhanushali R, et al. Deep anomaly detection on attributed networks[C]//Proceedings of the 2019 SIAM International Conference on Data Mining. Society for Industrial and Applied Mathematics, 2019: 594-602.

Li J, Dani H, Hu X, et al. Radar: Residual Analysis for Anomaly Detection in Attributed Networks[C]//IJCAI. 2017: 2152-2158.

Cai L, Chen Z, Luo C, et al. Structural temporal graph neural networks for anomaly detection in dynamic graphs[C]//Proceedings of the 30th ACM international conference on Information & Knowledge Management. 2021: 3747-3756. (Specifically for dynamic graphs)

- In my opinion, the motivation of this paper is not clear enough. The only strength of DGraph compared with other GAD datasets seems to be its scale. In section 4 and 5, some comparison of DGraph and other datasets which demonstrates the superiority of DGraph could be added. Moreover, the superiority of the temporal information in DGraph could be further discussed.

- Some typos in the paper should be fixed. (1) The citation is missing at line 19. (2) The punctuation is missing at line 59 and line 111. The information source is not clear at footnote 1.
correctness: The dataset is constructed in a sound way.
clarity: Overall, this paper is well organized. In each paragraph, the writing could be improved to make logic clearer.
relation to prior work: In my opinion, the improvement of this work from previous works is not fully discussed.
additional feedback: Suggestions are provided in the Weaknesses section.
review_title: Lack of important baselines

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors propose DGraph, a real-world and large-scale dynamic graph from financial scenarios. The authors provide a comprehensive observation of DGraph, which comprehensively explores the novel and promising properties of DGraph. The authors conduct extensive experiments on DGraph. The results demonstrate that DGraph offers opportunities to advance previous GAD methods.

 
strengths: The authors present DGraph, a real-world dynamic graph in finance domain, with the aim of enriching the variety of GAD datasets and overcoming the limitations of current datasets. In the construction of DGraph, the authors preserve missing values on node features, and those unlabeled nodes which are referred to as background nodes. The authors make a comprehensive observation on DGraph. It reveals that anomalous nodes and normal nodes generally have differences on various graph-related characteristic. Meanwhile, the importance of missing values and background nodes is covered by observation. Furthermore, the authors conduct abundant experiments on DGraph, and gain many thought-provoking discoveries. Compared with general GNNs, most current GAD methods present worse performance. It indicates that these GAD methods may overfit on several datasets. Meanwhile, results show that handling missing values and processing background nodes is indeed crucial in DGraph. It is expected that these discoveries can be extended to more general fields. In general, DGraph overcomes the limitations of current GAD datasets and enriches their varieties. DGraph could act as an essential resource for a broad range of GAD research.
weaknesses: The paper is not very well written and needs some further careful elaboration. The proposed graph is very sparse in terms of connections compared with the general graph data. The authors are suggested to explain the sparsity and discuss whether there is edge missing in the data collection.
correctness: The claims made in the submission are correct.
clarity: The paper is not very well written.
relation to prior work: How this work differs from previous contributions is discussed.
additional feedback: Please refer to strengths and weakness.
review_title: Review of DGraph: A Large-Scale Financial Dataset for Graph Anomaly Detection

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper introduces the Dgraph dataset, which is a dynamic graph for node anomaly detection. It is a large-scale graph with dynamic edges. The dataset comes from the fintech domain with a clear definition of nodes, edges, node features, and anomalies. Dgraph has the characteristics of missing value and background nodes. These characteristics are proved to be useful when conducting anomaly detection.  Experiments are conducted to show that some current GAD algorithms may overfit several datasets and this makes Dgraph promote the GAD area very possibly.
strengths: 1. There is a clear description of how the graph was constructed from real-world data. The nodes, edges, node features, as well as anomalies, are defined clearly meaningful in the real world. One concern when I was reading this paper is that if emergency contacts are reasonable to be edges and construct the graph topology. The concern was solved by the analysis in Figure 2. It shows that normal and abnormal users tend to have different topology structures.
2. It provides clear ablation studies about the missing value and the background nodes. It shows that both of them are necessary designs in GAD and tackling them appropriately improves the performance.
3. Because the data are real-world financial data, the privacy issue is emphasized in the paper.
weaknesses: When talking about the dynamic graph, Dgraph only uses dynamic edges. Are dynamic nodes also needed in the data? If we have dynamic nodes and their corresponding edges for each timestamp, we have a complete dynamic graph.
correctness:  The dataset is constructed in a sound way.
clarity: The paper is well-written and clear. Except for some punctuation marks in lines 19,59, and 111.
relation to prior work: In Table1, DGraph is compared with previous works in size, statistics, and some properties, showing that the authors propose a large-scale GAD dataset with missing values and background nodes that others do not have. These properties are proved to be helpful for GAD in the paper.
additional feedback: Thank you for the nice dataset!
review_title: This is a good paper about a graph anomaly detection dataset. It proposes a novel GAD dataset with some characteristics. It discusses the necessity and novelty of them.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This submission introduces a large-scale dynamic graph dataset from a financial company composed of users as nodes and their social connections as edges. Partial of the users are annotated based on whether they repay the loan in time or not and the overdue users are regarded as anomalies on the graphs. The authors highlight the two uniquenesses of the presented dataset: 1) many nodes are background nodes without labels; 2) many nodes contain missing values in their features. A preliminary analysis of the dataset demonstrates the anomaly nodes differ from the normal nodes from the graph perspective. Experiments using various graph anomaly detectors show the shortcomings of existing methods and also point out the importance of incorporating background nodes and handling missing values in node features. There is a leaderboard and a data competition associated with the dataset which could facilitate future research and practice. Overall, I think the proposed dataset fulfills the urgent need for high-quality datasets in graph-based fraud detection research.

strengths: 1. The dataset is the first large-scale graph dataset in graph-based anomaly detection and fraud detection which could help advance this promising research topic. Moreover, the temporal information, missing feature values, and background nodes in this dataset could motivate fruitful and interesting research works in graph machine learning.

2. The preliminary analysis of the dataset is informative and it shares some useful insights about the proposed dataset.

3. The leaderboard and website are well-designed and easy to access.
weaknesses: 1. The writing is rough with multiple grammar errors and typos.

2. The taxonomies used in the paper need to be carefully discussed and defined.

3. I have multiple questions regarding the data collection and the graph construction, please see my detailed comments in the Additional Feedback section.

4. The experiment misses the unsupervised graph anomaly detection method which is a major type of GAD method.
correctness: The reason behind choosing node features and connecting users with the emergency contact relationship is not well-explained.
clarity: The analysis and experiment are generally well-presented but the introduction and dataset curation miss some details and some terms are not used correctly. Please see my detailed comments in the Additional Feedback section.
relation to prior work: The authors give a comprehensive review of existing graph datasets with real-world anomalies. It is better to mention that most previous works only use synthetic graph anomaly datasets to highlight the significant contribution of the proposed real-world graph.
additional feedback: 1. Grammar errors and typos.
- Line 19, empty reference.
- Line 24, various scenarios. 
- Line 59 misses one period after *research*.
- Line 111 misses one period after *privacy*.
- Lines 146-147, repeated sentences.
- Line 173, with the extremely imbalanced classes.
- Line 184, its' -> its.
- Line 189, neighbor features.
- Line 220, training-set.
- experimental result -> experiment result.
- Line 302, conditions -> settings.
- Line 303, testset -> test set.
- Line 322, DGraph can be used to explore a general problem: How to process background nodes in any graphs?
- Line 322 misses one period in the end.



2. Writing suggestions.
- *telecom fraud* can be replaced by *wire fraud* which is more commonly used.
- Only one telecom fraud example cannot support the claim in Line 24 which mentions **various of scenarios**.
- Figure 3, explaining the BN in the caption.
- Line 232, introducing BN before first using it the body text.
- Line 326, the background nodes in the proposed dataset have no borrowing behavior which can be regarded as a type of label and it reflects the neutrality of the user. Therefore, I think the background nodes cannot be called as **unlabeled nodes**.



3. Questions.
- The definition of **fraud** in this dataset is not accurate. A user with only one loan overdue behavior indicates the user's low credibility and high risk. Whether the user is a fraudster (i.e., intentionally missing the payment due) needs more evidence. Thus, I think it is incorrect to describe the anomaly nodes in the dataset as fraudsters. I would like to know the authors' input on this point.

- I am confused about the motivation of each step in graph construction. Why can the demographic information be used to predict one's defaulting behavior? Why do the authors use the **emergency contact** as the relation to build the graph and how does it help identify loan defaulters?

- The proposed dataset is more like a semi-supervised node classification dataset instead of the graph anomaly dataset. Generally, GAD is an unsupervised task but the author did not review and compare the unsupervised GAD methods. If no unsupervised method can identify the anomalies to some extent (e.g., AUC > 60%), it suggests that the node class has a strong correlation with node feature instead of graph semantics. Then I would argue the dataset should be described as a low-credit user detection or node classification dataset in the financial domain. I would like to see the authors' insights on this issue.

- The authors mentioned in Line 105 that emergency contact information is a mandatory requirement for all users which means each user has at least one out-edge. However, Figure 2(a) shows the average out-degree for fraudsters is 0.75, indicating that some fraudsters have no emergency contact. Could authors explain this problem?

- In Line 199, what is the meaning of **cycle**?
- Since the graph is very large, I wonder how the different models are trained on a single GPU?
- In Lines 305-307, the reduced time is inconsistent with the time reported in Figure 4(b). Why is the training time per epoch below 0.3 seconds given such a large graph?
review_title: A useful dataset but the presentation is rough.

Meta-review: metareview: This paper presents a large-scale and real-world dynamic graph in the finance domain, which is applied for anomaly detection. In addition, nine supervised and seven unsupervised methods have been evaluated on DGraph. This paper may push forward the research in the anomaly detection community. However, the presentation of this paper should be further improved. The current version contains many typos and grammar errors (even in the abstract). Moreover, the authors should provide an in-depth analysis of the experimental results and dig deep into the reason behind the empirical observation. In summary, I vote for acceptance of this paper, although would not be upset if it were rejected (i.e., weak acceptance).
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept