Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper the problem of graph level anomaly detection is studied, for imbalanced data, and a dual discriminative graph neural network is proposed, called iGAD. The proposed method focus on two main aspects of anomaly detection in graphs; (i) anomalous nodes attributes, and (ii) anomalous graph structures. For this reason, the method uses (i) anomalous attribute-aware graph convolution and (ii) anomalous substructure-aware deep random walk graph kernels. To deal with the imbalance that appears usually in anomaly detection scenarios, a point wise mutual information (PMI) loss function is used between the input graphs and their anomalous/normal attributes. A series of experiments have been conducted to study the performance of iGAD against other state of the art comparison methods in four real world chemical compound datasets and investigate a couple of research questions.
strengths_and_weaknesses: Strengths:
1) The studied problem and the proposed method are very interesting to the research community. This work combines two main features in anomalous graphs with a novel loss function. The paper is well-written, well-structured and easy to follow.

2) The methodology is sound and the proposed components could be used in different use cases. 

3) The authors did a good job in the related work section, organizing and covering some of the related literature to give the reader more motivation. It can be improved if the authors mention the contribution of their method after every limitation shoed in the related works.

4) Experimental setup is very thorough and to the detail. The authors provide six research questions, and they show a detailed comparison with five comparison methods, five variants of the comparison methods and three variations of their method to prove the effectiveness of all the components combined.

Weaknesses:
1) The novelty of this work is limited, as most of the proposed methodology is either already published, or with small modifications from published methodology.

Minor comments:
1) In Figure 1, the step III is confusing, as it appears above step II. In step II, the light blue nodes (that show the random walks) is not clear to which paths they refer to from the graph G_S. I would suggest adding the node id to help the reader follow the thought.

2) In Table 2, in MCF-7 dataset, the best result is not highlighted (iGAD-2).

3) It would help if the authors repeat what M and n’ refer to in the caption of Figure 3.
questions: - In Table 2: Why the authors do not give also the results for the Precision metric?

- In Table 2: Are all these results tested for statistical significance? It is important to mention the statistical test and score for these results, as the 20% of the datasets that is being tested, is a small dataset.

- Did the authors try bigger size of random walks (> 8 nodes)? To better understand this experiment, it would be useful to add in the Table 1 with the dataset statistics: the number of connected components in each graph and the avg. min. max. size of the connected components in each graph.
limitations: The authors did not address the potential negative societal impact of the work, however there is no potential negative impact.
The authors did address some of the limitations of the work and they leave them for future work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper focuses on a problem of graph-level anomaly detection that distinguish anomalous graphs from a set of graphs. For one graph, it could be considered as anomaly due to the anomalous attributes of certain nodes and the anomalous substructures. It is argued in the paper that generally GNNs will aggregate the information of neighbors resulting the relatively poor representation for anomaly graph detection. Thus, the authors investigate an anomalous attribute-aware graph convolution and anomalous substructure-ware deep RWK to detect the anomalies. Specifically, in anomalous attribute-aware graph convolution, the node representation and the representation of its neighbor will be separately encoded instead of merging together as GCN. To detect the anomalous substructure, the paper proposes a way to compare the random walk paths of graphs and the anomalous substructure. The anomalous substructure's adjacency matrix is learned in an end-to-end manner.  A PMI-based loss function is used to balance the contributions of normal graphs and anomalous graphs. Experiments are conducted several large-scale real-world to evaluate the effectiveness of the proposed method.
strengths_and_weaknesses: **Strengths:**
1. The graph-level anomaly detection is an important problem to be solved. And the existing work generally focus on node-level anomaly detection in graph. The proposed iGAD can give some insights for the community.
2. The challenges of  graph-level anomaly detection are discussed. And the proposed mechanisms are convincing in detecting the anomalous attributes and anomalous sub-structures.
3. The paper is well organized and easy to follow. 
4. Based on the current results, the iGAD seems promising in detecting anomalous graphs on the real-world datasets.

**Weaknesses***
1. Though the iGAD focuses on a novel problem of graph-level anomaly detection, there are concerns of technical contributions of the proposed method. 
    * Firstly, as node-level anomaly detection also faces the anomalous nodes and anomalous graphs, many methods in node anomaly detection share similar high-level idea with iGAD. For example, the anomaly scores based on the aspects of attribute matrix and adjacency matrix are separately considered in [1,2] for node anomaly detection. Thus, it is suggested to have more discussions about how the iGAD can address the unique challenges of graph-level anomaly detection. 
    * Secondly, it is argued that GNNs will aggregate the information of neighbors resulting the relatively poor representation for anomaly graph detection. To address this challenge, the graph convolution in iGAD separates the encoding on a node itself from its neighborhood when representing nodes. This design also have been widely known in GraphSage and H2GCN [3] to avoid mixing dissimilar nodes in H2GCN.

2. Some important references and potential baselines are not covered. There are already works [4,5] noticed that the GNNs may not work well for anomaly detection because of the aggregation mechanism. They have proposed some special aggregation mechanism for anomaly detection. Though they are originally designed for node-level anomalies, it would be easy to extend for graph-level anomaly by adding readout function. I would suggest to compare with these state-of-the-art anomaly detection methods.

References
[1] Bandyopadhyay, Sambaran, Saley Vishal Vivek, and M. N. Murty. "Outlier resistant unsupervised deep architectures for attributed network embedding." Proceedings of the 13th international conference on web search and data mining. 2020.
[2]  Kumagai, Atsutoshi, Tomoharu Iwata, and Yasuhiro Fujiwara. "Semi-supervised anomaly detection on attributed graphs." 2021 International Joint Conference on Neural Networks (IJCNN). IEEE, 2021.
[3] Zhu, Jiong, et al. "Beyond homophily in graph neural networks: Current limitations and effective designs." Advances in Neural Information Processing Systems 33 (2020): 7793-7804.
[4] Kumagai, Atsutoshi, Tomoharu Iwata, and Yasuhiro Fujiwara. "Semi-supervised anomaly detection on attributed graphs." 2021 International Joint Conference on Neural Networks (IJCNN). IEEE, 2021.
[5] Dou, Yingtong, et al. "Enhancing graph neural network-based fraud detectors against camouflaged fraudsters." Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2020.
questions: 1. It is highly suggested to clarify  the technical contribution of the proposed iGAD and have more discussions about the differences of iGAD with existing anomaly detection methods on graphs.
2. It would be more convincing if experiments in comparing the potential baselines mentioned above in graph-level node classification can be added.
limitations: Please refer to the weaknesses for details.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes iGAD, a supervised anomaly detection model for distinguishing anomalous graphs from normal graphs. This paper identifies anomalous attributes and/or anomalous substructures as the anomaly notions of graphs, and also notices the imbalance nature of the anomaly problem. The authors employed three designs to achieve the graph-level anomaly detection model iGAD: a. Let the attributes of each node evolve individually in the graph convolution process to emphasize the anomalous attributes; b. Considering the defect of message passing-based GNNs in learning structural information, iGAD employs a deep random walk kernel that utilizes discriminative substructure learning and structure comparison to explore anomalous substructures; c. A new loss function was proposed to enable the proposed model to fit the minority class samples (i.e., anomalous graphs) well. Experimentally, iGAD shows strong performances in classifying anomalous and normal graphs on four graph datasets.
strengths_and_weaknesses: Originality: This paper proposes an interesting method to detect anomalous graphs in a graph dataset. It is a topic neglected by the anomaly detection and graph mining communities. The method comprehensively considers the anomalous notions in attributes and substructures of graphs, and the imbalance nature of the graph dataset with anomaly graph samples. 
Quality: The proposed method is technically sound, and the authors have basically made experimental verification of the proposed designs including the message passing graph convolution, deep RWK, and the PMI-based loss function. 
Clarity: This paper has good organization and writing. The author gives detailed preliminaries corresponding to the method to help readers understand the proposed algorithm. In terms of experiments, the author gives the details of parameter settings for readers to reproduce the experiments.
Significance: Few deep graph learning practitioners consider how to detect anomaly graphs. Most graph anomaly detection work focuses on detecting anomaly nodes in a single graph. This paper extends the graph anomaly detection to the whole graph level. Although the anomalous information/notions of a graph may reflect in other aspects in addition to its node attributes and graph structure (authors also admitted this viewpoint in the discussion about the limitations of this work). The learned discriminative substructures provide explainable information for anomalous graph detection.  In addition, the PMI-based loss proposed in this paper has a good generalization, it can be extended to other imbalanced classification problems easily. Overall, this paper can be a good starting point for graph-level anomaly detection topic.
questions: Some questions to help authors further improve their research:
1. The authors put forward a simple and general solution for learning the imbalanced data distribution. But different downstream application tasks have different tolerance for the false-positive rate and the false-negative rate. Considering it may help improve the applicability of iGAD.
2. The datasets used for model verification are all molecular graphs. Why did not the author use the graph dataset from other application domains and have multi labels and specify one of the labels as anomalous and others as normal to conduct experiments?
3. I suggest authors give more detailed explanations on why the cross-entropy loss cannot work on imbalanced data distribution in the main content or the appendix.
limitations: The authors clearly call out the limitation of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper  is applied in the imbalance problem of graph-level anomaly detection, they propose a dual-discriminative graph neural network for graph-level anomaly detection. The first module is a novel anomalous attribute-aware graph convolution, which separates self representation and neighborhood-representation when representing a node in a graph. The second module is  an anomalous graph
substructure-aware deep Random Walk Kernel to  encode anomalous structural information into graph representations.  Further,  they propose a Point Mutual Information-based loss function to address the imbalance nature of anomaly problem. 
strengths_and_weaknesses: Strengths:

1. Experiment are done on extensive real-world graph datasets and  evaluations are sufficient, moreover achieve great performance improvement.

2. The idea is relatively novel， which couples  attribute-aware and  substructure-aware knowledge for graph-level AD problem.

Weaknesses:

1. The paper is not organized well, especially in the section of  Introduction and Related Work.

2. Figure 1 is not intuitive enough to understand the intention of the authors.
questions: 1. I think anomaly detection task  is originally a problem of label imbalance, and the author should not repeatedly emphasize this.

2. This paper has limited application value. Due to there are very few anomaly detection labels, although supervised learning can also solve this problem, it is difficult to obtain labels of anomaly patterns in practical applications. The author should consider methods that do not rely on label information as much as possible, such as semi-supervised and unsupervised learning, e.g.

3. The authors should compare with the most advanced methods for graph-level AD task, instead of GCN, GIN, SOPool, e.g.

For example: 

1): Ma R, Pang G, Chen L, et al. Deep Graph-level Anomaly Detection by Glocal Knowledge Distillation[C]//Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. 2022: 704-714.

2): Qiu C, Kloft M, Mandt S, et al. Raising the Bar in Graph-level Anomaly Detection[J]. arXiv preprint arXiv:2205.13845, 2022.

limitations: Please see Questions part
review_title: null

Meta-review: metareview: This paper proposes a dual-discriminative graph neural network for detecting anomalous graphs given a set of graphs. For imbalanced data, a point wise mutual information (PMI) loss is used. The extensive experimental results using real-world datasets demonstrate the effectiveness of the proposed method. The graph-level anomaly detection is an important problem, and the motivation and its challenges are well presented. This paper is well-written. The proposed method of considering anomalous node attributes and anomalous graph structures is interesting. 
confidence: Certain
Decision: Accept