Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present a new problem, how to detect real graphs and "fake" graphs. The authors then explore four scenarios where the detection algorithm applies, and propose three different machine-learning methods to detect fake graphs.
strength_and_weaknesses: Strengths,

S1: Authors explore four scenarios, which provide both ideal circumstances and realistic circumstances.
S2: This paper proposes a new problem.

Weaknesses,

W1: The motivation is not strong and seems artificial. It is not clear how the generated graphs are used in malicious misuses or misinformation broadcasts.

W2: The novelty is limited. The proposed methods are based on existing works.
clarity,_quality,_novelty_and_reproducibility: This paper is well-written and well-organized. It is easy to follow the proposed ideas and technological details. 
The novelty of this paper is limited. 
summary_of_the_review: I recommend a weak reject for this paper because of its limited novelty and the weak motivation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: DISCLAIMER: I reviewed this paper for NeurIPS 2022 earlier this year. As far as I can tell, not much (if anything) has changed in this manuscript. As such, I will reiterate the points in my original review.

Summary:

The authors investigate the performance of existing graph neural network models to discern between real graphs and "fake" graphs generated synthetically, potentially for a nefarious purpose. The authors explore 4 scenarios of increasing complexity and real-world relevance, as well as 3 different approaches from the machine learning literature to learn to discern between real and fake graphs.
strength_and_weaknesses: Strengths:
- Experimental results provide some practical insights on the performance of different ML approaches for the target task
- Authors explore 4 scenarios of increasing complexity, which provides a valuable comparison on how the models perform under ideal circumstances vs. more realistic circumstances.

Weaknesses:
- Unclear significance and relevance of the addressed problem. The authors' argument is that they are preemptively studying a problem that may become relevant in the future. It is challenging to assess significance at this moment. It is also challenging to assess whether the performance obtained by the models would be good or not in practice for this hypothetical task. Having a case study would help justify the authors' claims of significance.
- No novelty. The evaluation and data generation is performed using existing methods.
clarity,_quality,_novelty_and_reproducibility: Originality: The proposed methods are not novel. The main novelty in the paper is proposing the problem of discerning between real graphs and fake graphs. I argue this is not a big intellectual/technical leap from the analogous problem in image/video data.

Clarity: The paper is very clear. The notation, proposed problem, datasets, and evaluation are all clear to me.

Quality: I believe the evaluation is of above average scientific quality. The technical quality is limited, as none of the methods are novel.
summary_of_the_review: See my comments above. My main concern with this paper is the lack of novelty. If the main contribution of the paper is a new ML task, I do not find compelling evidence in the paper to suggest that this task will have an application in the future. It may very well be the case that this task will become relevant for a certain domain at some point. But we simply do not know, and we can just "extrapolate" from other types of data, such as images.

I am curious as to how the authors arrived to the generated graph detection problem? Was there a particular application in mind where fake graphs are indeed a concern? If so, please discuss in the paper, as it would *significantly* strengthen the contribution. Or was the thought more along the lines of "if the task of detecting fake images is of interest to the community, why not detecting fake graphs instead"?

If the idea of a new task is indeed the crux of the contribution, I would recommend this paper to be accepted as a short letter, an extended abstract, or a poster, not as a full research paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper discusses the problem of detecting graphs that have been generated algorithmically. The paper discusses four different types of scenarios from the simplest (where the model knows all the possible details about the generation process) to the hardest (where nothing is known in advance).

The classifier used is an end-to-end trained GNN encoder with an MLP used to output the classification outcome.

The classifier has a decent performance, although not super high making the task challenging.
strength_and_weaknesses: # Strengths

1. Novel problem
2. Method is simple enough to be easy to implement


# Weaknesses

1. The method itself could be used to detect generated graphs that cannot be detected easily and thus being used in an attack
2. The classifiers that  authors tested on their datasets are pretty much "standard". I wonder wether there might be some better architecture that can be used to take specific care of this task
clarity,_quality,_novelty_and_reproducibility: The paper is clear and novel. It is also straightforward to replicate the experiments.
summary_of_the_review: Some questions to authors:

1. When you write: "... use Euclidean Distance to measure the 1-nearest-neighbor similarity between each generated graphs and real graph sets..." ==> how can you make sure that euclidean in the embedding space is the right distance? Have a look at these two papers (https://www.sciencedirect.com/science/article/abs/pii/S002002551630696X ; https://arxiv.org/abs/1812.09095) is there anything that you could use to better compute the 1-nearest neightbor?

2. If you have a good way to detect generated graphs, wouldn't this be useful to filter out generated graphs that are difficult to spot? How would you prevent this from happening?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The work considers the task of learning to detect generated graphs (fake graphs). More specifically, the work investigates the performance of existing graph neural network models to discern between real graphs and generated graphs. The authors explore four scenarios of increasing complexity and real-world relevance, as well as three different approaches from the machine learning literature to learn to discern between real and fake graphs. 

The overall impression of the reviewers:
- This is an interesting task
- But an interesting task that needs a clear case study. 
- Adversary should adapt to defense
- No novelty in the techniques (the idea of putting these methods together is novel, but reviewers also wanted to see some technical innovation)
justification_for_why_not_higher_score: This is an interesting idea that needs a more compelling case study and more technical depth.
Decision: Reject