Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a self-supervised learning based approach to generate representations of real and synthetic graphs  in order to evaluate the quality of latter.  The general idea is to train graph encoders on the same data used to train a graph generator, using self-contrastive learning and then use these trained encoders to embed generated graphs and compare the embeddings with the same of real graphs in a test set. The claim is that such contrastively trained GNNs are more capable of extracting meaningful features from the graphs, compared to randomly initialized GNNs. The choice of perturbation schemes used in contrastive learning to train graph encoders could be domain specific. To demonstrate the merit of the idea, the paper chooses Graph Isomorphism Network (GIN) as the encoder architecture and ideas from two well-known graph self-supervised learning works (GraphCL and InfoGraph) to pretrain the encoders. The encoders are pre-trained on six datasets (three synthetic and three real-world) and then used to encode the real graphs as well as synthetic graphs generated from the four traditional graph-perturbation algorithms: Mixing-Random, Rewiring Edges, Mode Collapse, and Mode Dropping. To evaluate the goodness of proposed encoders, authors monitor the correlation (higher the better) between the degree of graph-perturbations and eight discrepancy metrics (commonly used to compare similarity between two graph distributions) between the distributions of latent representations for real graphs and generated graphs.  The proposed approach is compared against state-of-the-art technique where GIN is initialized randomly and the results indicate better correlation values for proposed approach as opposed  to SOTA approach. Additionally, authors also show that graph encoders supplemented with features capturing graph substructural information further aids to the distinguishing power of graph encoders. 
strengths_and_weaknesses: Strengths: 
1. The paper highlights the limitations of using generic local metrics to evaluate the two distributions. 
2. The idea of using graph-substructural features in training graph encoders along with Lipschitz control is quite plausible and also shown to work well. 

Weaknesses: 
1. The technical contribution of the paper is somewhat limited. The performance of the proposed evaluator is largely dependent on the choice of perturbation scheme used in contrastive learning as well as the architecture of the encoder. The right choices for each of the two components could largely vary with application domains and thus it may not be obvious in every application. And in case it is obvious, wouldn't that pretty much seal the choice of graph generator? 

2. It might be difficult to use the proposed approach to evaluate two graph generators G1 and G2 that use different perturbation schemes P1 and P2 respectively. In such a case, what perturbation scheme shall be used to train the evaluation encoder?

3. For certain discrepancy metrics, the SOTA evaluator seem to be performing better than the proposed evaluator (see comments below), but it is not explained well as to why this might be happening.



questions: 1. Just to confirm, is there a particular reason to choose the contrastive learning schemes from InfoGraph and GraphCL in this paper?

2. All the results in Figures 3-6 correspond to which of the six datasets? Or are the results combined across all the datasets?

3. In several plots, I observe that rank correlations seem to be better for GIN-Random than the proposed approach (e.g. Figure 4(b), F1DC, F1PR, Coverage metrics). Am I reading the graph correctly? And if so, any explanation for why that might be the case?


limitations: I will recommend authors to talk about the limitations in the context of weaknesses I highlighted above if they agree with it.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work mainly argue that we should use contrastively learned representations to evaluate the performance of graph generation methods. The key idea is to use the same training set to train the generative model and the encoder. The encoder is trained by contrastive self-supervised learning and will be used to evaluate the performance of the generative model. Experiments on multiple datasets in terms of multiple metrics are conducted to support the claim.
strengths_and_weaknesses: #### Pros ####
(1) The intuition to use contrastively learned representations to evaluate the performance of graph generation methods is sound to me. Basically, contrastive learning teaches model which graphs are similar, thus being consistent with what we expect for generative models. In other words, we also hope that our generative models could generate graphs similar to the training graphs.

(2) The experiment design is reasonable and can support the main claim clearly.

(3) The presentation of this paper is great and easy to follow.

#### Cons ####
(1) The work claims that we should use contrastively learned representations to evaluate graph generative models. However, it does not consider evaluating any generative models in the experiments. It mainly evaluates the correlation between perturbation ratio and the defined value of measurements. Although it can support the claim to a large degree, it could be more convincing if any generative models can be evaluated and analyzed using the proposed strategy.
questions: (1) Does it take the same resource to train the GIN-random and GraphCL/InfoGraph? 

(2) For the encoder, no matter it is GIN-random or GraphCL/InfoGraph, how can we determine if it is trained well? Do we have a validation set for the encoder?
limitations: Not applicable.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a new method to evaluate graph generative models by evaluating the representations of the graph which are learned from a contrastive learning approach. They use the training data (which the generative model would also use to train on) and contrastive learning to learn a graph encoder, which they then use to generate representations of the test set and the generated graphs (or in the case of the this paper, of perturbed graphs, which they use to show the effectiveness of the method for evaluation). They show that neither GNNs nor local metrics are strictly better than the other for evaluating graph generative models, since each has a failure case that the other approach can distinguish, and use this as the motivation for trying to combine the respective strengths by using the Graph Substructure Network (which incorporates some local metrics) as their GNN and then train the encoder using contrastive learning and some predefined augmentations. They then assess their proposed method via various graph perturbations, and evaluate the spearman rank correlation between the perturbation and various metrics of interest.
strengths_and_weaknesses: Strengths:
1. The paper is very well written, clear, and easy to follow.
2. The topic is timely and seeks to solve an important problem in the graph representation learning community, namely how to evaluate graph generative models
3. Very nice motivation for why existing approaches (local metrics, random GNNs) are insufficient
4. Using contrastive learning to learn meaningful representations seems like a logical and promising approach to address this issue

Weaknesses:
	
1. I found it a bit difficult to assess the performance of the approach from the current paper. The visualizations are perhaps too overloaded, which obscured the takeaways (and there are no tables in the main paper).  I think the point is that GraphCL is better than GIN-Random, but the current visualizations don’t give me that immediate impression, they look quite similar. (When plots have so many variables to compare upon, I think it only works to easily digest if there is a clear, dramatic pattern - more subtle differences, even if consistently better, can get lost)  I think the authors should consider alternative visualizations/tables to convey their results more clearly; at a minimum, they could separate out GraphCL and GIN-Random into small multiples of the same plot (but there are probably better solutions). 
2. I think the experiments section would also benefit from a bit of restructuring. A lot of space is given to showing the results of not using structural features, using node degree features, and then using node clustering coefficients. I would naturally expect performance to improve as one keeps adding information, so I think there is too much space devoted to this conclusion at the expense of other data/visualization/textual explanations. I think the results you’re trying to convey are 1) graphCL outperforms random-GIN (in whatever comparison format makes most logical sense - e.g. with/without node degree features), and 2) GraphCL with node degree and clustering coefficients (i.e. local measures) further improve performance. I think the experiments section would be stronger if it was organized a bit more around these conclusions and providing the necessary data/plots to make that (or any other conclusions you’d like to highlight) very clear. If there are other conclusions from your experiments that I missed please feel free to include those as well.
3. In general, the plots would benefit from a bit more explaining. For instance, I believe each violin plot contains 6 datasets x 10 random seeds, but it took me a few times searching in the paper to confirm that for myself. I’d make this very explicit, and also make the captions more self-contained.
4. The authors highlight the limitations of using local metrics and using random GNNs, but only compare to using random GNNs. I think the local metrics should be added in as a comparison to confirm that the performance is also better than local metrics.
5. It seems to me that a key benefit of this approach is what the authors stated about graphs being so heterogenous and therefore static approaches (e.g. degree histograms) may appropriately capture the relevant aspects of the data. Contrastive learning has the potential to address this issues since it can use augmentations that are relevant to the dataset, but as the authors themselves note, it is not clear how to choose such subgraphs, and they relegate this choice to future work (where domain specific choice may apply). The choice of augmentations strikes me as a key question in determining how to train the encoder to learn a meaningful representation. I understand this is a hard question to answer, and would be okay accepting this to future work, assuming that the experimental results were strong given the current generic augmentations. But with the current results I am not yet convinced; but am open to being persuaded if the results/visualizations can be more effectively conveyed.
5. (Minor) The graph contrastive learning section in related work reads like a list - I would try to weave this into a more cohesive picture (as is done very nicely in the graph generative model related work section)
6. (Minor) Is Nei(v) neighborhood in Eq. 1? Please define it somewhere.

On the whole, I enjoyed reading this paper, and I think it can be a very compelling paper once the experiments are strengthened and clarified a bit.

questions: My questions/suggestions are described in the section above; I look forward to a discussion with the authors.
limitations: While there is no dedicated limitations section, the authors were upfront about the current limitations (no clear guidelines on how to choose the augmentations at present).
review_title: null

Meta-review: metareview: This paper proposed a new way of evaluating the quality of graph generative models, by leveraging the contrastively learned representations. The reviewers generally found the study is interesting and important, and the idea of leveraging contrastive learning for evaluation is also sound and novel. I also believe the work would be impactful for future research in the graph generative model domain, and I highly encourage the authors to properly release easy to use benchmark tools/datasets around it, if possible.

confidence: Certain
Decision: Accept