Evaluating Graph Generative Models with Contrastively Learned Features
Authors: Hamed Shirzad
Kaveh Hassani
Danica J. Sutherland
Venue: NeurIPS 2022 Accept (2022)
Keywords: generative model evaluation
graph generative models
self-supervised learning

Abstract: A wide range of models have been proposed for Graph Generative Models, necessitating effective methods to evaluate their quality. So far, most techniques use either traditional metrics based on subgraph counting, or the representations of randomly initialized Graph Neural Networks (GNNs). We propose using representations from constrastively trained GNNs, rather than random GNNs, and show this gives more reliable evaluation metrics. Neither traditional approaches nor GNN-based approaches dominate the other, however: we give examples of graphs that each approach is unable to distinguish. We demonstrate that Graph Substructure Networks (GSNs), which in a way combine both approaches, are better at distinguishing the distances between graph datasets.

Evaluating Graph Generative Models with Contrastively Learned Features
Hamed Shirzad
Kaveh Hassani Autodesk AI Lab kaveh. hassani @autodesk. com
Danica J. Sutherland UBC & Amii dsuth0cs. ubc.
UBC
shirzadecs. ubc.ca
Abstract
A wide range of models have been proposed for Graph Generative Models, cessitating effective methods to evaluate their quality. So far, most techniques use either traditional metries based on subgraph counting, or the representations of randomly initialized Graph Neural Networks (GNNs). We propose using representations from contrastively trained GNNs, rather than random GNNs, and show this gives more reliable evaluation metrics. Neither traditional ap- proaches nor GNN-based approaches dominate the other, however: we give ex- amples of graphs that each approach is unable to distinguish. We demonstrate that Graph Substructure Networks (GSNs), which in a way combine both ap- proaches, are better at distinguishing the distances between graph datasets. The code used for this project is available in: /tttp:///ithhu http://hhuu..mm/ammedd..... com/hamed1375/ Self-SSperviied- fooddessffr--oo EvaM--valtttion
Introduction
Quantitative evaluation of generative models is challenging 45 evaluating purely by visual inspec-
tion can introduce biases, particularly towards "precision" at cost of "recall" (37). The traditional metric, likelihood, is also not only difficult evaluate for implicit generative models complex, highly structured datasets, but can also be a poor fit to users' goals with generative modeling [37] 45). Many quantitative proxy measures for the discrepancy between generator and real distributions have thus been used recent work on generative modeling of images, some of the most important including the Inception Score (IS) [38). Fréchet Inception Distance (FID) (20). Precision/Recall (PR) (37]. and Density/Coverage (32). FID and PR measures in particular have been shown to correlate with human judgments in some practical settings. All of these measures use representations extracted from CNNS pretrained on ImageNet classification (36)- Nevertheless, it is recently shown that self-supervised representations archives while the ranking with prrerant- embeddings often can be misleading (30).
reasonable ranking in terms of FID/Precision/Recall,
These problems are exacerbated for the evaluation of generative models of graph data. Graphs often used to represent concepts in a broad range of specialized domains, including various fields of engineering, bioinformatics, and so "generally good" features like those available for natural images via ImageNet models. Many methods for evaluating graph generative models are thus based measuring discrepancies between statisties of generic low-level graph features, including local measurements like degree distributions, clustering coefficient distributions, and four-node orbit counts [4 (2).. simple global properties such as spectra (55). Differences between distributions of these features are generally measured via the maximum mean discrepancy (MMD) 11 total variation (T'V distance.
that it is more difficult perhaps impossible to find
More recent work (46) proposes instead extracting features via randomly initialized Graph Isomor- phism Networks (GINs) (5T). These features provide representation for each graph, so that metrics like FID, Precision/Recall, and Density/Coverage can be estimated using these representations. For
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

datasets with natural labels, we can alternatively find representations based on training a classifier; such features, however, are optimized for a very different task.
One significant challenge of graph data is that it is computationally difficult to even tell whether two graphs are the same (are isomorphic to one another). GINs are known to be as powerful at detecting isomorphism as the standard Weisteiler-Lchman (WL) test (51), but it is not known whether random GIN representations have this ability. Experimental evidence (5I) shows that trained GIN networks are as powerful as the WL test, but have substantially worse power at random initialization.
Unlike for distributions of natural images, however, the relevant graph features and particularly the semantics of node or edge features varies widely between datasets. On molecular graphs, adding any single edge will almost always violate physical constraints about stable atomic bonds; for house layouts, the feasibility of adding an edge depends significantly on the rest of the layout and the type of rooms (few front doors open into bathroom, and direct connections between rooms on the first and third floors are unlikely): Traditional metrics and random graph neural networks are both unlikely be able to capture these complex inter-dependencies functioning very differently across domains, and it similarly seems quite unlikely that pretraining on some "graph ImageNet" would be able to find useful generic graph features.
We therefore propose to train graph encoders on the same data used to train the generative model, using self- superviue contrastive learning to find meaningful feature extractors [[8] 19
[[8][9] 34 41 44
48 (5)) which we then use to compare the generated graphs to a test set. The set of perturbations
introduced in contrastive learning teaches the model which kinds of graphs should be considered
similar to one another. In this work we use types of perturbations traditionally used for training contrastive learning methods on graphs, but point out that future work focusing on domain-specific modeling can directly incorporate knowledge about which graphs should be considered similar by choosing different perturbation sets.
Inspired by the theoretical results for contrastive learning of (50), we also propose two variations to our representation learning procedures. This upper bound shows that contrastively learned representations work well for downstream tasks as long as the probability of data points yielding overlapping augmentations is relatively large for within-class dataset pairs and relatively small for cross-class pairs. Edit distances between graphs on typical training sets are large, however; we propose to use subgraphs ("crops") in our set of data augmentations for contrastive learning, which is more likely to yield overlaps. We also suggest enforcing a layer-wise Lipschitz constraint on feature extractors, which encourages similar graphs have similar learned representations. We show experimentally that both changes improve learning.
With all of these improvements, we further ask: can we theoretically guarantee that our learned GNN representations outperform traditional local metries? We prove that we cannot: we give examples of graphs easily distinguishable by local metrics that first-order GNNs cannot distinguish. Yet the converse is also true: we show graphs casily distinguishable by GNNs that appear equivalent to local metrics. We thus propose
models based Graph Substructure Networks (GSNs) (6) (using node degrees, and node clustering coefficients), and as a result, explicitly incorporating local metrics into models and surpassing the power of the WL test, and yield further improvements.
Related Work
Graph generative models. Our work is not particular to any type of graph generative model, as it focuses on simply evaluating samples; nonetheless, it is worth briefly reviewing some methods. Sequential generation models, such GraphRNN (52) and GRAN [25), generate nodes and edges in an auto-regressive manner. These models are efficient, but can struggle to capture global properties of the graphs, and require a predefined ordering. One-shot models such as MoIGAN [10). GraphDeconv
[13), Graph VAE [23), and GraphNVP (27). on the other hand, generate all nodes and edges in one shot; they can model long-range dependencies, but generally are not efficient and cannot scale to large graphs. For a detailed overview of graph generative models, see [16]
Evaluating graph generative models. Graph generative models can be even more challenging to evaluate than visual or textual models, because it is generally more difficult for humans to judge the quality of generated graph. The classic measure of the quality of a generative model, likelihood, also has significant issues with graphs: in addition to the kinds of issues that appear in generative

---

(a)
(b)
(c)
Figure (a) An example two graphs that can be differentiated by GNNs but not local metrics. Both graphs have same degree distribution, clustering coefficient and 4 node orbits. (b) An example of two graphs that can be differentiated by local metrics (clustering coefficient and smallest cycle) but not GNNs. (c) Adding single edge drastically change cluster coefficient and 4-node orbits.
models of images 455 the likelihood is particularly hard to evaluate on graphs where even checking equality is quite difficult 333. The most common method is to compute the Maximum Mean Discrepancy (MMD) [5] between distributions of local graph statistics. including node degrees,
cluster coefficients, and counts of orbits up particular size (25 [52]. Global statistics, such as
eigenvalues of the normalized graph Laplacian, are also used (52). These metrics, however, focus only on low-level structure of the graph, and ignore any features that might be present on nodes or edges [46]. Choosing an appropriate kernel is also very important to consistency of these metrics
[33]. These types of graph statisties might also encourage models to overfit to the training data, rather than truly learning the target distribution (40)
For unlabeled mmsge-based generative models, most work focuses metries including the In- ception Score (IS) (38], Fréchet Inception Distance (FID) [20), Precision/Recit (PR) [37]. and fixed latent space (typically activations of late layer in an Inception (43) model trained on ImageNet). These methods are rarely adopted in graph generative models, due to challenges with "general-purpose
Density/Coverage (2). all of which compare distributions
graph models discussed in Section [ Thompson et al. [46) thus used random (untrained) graph encoders in these metrics. We discuss these methods in more detail in the appendix. Our work, inspired by Morozov et similar proposal in image domains (31), explores the use of self- supervised contrastive training to find representations that work better than random initializations.
Graph Contrastive Learning. A popular method for self- self-supervised learning, contrastive learning generally aims to find a representation roughly invariant to various operations (c.g. for images, taking random crops, horizontal flipping, shifting colors) but able to identify different source data points. Ideally, such a representation will be useful for downstream tasks not known when learning the representation. In graph settings, learned representations may be the node, edge, or graph levels.
Early works adopting contrastive learning to encode graphs used DeepInfoMax (2T) loss enforce consistency between local (node) and global (graph) representations (e.g., DGI (48 and InfoGraph 41)) These methods did not use any specific augmentations and simply used distinct graphs as negative examples. Following works begin utilizing graph augmentations to further improve the learned representations. For example, MVGRL (18) uses graph diffusion and sub-graph inducing as two types of augmentations, where GCC (34] only uses multi-hop ego network inducing. More recent works use four types of graph augmentations including: feature masking, node dropping, edge perturbation, and sub-graph inducing [9 44 56). Among them, a few works rely trial-and- error or heuristics to choose those augmentations for given dataset (e.g., GraphCL [54), GRACE (55). and GCA (56)), whereas others introduce policy network to lear to sample augmentations and compute their parameters end-to-end along with graph representations (e.g., JOAO (53 and L.G2RR 19
3 GNNs Versus Local Metrics
Different methods for understanding graphs can "understand" the difference between graphs in very different ways: a particular change to graph might barely affect some features, while drastically changing others. One extreme case is when given metric cannot detect that two distinct (non-
3

---

isomorphic) graphs are different. Since graph isomorphism is a computationally difficult problem, we expect that all efficiently computable graph representations "collapse" some pairs of input graphs It is conceivable, however, that one method could be strictly more powerful than another. For instance, since recent GNN models have overcome traditional models based on local metric representations in variety of problems 23 47 5 is it the case that GNNs are strictly more powerful than local metrics?
We show, constructively, that the answer is no: there are indeed graphs which GNNs can distinguish and local metrics cannot, but there are also graphs which local metrics can distinguish but first-order GNNs cannot. Figure [ shows a pair of graphs with the same degree distribution, clustering coeffi- cient, and four-node orbits, which can nonetheless be distinguished by GNNs (proof in Appendix BB). On the other hand, the graphs in Fig. [b) have different clustering coefficient and smallest cycle, but first-order GNNs cannot tell them apart. Thus, neither method strictly outperforms the other on all problems, and so there are theoretical generative models which perfectly match in GNN-based representations but differ in local metries, and vice versa. This motivates our addition of local features to our graph representation models (Section 4
It is much easier to incorporated such hard-coded structures into GNNs than it would be add learning to feature metrics; in particular, counting higher-order local patterns quickly becomes prohibitively expensive, with super-exponentic time complexity. GNNs can also easily handle node and/or edge features on the underlying graphs, which is far more difficult to add to local metrics.
Another quality would like in our graph representations, in addition to the ability to distinguish distinct graphs, is some form of stability: if a distribution of graphs only changes slightly, we would like evaluation methods give "partial credit" as opposed to distribution where all graphs are dramatically different. (This is closely related to issues in training and evaluating mmage-based generative models [1-3 45). )) As previously discussed, the notion of "similar graphs" is very domain- dependent, but traditional local metrics can be highly sensitive to changes like adding a single edge: Fig. (e) shows an example where two graphs differing only by single edge can have drastically different statistics. By learning GNN representations, we can have some control over these types of smoothness properties; we exploit this explicitly in our methodology in Section [
Self-supervised Training of Graph Representations for Evaluation
Suppose we have two sets of graphs Gtrain = (Gi,... Gs} and Grest = -GI.-.- GM }. each sampled from the same data distribution p (G). Also suppose that we have access to unconditional graph generative model ga (-), which is trained on Grrain to learn the distribution of the observed set of graphs. We sample set of generated graphs Ggen = (G1,--. --. GN) ) ~ Pso (G) from this model. In order to evaluate the quality of the sampled graphs (i.e., decide whether the model (-) has successfully recovered the underlying distribution p (G)). we can define a measure of divergence D (Gtest Ggen) to quantify the discrepancy between distributions of the real and generated graphs. One robust way to achieve this is to define the metric on latent vector representation spaces, and expect representations of graphs rather than the original objects. Thus, to train a shared encoder f-(-) and then compute the discrepancy as 'D (fo (Gtet)) (G9e))) There are a few such metrics well-studied in visual domains that can differentiate the fidelity and diversity of the model, and which we can adopt in graph domains.
these metrics, we need
For evaluating image generative models, due to similar feature space across image datasets and also availability of large-scale data, the trunk of a model trained over ImageNet with explicit supervisory signals is usually chosen as the encoder. However, it is not straightforward to adopt the same trick to graph-struse strut-strectured data: there are no ImageNet-scall graph datasets available, and more importantly, the semantics of graphs and their features vary wildly across commonly used graph datasets, far more than occurs across distributions of natural images. For instance, even datasets of molecular graphs may use different feature sets represent the molecules [17), in addition the many cross-domain challenges discussed in Section [] Thus, it is not feasible to imagine a single "universal" graph representation; we would like a general-purpose purpose method for finding representations useful for a new graph dataset.
Alternatively, an efficient graph representation might be able to distinguish non isomorphic graphs, if it also sometimes distinguishes isomorphic graphs. We do not consider such representations in this paper.

---

Training
Encoder
((+))
Boorooo
fol.)
Graaa
Generstor
(x(-)
3 (G,)
Encoder fo(.)
Evaluation
Getil
Encoder
Metries
fa(.)
FID PR
DC
C
MMD
Generator
Encoder
(+(.)
fo(.)
Figure 2: During the training phase, we use the same training data (Gtrain) to train both the generator and the encoder networks. The encoder is trained using contrastive loss where two augmentation randomly sampled from of rational augmentations evaluation phase, sample the generator held-out set of real graphs Geest to the encoder to compute the graph representations. The representations are then used to compute robust metrics to quantify the discrepency between real and generated graphs.
and тÿ construct views of sampled graph Gg. During a generated
graphs G and feed it along with
Training with Graph Contrastive Learning To find expressive representation real and gener- ated graphs, train the encoder using contrastive objective. Assuming set of rational augmenta- tions Tover where each augmentation т; E T is defined as a function over graph G2 that generates an
view of the graph: G sampling strategy uses Tto draw positive samples from the joint distribution p (7, (Gx), (G&))) in
(Gx), ) contrastive framework with negative
order maximize the agreement between different views of the same graph Gy and to draw negative samples from the product of marginals p (T:(G&)) x (Ty(Gw:)) to minimize it for views from two distinct graphs G5 and GR # k.
As mentioned previously, we use a GIN architecture for our feature extractor. Other than the training process, the rest of evaluation pipeline is similar to that of Thompson et al. [46). who use random GIN weights. We consider two methods for training our GIN's parameters: GraphCL [54) and InfoGraph [42]. GraphCL randomly samples augmentation from four possible augmentaiotms including node dropping, edge perturbation, attribute masking, and sub-graph inducing based on random walk to which would like the representation be roughly invariant. GraphCL uses normalized temperature-scaled cross-entropy (NT--een) objective [7] to maximize the agreement between positive graph-level representations. InfoGraph works differently: it contrasts the graph-level representation with the representations individual nodes, which encode neighborhood structure information. InfoGraph uses DeepInfoMax (21) objective to maximize the mutual information between graph-level and node-level representations of each individual graph.
To build on the insight of Section 3 also consider various methods for adding information about the local graph structure node features, similarly to Graph Substructure Networks (6). The simplest such method is to add the degree of node explicit feature for the GIN consider. We do this, but also add, concatenation node features, higher-order local information as well: three-node and four-node clustering features for each node. Four node clustering coefficient calculated as:
Using Local Subgraph Counts as Input Features for GINs
E [(u.w))NN()) go (u, w)
Ca(v) E E(mmieNii) (deg(u) + deg(w) (u, w) 21(u N(ww))
(1)
5

---

where N()) denotes immediate neighbors of node deg(v) denotes the degree of node v, and qu,u, w) the number of common neighbors of and w, not counting v. Aggregating these features across the whole graph would give information on distribution of 4 node orbits of the graph, but this provides more localized information across the graph that is nonetheless difficult or impossible for a GIN to examine otherwise (8)
Choice of Augmentations Wang et al. [50,, building off work of Wang and Isola [49), study contrastive learning in general settings (with an eye towards vision applications), and provide an intriguing bound for the performance of downstream classifiers. To explain it, consider the "augmentation graph" of the training samples: if G, is the ith training example and G a random augmentation of that example, we connect the nodes G; and Gj in the augmentation graph if there are feasible augmentations GJ and G such that G Gj Wang et al. argue that downstream linear classifiers are likely to succeed if this augmentation graphi has stronger intra-class connectivity than it does cross-class connectivity, proving a tight connection between the two under particular setting with strong assumptions about this and related aspects of the setup.
Given the connection between the distribution metrics discussed in Appendix A.2 and classifier per- formance [c.g. 26] Section 4], if we accept the argument Wang et al. [50]. evaluation metries based on contrastively trained graph representation will give poor values (good classifier performance) when generated samples are not well-coneected real samples in the augmentation graph, and vice versa. If we choose augmentations appropriately, this is sensible behavior.
Enforcing Lipschitz Layers in Representation Networks The prior line of reasoning also suggests that we should choose augmentations that are capable of making real graphs look like one another. Edit distances between graphs, however, are typically large on the datasets we consider, and so augmentations based on adding or deleting individual nodes and/or edges will struggle to do this. The same is true for many of the augmentations used on images, except as Wang et al. (50) note for crop-type augmentations, where e.g. two different car pictures might become quite similar if we crop to just a wheel. On graphs, an analogous operation is subgraph sampling, which we include in our GraphCL. setup; InfoGraph already naturally looks at subgraph features as a core component.
Taking this line of reasoning as well as the general motivations of contrastive learning further, it is also natural to think that if can inherently enforce "similar graphs" to have similar representations, this could improve the process of contrastive learning: we would save on needing to train the model to learn these similarities, and it could help decrease the classifier performance for good generative models whose output graphs are legitimately near the distribution of target graphs
A line of work on GANs in visual settings 3
29 35 has made clear the importance of this type of reasoning in losses for training generative models: the loss should smoothly improve as generator samples approach the target distribution, even if the supports differ. Viewing model evaluation metrics as a kind of "out-of-the-loop loss function for training generative models hyperparameter selection and model development focusing on variants with better evaluation metrics suggests that these kinds of properties may be important for the problem of evaluation as well.
28
We thus explicitly enforce the layers of our GIN to have controlled Lipschitz constant, similarly to e.g. spectral normalization in GAN discriminators [29). To this end, we fix the À Lipschitz factor to 1.0 in the experiments. For each linear layer with weights We, after each update the weights, if Well > 1.0, update the value of the weights to W. T This guarantees small changes in the graphs, such adding/removing edges, or change in the input features, will not drastically change the final representation.
projected gradient descent; W:
5 Experimental Results
In all of the experiments we train the model on the full dataset in an self-supervised manner. Following 446). we take the dataset and make perturbations on the dataset and see what is the trend in the
Teehhniaall,, the augmentations we use (described Section would result densely-connecter augmentation graph: for instance, each graph has some vanishingly small probability of being reduced to an empty graph. We can instead think about the augmentation graph based on the augmentations from a high- gigbbprobability set for each training example.

---

measurements as the perturbation degree increases. We denote perturbation degree with and define it for each type of perturbation. We use these type of perturbations:
1.00
mixing-random
1.0
rewiring-edgee
0.75
0.8
Rank Correlation
Rank Correlation
0.6
0.25
0.4
0.00
-0.55
GraphCL GIN-Random
GraphCL GIN-Random
-0.50
0.2
Precision
Density
Recall
Coverage
FI PR
F1 DC
FD
MMD Linear
MHD RBF
Precision
Density
Recall
Coverage
DC
FD
MMD Linear
MMD RBF
F1
Fi
(b)
mode-collapse
mode-dropping
1.0
1.0
Rank Correlation
Rank Correlation
0.0
0.0
00.
GraphCL
GraphCL
-10
-1.0
GIN-Random
GIN-Random
Preciston
Density
Recall
Coverage
PR
Fi DC
FD
MMD Linear
MHD RBF
Precision
Density
Recall
Coverage
PR
DC
PD
MHD Linear
MHD ReF
Fl
F1
(c)
Figure 3: Pretrained GraphCL model versus randomly initialized GIN networks, violin results. The lines in the plots represent the quartiles. The results are gathered over all datasets and all random seeds. Sclf-supervised training shows better performance overall.
perturbation with ratio r, we remove r chunk of the reference samples, and replace them with Erdōs-Rényi (ER) graphs with the same ratio of edges.
Mixing-Randond
Rewiring Edges: This perturbation, rewires each edge of the graph with probability Each rewired edge, will change one of the sides of the edge with equal probability to another node that is not already connected to the stable node.
Mode Collapse and Mode Dropping:
For these perturbation, first we cluster the graphs using the WL-Kernel. First, we choose ratio of clusters, then, for mode collapse, we replace each graph that dataset with the center of the cluster. For, mode dropping, remove the selected clusters and then for making size the dataset fixed, we randomly repeat samples from other clusters.
For each experiment, measure the Spearman rand correlation between the perturbation ratio, and the value of the measurement. For measurements that supposed to decrease by the increase of ratio, flip the values. As result, in all experiments higher is better. We gather the results among different datasets and several random seeds and plot them for distribution of the correlations. For detailed experiments the individual datasets Appendix D
Datasets: Following 46 we use six diverse datasets (three synthetic and three real-world) that are frequently used in literature including: (1) Lobster [9] consisting of stochastic graphs with each node at most 2-hops away from backbone path, (2) 2D Grid graphs [9 25 52), (3) Proteins [1]] where represents connectivity among amino acids their physical distance is less than a threshold, (4) 3-hop Ego networks [52) extracted from the CiteSeer network (39) representing
staaton-based connectivity among documents, (5) Community (52) graphs generated by Brdōs-RRéníí model [12), and (6) Zinc
22 a dataset of attributed molecular graphs which allows study sensitivity of metrics to changes
7

---

in node/edge feature distributions. We follow the exact protocols used in 9 25 46 52] as follows.
We randomly sample 1000 graphs from Zine dataset and use one-hot encoding for edge/node features. For community dataset, we set = 11/ p = 0.3, and add 0.05/)/ inter-community edges with uniform probability. For all datasets, we use the node size range indicated in Table 4 in Appendix.
In the first experiment, we will examine the effects of contrastive training. We compare results of self-supervised GraphCL models versus randomly initialized GINs. For this comparison, we do not use structural features. The results are shown in Figure 3 and mean/median values are given in Table II In general we can see that in most measurements pretraining shows superior performance compared to the random network. Overall InfoGraph shows similar results. For InfoGraph results check Appendix [D] In this experiments we use random seeds and gather data all datasets. In Appendix D same results separated for each dataset can be seen. In our experience, pretraining shows near perfect results on larger datasets, but for Lobster and Grid datasets some correlations are not near 1. Our observation is these measurements are moving with the correct trend up to some perturbation ratio; but for example precision/recall become zero after some perturbation and keeps still. Our intuition here is because of highly regular structures in these datasets, model learns instantly to discriminate the real graphs from the fake ones very easily. And after small amount of perturbation the perturbed graphs are all very far from the reference graphs.
Contrastive Training Versus Random GIN:
Table 1: Mcan/Median values from Figures 3 and 4
Table summarizes the distributions by their mean and The results are gathered across datasets and random sceds.
median values.
Experiment
Model Name
Precision
Densiry
Recall
Covergge
FIPR
FIDC
FD
MMD Lin
MMD RBF
GIN-Random
0.7/1.
0
0.590160
0.9200955
0.977100
0.97/110
D.94/1.
0.91/10
0.97/10
Mixing Random
GrphCLL
1./1.0
10/10
0.77/0.9
0.95/0.77
1,01.0
10/1.0
10/10
L0/10
1,0/100
GrgliCL Full
LAV10
10/10
0.7840.9
0.96/0.97
L8710
1.0/1.0
0.97/100
0.95/100
L8V10
Random
0.578996
0.330035
0.7180.22
0.5778.56
0.990.97
0.560.97
0.770.99
0.9100.99
0,93A0.95
Rewiring Edges
GraphCI.
0.799998
0,91/0.99
0.91/0.33
0.9/0.99
0.930.97
0,910999
0.04.0999
0.99/100
0.99/10
GraphCL. Full
0.8600.88
0.86/0.99
0.79/0.87
0.86/0.98
0.8770.97
0.86/0.9
0.95/0.98
0.97/0.99
0,96/0 .9
GiN.Rannom
60.11-
-6.61-0866
0.880198
0.95/19
D3600 96
0.730188
0.572888
0.4300.74 74
0.98/1.
Mode Collapse
GraphC1.
-0.6510.99
-0.73/-0.88
0.94/0.99
0.94/19
0.8900997
0.58/0.75
0.71/0.99
0.59/0.84
0.98/100
GaphCT. Full
-5632-0.5
-0.61-0.76
0.95/0.99
0.97/1.9
0.91N1977
0.630.73
0.75ML99
0.7400.91
0.991.0
GIN Random
0.05--008
0,98771
0,99/10
0.384099
0.9130.55
0. 0,7500.92
0.71/0883
0.9859.99
Mode Dropping
GraphCL.
0.01.000
0.1580.23
0.98/0.9
0.99/1.0
0.98/0999
0.9660999 99
0.84/0.96
0.81/0.92
0.99/0999
GraphCL Foll
0.03/0.0
0.28/0.3
0.96/1.0
0.99/1.0
0.95/0.99
0.97/095
0.89/0.97
0.86/0.92
0,99/0.99
Effect of Adding Structural Information: To conduct this experiment, we concatenated node degrees and clustering coefficients for each node feature. Pretrained GraphCL models with and without structural features are compared. Figure shows the results, gathered over all datasets and random seeds; shoows mean/median values. Here we can see that adding these features improves the models power for some measurements. However, on highly regular and small datasets such as Grid results become poorer. Again intuition more powerful model can easier distinguish main graphs from the fake ones. Break down of the results on the datasets in Appendix D proves this point.
Generative Models Benchmark In this experiment, we use samples generated from the GRAN (25) after 25% of epochs, after full training, and also compare to holdout set of the real data (half of the original dataset). We pretrain a model with full structural features and evaluate the criteria on them. As expected, the curly-training results are worse than post-training results, while the real data gives almost perfect results. Results are provided in Table 2
Table 2: Comparative results for training GRAN model for 25% of epochs, full training, and 50/50 split of the datasets. GGM stands for Graph Generative Model.
Dataset
GGM
Precision
Density 1
Recall
Coverage
FIPRI
FIDC1
FDJ
MMD Lin|
MMD RBF]
GRAN-25%
0.21
0.23
0.56
0.43
0.31
0.30
6.4e5
3.1e4
0.21
Lobster
GRAN-1006
0.38
0.27
0.75
0.47
0.50
0.34
4.7e5
2.1e4
0.14
50/50 split
1.0
1.0
0.98
1.0
0.99
1.0
3.7
0.53
0.001
GRAN
0.0
0.0
1.0
0.0
0.0
0.0
3.5c9
2.3c9
0.94
Grid
GRAN-1O0K,
0.33
0.29
1.0
0.76
0.50
0.42
1.4c6
1.57c5
0.075
50/50 split
1.0
1.0
1.0
1.0
0.36
0.81
0.009
We analyze how layer normalization and using subgraph augmentations in GraphCL reflects on the final results. We have used no structural feature setup and conducted
Ablation Study:
8

---

rewiring-edges
mixing-random
1.0
1.0
0.8
Rank Correlation
Rank Correlation
0.6
0.6
0.4
GraphCL Full GraphCL
0.2
GraphCL
GraphCL
-0.2
Precision
Density
Recall
Coverage
F1 PR
FD
MMD Linear
MMD RBF
Preciston
Dersity
Recall
Coverage
F1 PR
DC
FD
MMD Linear
MMD ReF
Fi
mode-collapse
mode-dropping
1.0
1.0
0.5
Rank Correaation
Rank Correlation
0.0
0.0
005
GraphCL_Full GraphCL
GraphCL_Full
-1.0
-1.0
GraphCL
Precision
Density
Recall
Coverage
F1 PR
DC
FD
MMD Linear
MMD RBF
Precision
Density
Recail
Coverage
PR
F1 DC
FD
MMD Linear
MMD RBF
(d)
Figure Pretrained GraphCI. models with and without structural features were compared using violin results.
The GraphCL model does not have structural features, while the GraphCL Full model has structural features in addition to GraphCL. In the plots, the lines represent quartiles. The results are gathered across all datasets and random seeds. The overall distribution improves with structural features in some criteria.
mixing-random
mode-collapse
MY
0.8
0.8
Rank Correlation
Rank Correlation
0.6
0.6-
0.4
GraphCL
GraphCL
0.2
GraphCL2
0.2-
GraphCL2
GraphCL3
GraphCL3
0.0
Precision
Density
Recall
Coverage
F1 PR
F1 DC
FD
MMD Linear
MMD RBF
Precision
Density
Recall
Coverage
Fi PR
DC
FD
MMD Linear
MMD RBF
(b)
method on no feature data versus removing layer normalization versus removing subgraph data augmentations on pretraining the network. GraphCI. is normal training. In GraphCI 222 we do not enforce lipschitzness. In GraphCL3 remove the subgraph constraints and reduce the probability of node and edge dropping in the augmentations.
Figure 5: Comparison
the experiment for mixing-randon and mode-collapse experiments. Figure shows the results on this task. The results prove that both of these improvements are essential for getting better results.

---

Conclusion
We have demonstrated that self-supervised pretraining of representations can yield significantly better metrics for graph evaluation than random ones, particularly when incorporating local graph features with Lipschitz control, as inspired by theory. We suggest graph generative modeling papers should consider evaluating with these metrics in addition to or instead of their existing ones.
Acknowledgments and Disclosure of Funding
This research was enabled part by support, computational resources, and services provided by the Canada CIFAR AI Chairs program, the Natural Sciences and Engineering Research Council of Canada, WestGrid, and the Digital Research Alliance of Canada.
References
[1] Michael Arbel, Danica Sutherland, Mikotaj Binkkowsk,, and Arthur Gretton. "On Gradient Regularizers for MMD GANs." NeurIPS. 2018. arXiv: 1805. 11565
[2] Martin Arjovsky and Léon Bottou. "Towards Principled Methods for Training Generative Adversarial Networks." ICLR. 2017. Xiv: 1701. 04862
[3] Martin Arjovsky, Soumith Chintala, and Léon Bottou. "Wasserstein GAN." JCML. 2017. arXiv:
1701.07875.
[4] Mikotaj Binikowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. "Demystifying
MMD GANs." ICLR. 2018. arXiv: 18w.:
[5] Ali Borji. "Pros and cons of GAN evaluation measures." Computer Vision and Image Under- standing 179 (2019), pp. 41-65.
[6] Giorgos Bouritsas, Fabrizio Frasca, Stefanos P Zafeiriou, and Michael Bronstein. "Improving graph neural network expressivity via subgraph isomorphism counting." IEEE Transactions on Patlern Analysis and Machine Intelligence (2022).
[7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A framework for contrastive learning of visual representations. 2020. arxiv: 2002, 05709
[8] Zhengdao Chen, Lei Chen, Soledad Villar, and Joan Bruna. "Can graph neural networks count substructures?" NeurIPS 33 (2020). pp. 10383-10395.
[9] Hanjun Dai, Azade Nazi, Yujia Li, Bo Dai, and Dale Schuurmans. "Scalable deep generative modeling for sparse graphs." /CML. PMLR. 2020, pp. 2302-2312.
[10] Nicola De Cao and Thomas Kipf. "MoIGAN: An implicit generative model for small molecular
graphs" (2018). Xrv: 1805 11973
[11] Paul D Dobson and Andrew Doig. "Distinguishing enzyme structures from non-enzymes
without alignments."" Journal of Molecular Biology 330.4 (2003), pp. 771-783.
[12] Paul Erdos, Alfréd Rényi, et al. "On the evolution of random graphs." Publ. Math. Inst. Hung. Acad. Sci 5.1 (1960), pp. 17-60. Tony Wu. and Alan Aspuru-Guzik. "Graph deconvolutional generation" (2020). arXiv: 07087
[13]
[14] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. "Neural message passing for quantum chemistry." ICML. 2017, pp. 1263-1272.
[15] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. "A Kernel Two-Sample Test." Journal of Machine Learning Research 13.25 (2012), pp. 723-773.
[16] Xiaojie Guo and Liang Zhao. "A systematic survey on deep generative models for graph generation" (2020). arXiv: 2007.06686)
[17] Kaveh Hassani. Ctross-Domiin Few-Shot Graph Classification." Proceedings of the AAAI Conference on Artificial Intelligence 36.6 (2022), pp. 6856-688 6864.
[18] Kaveh Hassani and Amir Hosein Khasahmadi. "Contrastive Multi-View Representation Learn-
ing on Graphs." ICML. 2020, pp. 4116-4126.
[19] Kaveh Hassani and Amir Hosein Khasahmadi. "Learning Graph Augmentations to Learn Graph Representations" (2022). arXiv: 2201 09830
10

---

[20]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. "GANs trained by a two time-scale update rule converge to local Nash equilibrium." " NeurIPS (2017).
[21] R Devon Hjelm, Alex Fedorov, Samuel
Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. "Learning deep representations by mutual information estimation and maximization." ICLR. 2019.
[22] John J Irwin, Teague Sterling, Michael M Mysinger. Erin S Bolstad, and Ryan G Coleman. "ZINC: a free tool to discover chemistry for biology." Journal of chemical information and modeling 52.7 (2012), pp. 1757-1768.
[23] Thomas N Kipf and Max Welling. "Semi-supervised classification with graph convolutional networks." ICLR. 2016. arXiv:
[24] Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. "Learning Deep Generative Models of Graphs." ICLR. 2018.
[25] Renjie Liao, Yujia Li, Yang Song, Shenlong Wang, Charlie Nash, William L. Hamilton, David Duvenaud, Raquel Urtasun, and Richard S Zemel. "Efficient graph generation with graph recurrent attention networks" (2019). arXiv: 09910.0066
[26] Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica J. Sutherland. "I "Learning Deep Kernels for Non-Parametric Two-Sample Tests." ICMI.. 2020. Xiv: 2002
09116
[27]
Kaushalya Madhawa, Katushiko Ishiguro, Kosuke Nakago, and Motoki Abe. "GraphNVP: An invertible flow model for generating molecular graphs" (2019). arXiv: 1905. 11600
[28] Lars M. Mescheder, Andreas Geiger, and Sebastian Nowozin. "Which Training Methods for GANs do actually Converge?" ICML. 2018, arXiv: 1801 .44066
[29] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. "Spectral Normal- ization for Generative Adversarial Networks." ICLR. 2018. arXiv: (8021 05957
[30] Stanislav Morozov, Andrey Voynov, and Artem Babenko. "On Self-Supervised Image Repre- sentations for GAN Evaluation." /CLR. 2021.
[31] Stanislav Morozov, Andrey Voynov, and Artem Babenko. "On self-supervised image represen- tations for GAN evaluation." ICI.R. 2020.
[32] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh Yunjey Choi, and Jaejun Yoo.
"Reliable fidelity and diversity metries for generative models." ICML. PMLR. 2020, pp. 7176- 7185.
[33]
Leslie O' O'Bray, Max Horn, Bastian Rieck, and Karsten Borgwardt. "Evaluation Metrics for Graph Generative Models: Problems, Pitfalls, and Practical Solutions" (2021). arviv: 2106 01098
[34] Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang. and Jie Tang. "GCC: Graph Contrastive Coding for Graph Neural Network Pre- Training." International Conference on Knowledge Discovery and Data Mining. 2020. Kevin Roth, Aurélien Lucchi, Sebastian Nowozin, and Thomas Hofmann. "Stabilizing Training of Generative Adversarial Networks through Regularization." NeurIPS. NeurtIPS. 2017. arXiv: 1705
[35]
09367
[36] Olga Russakovsky et al. "ImageNet Large Scale Visual Recognition Challenge." International Journal of Computer Vision (IJCV) 115.3 (2015), pp. 211-252.
[37] Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. "As- sessing generative models via precision and recall." NewrIPS. 2018, pp. 5234-5243.
[38] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. "Improved techniques for training GANs." " NeurIPS (2016), pp. 2234-2242.
[39] Prithviraj Sen, Galileo Namata, Mustafa Bilgic. Lise Getoor, Brian Galligher, and Tina Eliassi- Rad, "Collective classification in network data." " Al magazine 29.3 (2008), pp. 93-93.
[40] Hamed Shirzad, Hossein Hajimirsadeghi, Amir Abdi, and Greg Mori. "TD-GEN: Graph Generation With Tree Decomposition" (2021). arXiv: 2106. 10656
[41] Fan- Yun Sun, Jordan Hoffman, Vikas Verma, and Jian Tang. "InfoGraph: Unsupervised and Semi- cuni-super Graph-Level Representation Learning via Mutual Information Maximization."
ICLR. 2020.

---

[42] Fan- Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. "Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization" (2019). arXiv: 1908. 01000
[43] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. "Re- thinking the Inception Architecture for Computer Vision." IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016.
[44] Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Rémi Munos, Petar Veličković, and Michal Valko. "Bootstrapped representation learning on graphs" (2021).
arXiv: 2102.06514
[45] Lucas Theis, Aäron van den Oord, and Matthias Bethge. "A note on the evaluation generative models." ICLR (2016).
[46] Rylee Thompson, Boris Knyazev, Elahe Ghalebi, Jungtaek Kim, and Graham W Taylor. "On Evaluation Metrics for Graph Generative Models" (2022). arXiv: 2201 .99811
[47] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. "Graph Attention Networks." ICLR. 2018. arXiv: 1710. 1710.10903,
[48] Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. "Deep Graph Infomax." ICLR. 2019.
[49] Tongzhou Wang and Phillip Isola. "Understanding contrastive representation learning through alignment and uniformity on the hypersphere." ICML. PMLR. 2020, pp. 9929-9939.
[50] Yifei Wang. Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. "Chaos is Ladder:
A New Theoretical Understanding Contrastive Learning via Augmentation Overlap" (2022).
arXiv: 2203. 13457
[51] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. "How powerful are graph neural networks?" (2018). arXiv: 1810.00826
[52] Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. "Graphrnn: Gen- erating realistic graphs with deep auto-regressive models." ICML. PMLR. 2018, pp. 5708- 5717.
[53] Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. "Graph Contrastive Learning Automated" (2021). Xrvv: 22166.00553
[54] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. "Graph contrastive learning with augmentations." NeurIPS (2020).
[55] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. "Deep graph
contrastive representation learning" (2020). arXiv: N006:200..r 04131
[56] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. "Graph contrastive
learning with adaptive augmentation." " Web Conference. 2021, pp. 2069-2080.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "v1Rgz-iME_j", "reviewer": "Reviewer_tjmf", "summary": "The paper proposes a self-supervised learning based approach to generate representations of real and synthetic graphs  in order to evaluate the quality of latter.  The general idea is to train graph encoders on the same data used to train a graph generator, using self-contrastive learning and then use these trained encoders to embed generated graphs and compare the embeddings with the same of real graphs in a test set. The claim is that such contrastively trained GNNs are more capable of extracting meaningful features from the graphs, compared to randomly initialized GNNs. The choice of perturbation schemes used in contrastive learning to train graph encoders could be domain specific. To demonstrate the merit of the idea, the paper chooses Graph Isomorphism Network (GIN) as the encoder architecture and ideas from two well-known graph self-supervised learning works (GraphCL and InfoGraph) to pretrain the encoders. The encoders are pre-trained on six datasets (three synthetic and three real-world) and then used to encode the real graphs as well as synthetic graphs generated from the four traditional graph-perturbation algorithms: Mixing-Random, Rewiring Edges, Mode Collapse, and Mode Dropping. To evaluate the goodness of proposed encoders, authors monitor the correlation (higher the better) between the degree of graph-perturbations and eight discrepancy metrics (commonly used to compare similarity between two graph distributions) between the distributions of latent representations for real graphs and generated graphs.  The proposed approach is compared against state-of-the-art technique where GIN is initialized randomly and the results indicate better correlation values for proposed approach as opposed  to SOTA approach. Additionally, authors also show that graph encoders supplemented with features capturing graph substructural information further aids to the distinguishing power of graph encoders. ", "questions": "1. Just to confirm, is there a particular reason to choose the contrastive learning schemes from InfoGraph and GraphCL in this paper?\n\n2. All the results in Figures 3-6 correspond to which of the six datasets? Or are the results combined across all the datasets?\n\n3. In several plots, I observe that rank correlations seem to be better for GIN-Random than the proposed approach (e.g. Figure 4(b), F1DC, F1PR, Coverage metrics). Am I reading the graph correctly? And if so, any explanation for why that might be the case?\n\n", "limitations": "I will recommend authors to talk about the limitations in the context of weaknesses I highlighted above if they agree with it. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The paper highlights the limitations of using generic local metrics to evaluate the two distributions. \n2. The idea of using graph-substructural features in training graph encoders along with Lipschitz control is quite plausible and also shown to work well. \n\nWeaknesses: \n1. The technical contribution of the paper is somewhat limited. The performance of the proposed evaluator is largely dependent on the choice of perturbation scheme used in contrastive learning as well as the architecture of the encoder. The right choices for each of the two components could largely vary with application domains and thus it may not be obvious in every application. And in case it is obvious, wouldn't that pretty much seal the choice of graph generator? \n\n2. It might be difficult to use the proposed approach to evaluate two graph generators G1 and G2 that use different perturbation schemes P1 and P2 respectively. In such a case, what perturbation scheme shall be used to train the evaluation encoder?\n\n3. For certain discrepancy metrics, the SOTA evaluator seem to be performing better than the proposed evaluator (see comments below), but it is not explained well as to why this might be happening.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kZQNDkpkPLe", "reviewer": "Reviewer_KHXf", "summary": "The authors propose a new method to evaluate graph generative models by evaluating the representations of the graph which are learned from a contrastive learning approach. They use the training data (which the generative model would also use to train on) and contrastive learning to learn a graph encoder, which they then use to generate representations of the test set and the generated graphs (or in the case of the this paper, of perturbed graphs, which they use to show the effectiveness of the method for evaluation). They show that neither GNNs nor local metrics are strictly better than the other for evaluating graph generative models, since each has a failure case that the other approach can distinguish, and use this as the motivation for trying to combine the respective strengths by using the Graph Substructure Network (which incorporates some local metrics) as their GNN and then train the encoder using contrastive learning and some predefined augmentations. They then assess their proposed method via various graph perturbations, and evaluate the spearman rank correlation between the perturbation and various metrics of interest.", "questions": "My questions/suggestions are described in the section above; I look forward to a discussion with the authors.", "limitations": "While there is no dedicated limitations section, the authors were upfront about the current limitations (no clear guidelines on how to choose the augmentations at present).", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is very well written, clear, and easy to follow.\n2. The topic is timely and seeks to solve an important problem in the graph representation learning community, namely how to evaluate graph generative models\n3. Very nice motivation for why existing approaches (local metrics, random GNNs) are insufficient\n4. Using contrastive learning to learn meaningful representations seems like a logical and promising approach to address this issue\n\nWeaknesses:\n\t\n1. I found it a bit difficult to assess the performance of the approach from the current paper. The visualizations are perhaps too overloaded, which obscured the takeaways (and there are no tables in the main paper).  I think the point is that GraphCL is better than GIN-Random, but the current visualizations don’t give me that immediate impression, they look quite similar. (When plots have so many variables to compare upon, I think it only works to easily digest if there is a clear, dramatic pattern - more subtle differences, even if consistently better, can get lost)  I think the authors should consider alternative visualizations/tables to convey their results more clearly; at a minimum, they could separate out GraphCL and GIN-Random into small multiples of the same plot (but there are probably better solutions). \n2. I think the experiments section would also benefit from a bit of restructuring. A lot of space is given to showing the results of not using structural features, using node degree features, and then using node clustering coefficients. I would naturally expect performance to improve as one keeps adding information, so I think there is too much space devoted to this conclusion at the expense of other data/visualization/textual explanations. I think the results you’re trying to convey are 1) graphCL outperforms random-GIN (in whatever comparison format makes most logical sense - e.g. with/without node degree features), and 2) GraphCL with node degree and clustering coefficients (i.e. local measures) further improve performance. I think the experiments section would be stronger if it was organized a bit more around these conclusions and providing the necessary data/plots to make that (or any other conclusions you’d like to highlight) very clear. If there are other conclusions from your experiments that I missed please feel free to include those as well.\n3. In general, the plots would benefit from a bit more explaining. For instance, I believe each violin plot contains 6 datasets x 10 random seeds, but it took me a few times searching in the paper to confirm that for myself. I’d make this very explicit, and also make the captions more self-contained.\n4. The authors highlight the limitations of using local metrics and using random GNNs, but only compare to using random GNNs. I think the local metrics should be added in as a comparison to confirm that the performance is also better than local metrics.\n5. It seems to me that a key benefit of this approach is what the authors stated about graphs being so heterogenous and therefore static approaches (e.g. degree histograms) may appropriately capture the relevant aspects of the data. Contrastive learning has the potential to address this issues since it can use augmentations that are relevant to the dataset, but as the authors themselves note, it is not clear how to choose such subgraphs, and they relegate this choice to future work (where domain specific choice may apply). The choice of augmentations strikes me as a key question in determining how to train the encoder to learn a meaningful representation. I understand this is a hard question to answer, and would be okay accepting this to future work, assuming that the experimental results were strong given the current generic augmentations. But with the current results I am not yet convinced; but am open to being persuaded if the results/visualizations can be more effectively conveyed.\n5. (Minor) The graph contrastive learning section in related work reads like a list - I would try to weave this into a more cohesive picture (as is done very nicely in the graph generative model related work section)\n6. (Minor) Is Nei(v) neighborhood in Eq. 1? Please define it somewhere.\n\nOn the whole, I enjoyed reading this paper, and I think it can be a very compelling paper once the experiments are strengthened and clarified a bit.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EyEfQN0kMd", "reviewer": "Reviewer_Mgpw", "summary": "This work mainly argue that we should use contrastively learned representations to evaluate the performance of graph generation methods. The key idea is to use the same training set to train the generative model and the encoder. The encoder is trained by contrastive self-supervised learning and will be used to evaluate the performance of the generative model. Experiments on multiple datasets in terms of multiple metrics are conducted to support the claim.", "questions": "(1) Does it take the same resource to train the GIN-random and GraphCL/InfoGraph? \n\n(2) For the encoder, no matter it is GIN-random or GraphCL/InfoGraph, how can we determine if it is trained well? Do we have a validation set for the encoder?", "limitations": "Not applicable.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### Pros ####\n(1) The intuition to use contrastively learned representations to evaluate the performance of graph generation methods is sound to me. Basically, contrastive learning teaches model which graphs are similar, thus being consistent with what we expect for generative models. In other words, we also hope that our generative models could generate graphs similar to the training graphs.\n\n(2) The experiment design is reasonable and can support the main claim clearly.\n\n(3) The presentation of this paper is great and easy to follow.\n\n#### Cons ####\n(1) The work claims that we should use contrastively learned representations to evaluate graph generative models. However, it does not consider evaluating any generative models in the experiments. It mainly evaluates the correlation between perturbation ratio and the defined value of measurements. Although it can support the claim to a large degree, it could be more convincing if any generative models can be evaluated and analyzed using the proposed strategy.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept