On Evaluation Metrics for Graph Generative Models

In image generation , generative models can be evaluated naturally by visually inspecting model outputs . However , this is not always the case for graph generative models ( GGMs ) , making their evaluation challenging . Currently , the standard process for evaluating GGMs suffers from three critical limitations : i ) it does not produce a single score which makes model selection challenging , ii ) in many cases it fails to consider underlying edge and node features , and iii ) it is prohibitively slow to perform . In this work , we mitigate these issues by searching for scalar , domain-agnostic , and scalable metrics for evaluating and ranking GGMs . To this end , we study existing GGM metrics and neural-network-based metrics emerging from generative models of images that use embeddings extracted from a task-specific network . Motivated by the power of certain Graph Neural Networks ( GNNs ) to extract meaningful graph representations without any training , we introduce several metrics based on the features extracted by an untrained random GNN . We design experiments to thoroughly test metrics on their ability to measure the diversity and fidelity of generated graphs , as well as their sample and computational efficiency . Depending on the quantity of samples , we recommend one of two random-GNN-based metrics that we show to be more expressive than pre-existing metrics . While we focus on applying these metrics to GGM evaluation , in practice this enables the ability to easily compute the dissimilarity between any two sets of graphs regardless of domain . 1 INTRODUCTION . Graph generation is a key problem in a wide range of domains such as molecule generation ( Samanta et al. , 2020 ; Popova et al. , 2019 ; Li et al. , 2018 ; Kong et al. , 2021 ; Jin et al. , 2020 ) and structure generation ( Bapst et al. , 2019 ; Thompson et al. , 2020 ) . An evaluation metric that is capable of accurately measuring the distance between a set of generated and reference graphs is critical for advancing research on graph generative models ( GGMs ) . This is frequently done by comparing empirical distributions of graph statistics such as orbit counts , degree coefficients , and clustering coefficients through Maximum Mean Discrepancy ( MMD ) ( You et al. , 2018 ; Gretton et al. , 2006 ) . While these metrics are capable of making a meaningful comparison between generated and real graphs ( You et al. , 2018 ) , this evaluation method yields a metric for each individual statistic . In addition , recent works have further increased the number of metrics by performing MMD directly with node and edge feature distributions ( Goyal et al. , 2020 ) , or on alternative graph statistics such as graph spectra ( Liao et al. , 2019 ) . While this is not an issue provided there is a primary statistic of interest , all metrics are frequently displayed together to approximate generation quality and evaluate GGMs ( You et al. , 2018 ; Liao et al. , 2019 ) . This process makes it challenging to measure progress as the ranking of generative models may vary between metrics . In addition , the computation of the metrics from You et al . ( 2018 ) can be prohibitively slow ( Liao et al. , 2019 ; O ’ Bray et al. , 2021 ) , and they are based only on graph structure , meaning they do not incorporate edge and node features . Therefore , they are less applicable in specific domains such as molecule generation where such features are essential . This particular limitation has led to the use of the Neighborhood Subgraph Pairwise Distance kernel ( NSPDK ) ( Costa & Grave , 2010 ) in GGM evaluation ( Goyal et al. , 2020 ; Podda & Bacciu , 2021 ; Kawai et al. , 2019 ) as it naturally incorporates edge and node features . However , this metric is still unable to incorporate continuous features in evaluation ( Costa & Grave , 2010 ) . Faced with a wide array of metrics and ambiguity regarding when each should be the focus , the community needs robust and scalable standalone metrics that can consistently rank GGMs . While less popular , metrics from image generation literature have been successfully utilized in GGM evaluation . These metrics rely on the use of a task-specific neural network to extract meaningful representations of samples , enabling a more straightforward comparison between generated and reference distributions ( Preuer et al. , 2018 ; Liu et al. , 2019 ; Thompson et al. , 2020 ) . Although these metrics have been validated empirically in the image domain , they are not universally applicable to GGMs . For example , Fréchet Chemnet Distance ( Preuer et al. , 2018 ) uses a language model trained on SMILES strings , rendering it unusable for evaluation of GGMs in other domains . Furthermore , a pretrained GNN can not be applied to datasets with a different number of edge or node labels . Pretraining a GNN for every dataset can be prohibitive , making the use of such metrics in GGM evaluation less appealing than in the more established and standardized image domain . In image generation evaluation , classifiers trained on ImageNet ( Deng et al. , 2009 ) are frequently used to extract image embeddings ( Bińkowski et al. , 2018 ; Heusel et al. , 2017 ; Kynkäänniemi et al. , 2019 ; Xu et al. , 2018 ; Naeem et al. , 2020 ) . While classifiers such as Inception v3 ( Szegedy et al. , 2016 ) are consistently used , recent works have investigated the use of randomly-initialized CNNs with no further training ( hereafter referred to as a random network ) in generative model evaluation . Xu et al . ( 2018 ) ; Naeem et al . ( 2020 ) found that a random CNN performs similarly to ImageNet classifiers on natural images and is superior outside of the natural image domain . In the graph domain , random GNNs have been shown to extract meaningful features to solve downstream graph tasks without training ( Kipf & Welling , 2017 ; Morris et al. , 2019 ; Xu et al. , 2019 ) . However , the applicability of random GNNs for the evaluation of GGMs remains unexplored . In this work , we aim to identify one or more scalar metrics that accurately measures the dissimilarity between two sets of graphs to simplify the ranking of GGMs regardless of domain . We tackle this problem by exploring the use of random GNNs in the evaluation of GGMs using metrics that were developed in the image domain . In addition , we perform objective evaluation of a large number of possible evaluation metrics . We design experiments to thoroughly test each metric on its ability to measure the diversity and fidelity ( realism ) of generated graphs , as well as their sample and computational efficiency . We study three families of metrics : existing GGM evaluation metrics based on graph statistics and graph kernels , which we call classical metrics ; image domain metrics using a random GNN ; and image domain metrics using a pretrained GNN . We aim to answer the following questions empirically : ( Q1 ) What are the strengths and limitations of each metric ? ( Q2 ) Is pretraining a GNN necessary to accurately evaluate GGMs with image domain metrics ? ( Q3 ) Is there a strong scalar and domain-agnostic metric for evaluating and ranking GGMs ? Addressing these questions enabled us to reveal several surprising findings that directly impact the future of GGM evaluation . For example , regarding Q1 , we identify a failure mode in the classical metrics in that they are poor at measuring the diversity of generated graphs . Consequently , we find several metrics that are more expressive . In terms of Q2 , we determine that pretraining is unnecessary to utilize neural-network-based ( NN-based ) metrics . Regarding Q3 , we find two scalar metrics that are ideal for evaluating and ranking GGMs in certain scenarios ; they are scalable , powerful , and easily incorporate continuous or discrete node and edge features . These findings enable computationally inexpensive and domain-agnostic GGM evaluation . 2 BACKGROUND & RELATED WORK . Evaluating generative models in any domain is a notoriously difficult task ( Theis et al. , 2016 ) . However , researchers have found success through the use of sample-based evaluation metrics that estimate the distance ρ between real and generated distributions Pr and Pg by drawing random samples ( Heusel et al. , 2017 ; You et al. , 2018 ; Bińkowski et al. , 2018 ) . That is , they compute ρ̂ ( Sg , Sr ) ≈ ρ ( Pg , Pr ) , with Sr = { xr1 , . . . , xrm } ∼ Pr and Sg = { x g 1 , . . . , x g n } ∼ Pg , where xi is defined as some feature vector extracted from a corresponding graph Gi . These metrics are model agnostic and therefore applicable to all generative models . 2.1 CLASSICAL METRICS . Metrics based on graph statistics ( You et al. , 2018 ) are standard in evaluating GGMs ( Liao et al. , 2019 ; Dai et al. , 2020 ) . These metrics require extracting the clustering coefficient , node degree , and 4-node orbit count histograms1 that are then used to compute the empirical MMD between generated and reference sets Sg , Sr ( Gretton et al. , 2006 ) : MMD ( Sg , Sr ) : = 1 m2 m∑ i , j=1 k ( xri , x r j ) + 1 n2 n∑ i , j=1 k ( xgi , x g j ) − 2 nm n∑ i=1 m∑ j=1 k ( xgi , x r j ) , ( 1 ) where k ( · , · ) is a general kernel function . You et al . ( 2018 ) proposed a form of the RBF kernel : k ( xi , xj ) = exp ( −d ( xi , xj ) 2σ2 ) , ( 2 ) where d ( · , · ) computes pairwise distance , and in that work was chosen to be the Earth Mover ’ s Distance ( EMD ) . This yields three metrics , one for each graph statistic . The computational cost of these metrics may be decreased by using the total variation distance as d ( · , · ) in Equation 1 ( Liao et al. , 2019 ) . However , this change leads to an indefinite kernel and undefined behaviour ( O ’ Bray et al. , 2021 ) . Therefore , we only compute these metrics using EMD ( You et al. , 2018 ) . In addition , several works ( Goyal et al. , 2020 ; Podda & Bacciu , 2021 ; Kawai et al. , 2019 ) evaluate GGMs by replacing k ( · , · ) with the Neighborhood Subgraph Pairwise Distance graph kernel ( NSPDK ) . This metric has the benefit of incorporating discrete edge and node features along with the underlying graph structure in evaluation . Similar to the metrics proposed by You et al . ( 2018 ) , Moreno et al . ( 2018 ) extract graph structure properties such as node degree , clustering coefficient , and geodesic distance . However , these properties are then combined into a scalar metric through the KolmorogovSmirnov ( KS ) multidimensional distance ( Justel et al. , 1997 ) . We exclude KS from our experiments as it is unable to incorporate edge and node features , which is one of the key properties we seek . Finally , note that other domain-specific metrics such as “ percentage of valid graphs ” exist . Our goal is not to incorporate , eliminate , or evaluate such metrics ; they are properties of generated graphs , and unlike the metrics described above do not provide a comparison to a reference distribution . We believe that such metrics can still provide valuable information in GGM evaluation .

Summary: The paper shows a detailed comparison of different graph generative model evaluation metrics and highlights that current approaches for the evaluation of GGMs are insufficient and perform poorly in terms of assessing diversity and fidelity of generated samples. The paper pinpoints these issues to the reliance of many metrics on a predefined set of features extracted from the generated graphs. It claims that the efficacy of prior approaches is typically dependent on the graph featurization and that this leads to inconsistencies in the ranking of models when using different feature representations. It proposes to instead use features from randomly initialized Graph Neural Networks as a basis for the analysis and shows that this represents a competitive evaluation approach.