On Evaluation Metrics for Graph Generative Models

In image generation , generative models can be evaluated naturally by visually inspecting model outputs . However , this is not always the case for graph generative models ( GGMs ) , making their evaluation challenging . Currently , the standard process for evaluating GGMs suffers from three critical limitations : i ) it does not produce a single score which makes model selection challenging , ii ) in many cases it fails to consider underlying edge and node features , and iii ) it is prohibitively slow to perform . In this work , we mitigate these issues by searching for scalar , domain-agnostic , and scalable metrics for evaluating and ranking GGMs . To this end , we study existing GGM metrics and neural-network-based metrics emerging from generative models of images that use embeddings extracted from a task-specific network . Motivated by the power of certain Graph Neural Networks ( GNNs ) to extract meaningful graph representations without any training , we introduce several metrics based on the features extracted by an untrained random GNN . We design experiments to thoroughly test metrics on their ability to measure the diversity and fidelity of generated graphs , as well as their sample and computational efficiency . Depending on the quantity of samples , we recommend one of two random-GNN-based metrics that we show to be more expressive than pre-existing metrics . While we focus on applying these metrics to GGM evaluation , in practice this enables the ability to easily compute the dissimilarity between any two sets of graphs regardless of domain . 1 INTRODUCTION . Graph generation is a key problem in a wide range of domains such as molecule generation ( Samanta et al. , 2020 ; Popova et al. , 2019 ; Li et al. , 2018 ; Kong et al. , 2021 ; Jin et al. , 2020 ) and structure generation ( Bapst et al. , 2019 ; Thompson et al. , 2020 ) . An evaluation metric that is capable of accurately measuring the distance between a set of generated and reference graphs is critical for advancing research on graph generative models ( GGMs ) . This is frequently done by comparing empirical distributions of graph statistics such as orbit counts , degree coefficients , and clustering coefficients through Maximum Mean Discrepancy ( MMD ) ( You et al. , 2018 ; Gretton et al. , 2006 ) . While these metrics are capable of making a meaningful comparison between generated and real graphs ( You et al. , 2018 ) , this evaluation method yields a metric for each individual statistic . In addition , recent works have further increased the number of metrics by performing MMD directly with node and edge feature distributions ( Goyal et al. , 2020 ) , or on alternative graph statistics such as graph spectra ( Liao et al. , 2019 ) . While this is not an issue provided there is a primary statistic of interest , all metrics are frequently displayed together to approximate generation quality and evaluate GGMs ( You et al. , 2018 ; Liao et al. , 2019 ) . This process makes it challenging to measure progress as the ranking of generative models may vary between metrics . In addition , the computation of the metrics from You et al . ( 2018 ) can be prohibitively slow ( Liao et al. , 2019 ; O ’ Bray et al. , 2021 ) , and they are based only on graph structure , meaning they do not incorporate edge and node features . Therefore , they are less applicable in specific domains such as molecule generation where such features are essential . This particular limitation has led to the use of the Neighborhood Subgraph Pairwise Distance kernel ( NSPDK ) ( Costa & Grave , 2010 ) in GGM evaluation ( Goyal et al. , 2020 ; Podda & Bacciu , 2021 ; Kawai et al. , 2019 ) as it naturally incorporates edge and node features . However , this metric is still unable to incorporate continuous features in evaluation ( Costa & Grave , 2010 ) . Faced with a wide array of metrics and ambiguity regarding when each should be the focus , the community needs robust and scalable standalone metrics that can consistently rank GGMs . While less popular , metrics from image generation literature have been successfully utilized in GGM evaluation . These metrics rely on the use of a task-specific neural network to extract meaningful representations of samples , enabling a more straightforward comparison between generated and reference distributions ( Preuer et al. , 2018 ; Liu et al. , 2019 ; Thompson et al. , 2020 ) . Although these metrics have been validated empirically in the image domain , they are not universally applicable to GGMs . For example , Fréchet Chemnet Distance ( Preuer et al. , 2018 ) uses a language model trained on SMILES strings , rendering it unusable for evaluation of GGMs in other domains . Furthermore , a pretrained GNN can not be applied to datasets with a different number of edge or node labels . Pretraining a GNN for every dataset can be prohibitive , making the use of such metrics in GGM evaluation less appealing than in the more established and standardized image domain . In image generation evaluation , classifiers trained on ImageNet ( Deng et al. , 2009 ) are frequently used to extract image embeddings ( Bińkowski et al. , 2018 ; Heusel et al. , 2017 ; Kynkäänniemi et al. , 2019 ; Xu et al. , 2018 ; Naeem et al. , 2020 ) . While classifiers such as Inception v3 ( Szegedy et al. , 2016 ) are consistently used , recent works have investigated the use of randomly-initialized CNNs with no further training ( hereafter referred to as a random network ) in generative model evaluation . Xu et al . ( 2018 ) ; Naeem et al . ( 2020 ) found that a random CNN performs similarly to ImageNet classifiers on natural images and is superior outside of the natural image domain . In the graph domain , random GNNs have been shown to extract meaningful features to solve downstream graph tasks without training ( Kipf & Welling , 2017 ; Morris et al. , 2019 ; Xu et al. , 2019 ) . However , the applicability of random GNNs for the evaluation of GGMs remains unexplored . In this work , we aim to identify one or more scalar metrics that accurately measures the dissimilarity between two sets of graphs to simplify the ranking of GGMs regardless of domain . We tackle this problem by exploring the use of random GNNs in the evaluation of GGMs using metrics that were developed in the image domain . In addition , we perform objective evaluation of a large number of possible evaluation metrics . We design experiments to thoroughly test each metric on its ability to measure the diversity and fidelity ( realism ) of generated graphs , as well as their sample and computational efficiency . We study three families of metrics : existing GGM evaluation metrics based on graph statistics and graph kernels , which we call classical metrics ; image domain metrics using a random GNN ; and image domain metrics using a pretrained GNN . We aim to answer the following questions empirically : ( Q1 ) What are the strengths and limitations of each metric ? ( Q2 ) Is pretraining a GNN necessary to accurately evaluate GGMs with image domain metrics ? ( Q3 ) Is there a strong scalar and domain-agnostic metric for evaluating and ranking GGMs ? Addressing these questions enabled us to reveal several surprising findings that directly impact the future of GGM evaluation . For example , regarding Q1 , we identify a failure mode in the classical metrics in that they are poor at measuring the diversity of generated graphs . Consequently , we find several metrics that are more expressive . In terms of Q2 , we determine that pretraining is unnecessary to utilize neural-network-based ( NN-based ) metrics . Regarding Q3 , we find two scalar metrics that are ideal for evaluating and ranking GGMs in certain scenarios ; they are scalable , powerful , and easily incorporate continuous or discrete node and edge features . These findings enable computationally inexpensive and domain-agnostic GGM evaluation . 2 BACKGROUND & RELATED WORK . Evaluating generative models in any domain is a notoriously difficult task ( Theis et al. , 2016 ) . However , researchers have found success through the use of sample-based evaluation metrics that estimate the distance ρ between real and generated distributions Pr and Pg by drawing random samples ( Heusel et al. , 2017 ; You et al. , 2018 ; Bińkowski et al. , 2018 ) . That is , they compute ρ̂ ( Sg , Sr ) ≈ ρ ( Pg , Pr ) , with Sr = { xr1 , . . . , xrm } ∼ Pr and Sg = { x g 1 , . . . , x g n } ∼ Pg , where xi is defined as some feature vector extracted from a corresponding graph Gi . These metrics are model agnostic and therefore applicable to all generative models . 2.1 CLASSICAL METRICS . Metrics based on graph statistics ( You et al. , 2018 ) are standard in evaluating GGMs ( Liao et al. , 2019 ; Dai et al. , 2020 ) . These metrics require extracting the clustering coefficient , node degree , and 4-node orbit count histograms1 that are then used to compute the empirical MMD between generated and reference sets Sg , Sr ( Gretton et al. , 2006 ) : MMD ( Sg , Sr ) : = 1 m2 m∑ i , j=1 k ( xri , x r j ) + 1 n2 n∑ i , j=1 k ( xgi , x g j ) − 2 nm n∑ i=1 m∑ j=1 k ( xgi , x r j ) , ( 1 ) where k ( · , · ) is a general kernel function . You et al . ( 2018 ) proposed a form of the RBF kernel : k ( xi , xj ) = exp ( −d ( xi , xj ) 2σ2 ) , ( 2 ) where d ( · , · ) computes pairwise distance , and in that work was chosen to be the Earth Mover ’ s Distance ( EMD ) . This yields three metrics , one for each graph statistic . The computational cost of these metrics may be decreased by using the total variation distance as d ( · , · ) in Equation 1 ( Liao et al. , 2019 ) . However , this change leads to an indefinite kernel and undefined behaviour ( O ’ Bray et al. , 2021 ) . Therefore , we only compute these metrics using EMD ( You et al. , 2018 ) . In addition , several works ( Goyal et al. , 2020 ; Podda & Bacciu , 2021 ; Kawai et al. , 2019 ) evaluate GGMs by replacing k ( · , · ) with the Neighborhood Subgraph Pairwise Distance graph kernel ( NSPDK ) . This metric has the benefit of incorporating discrete edge and node features along with the underlying graph structure in evaluation . Similar to the metrics proposed by You et al . ( 2018 ) , Moreno et al . ( 2018 ) extract graph structure properties such as node degree , clustering coefficient , and geodesic distance . However , these properties are then combined into a scalar metric through the KolmorogovSmirnov ( KS ) multidimensional distance ( Justel et al. , 1997 ) . We exclude KS from our experiments as it is unable to incorporate edge and node features , which is one of the key properties we seek . Finally , note that other domain-specific metrics such as “ percentage of valid graphs ” exist . Our goal is not to incorporate , eliminate , or evaluate such metrics ; they are properties of generated graphs , and unlike the metrics described above do not provide a comparison to a reference distribution . We believe that such metrics can still provide valuable information in GGM evaluation .

Summary: This paper evaluates the effectiveness of different metrics for graph generative models from many perspectives. They thoroughly study the following factors: fidelity, diversity, sensitivity to node/edge features. They find that pre-existing GGM metrics fail to capture the diversity of data and find several random GIN-based metrics that are more expressive and have low computational costs.