Evaluation Metrics for Graph Generative Models: Problems, Pitfalls, and Practical Solutions

1 INTRODUCTION . Graph generative models have become an active research branch , making it possible to generalise structural patterns inherent to graphs from certain domains—such as chemoinformatics—and actively synthesise new graphs ( Liao et al. , 2019 ) . Next to the development of improved models , their evaluation is crucial . This is a well-studied issue in other domains , leading to metrics such as the ‘ Fréchet Inception Distance ’ ( Heusel et al. , 2017 ) for comparing image-based generative models . Graphs , however , pose their own challenges , foremost among them being that an evaluation based on visualisations , i.e . on perceived differences , is often not possible . In addition , virtually all relevant structural properties of graphs exhibit spatial invariances , e.g. , connected components and cycles , are invariant with respect to rotations—that have to be taken into account by a comparison metric . While the community has largely gravitated towards a single comparison metric , the maximum mean discrepancy ( MMD ) ( Chen et al. , 2021 ; Dai et al. , 2020 ; Goyal et al. , 2020 ; Liao et al. , 2019 ; Mi et al. , 2021 ; Niu et al. , 2020 ; Podda & Bacciu , 2021 ; You et al. , 2018 ; Zhang et al. , 2021 ) , neither its expressive power nor its other properties have been systematically investigated in the context of graph generative model comparison . The goal of this paper is to provide such an investigation , starting from first principles by describing the desired properties of such a comparison metric , providing an overview of what is done in practice today , and systemically assessing MMD ’ s behaviour using recent graph generative models . We highlight some of the caveats and shortcomings of the existing status quo , and provide researchers practical recommendations to address these issues . We note here that our investigations focus on assessing the structural similarity of graphs , which is a necessary first step before one can jointly assess graphs based on structural and attribute similarity . This paper purposefully refrains from developing its own graph generative model to avoid any bias in the comparison . 2 COMPARING GRAPH DISTRIBUTIONS . In the following , we will deal with undirected graphs . We denote such a graph as G = ( V , E ) with vertices V and edges E. We treat graph generative models as black-box models , each of which results in a set1 of graphs G. The original distribution of graphs is denoted as G∗ . Given models { M1 , M2 , . . . } , with generated sets of graphs { G1 , G2 , . . . } , the goal of generative model evaluation is to assess which model is a better fit , i.e . which distribution is closer to G∗ . This requires the use of a ( pseudo- ) metric d ( · , · ) to assess the dissimilarity between G∗ and generated graphs . We argue that the desiderata of any such comparison metric are as follows : 1 . Expressivity : if G and G′ do not arise from the same distribution , a suitable metric should be able to detect this . Specifically , d ( G , G′ ) should be monotonically increasing the further apart G and G′ are from one another . 2 . Robustness : if a distribution G is subject to a perturbation , a suitable metric should be robust to small perturbations . Changes in the metric should be ideally upper-bounded by a function of the amplitude of the perturbation . Robust metrics are preferable because of the inherent stochasticity of training generative models . 3 . Efficiency : model comparison metrics should be reasonably fast to calculate ; even though model evaluation is a post hoc analysis , a metric should scale well with an increasing number of graphs and an increasing size of said graphs . While there are many ways to compare two distributions , ranging from statistical divergence measures to proper metrics in the mathematical sense , the comparison of distributions of graphs is exacerbated by the fact that individual graphs typically differ in their cardinalities and are only described up to permutation . With distances such as the graph edit distance being NP-hard in general ( Zeng et al. , 2009 ) , which precludes using them as an efficient metric , a potential alternative is provided by using descriptor functions . A descriptor function f maps a graph G to an auxiliary representation in some space Z . The problem of comparing a generated distribution G = { G1 , . . . , Gn } to the original distribution G∗ = { G∗1 , . . . , G∗m } thus boils down to comparing the images f ( G ) : = { f ( G1 ) , . . . , f ( Gn ) } ⊆ Z and f ( G∗ ) ⊆ Z by any preferred statistical distance in Z . Figure 1 depicts an overview of this approach as it is being done today . 3 CURRENT STATE OF GRAPH GENERATIVE MODEL EVALUATION : MMD . Of particular interest in previous literature is the case of Z = Rd and using the maximum mean discrepancy dMMD ( · , · ) as a metric . MMD is one of the most versatile and expressive options available for comparing distributions of structured objects such as graphs ( Borgwardt et al. , 2006 ; Gretton et al. , 2007 ) , providing also a principled way to perform two-sample tests ( Bounliphone et al. , 2016 ; Gretton et al. , 2012a ; Lloyd & Ghahramani , 2015 ) . It enables the comparison of two statistical distributions by means of kernels , i.e . similarity measures for structured objects . Letting X refer to a 1Formally , the output can also be a multiset , because graphs are allowed to be duplicated . non-empty set , a function k : X × X → R is a kernel if k ( xi , xj ) = k ( xj , xi ) for xi , xj ∈ X and∑ i , j cicj k ( xi , xj ) ≥ 0 for xi , xj ∈ X and ci , cj ∈ R. MMD uses such a kernel function to assess the distance between two distributions . Given n samples X = { x1 , . . . , xn } ⊆ X and m samples Y = { y1 , . . . , ym } ⊆ X , the empirical estimate of the MMD between X and Y is obtained as MMD ( X , Y ) : = 1 n2 n∑ i , j=1 k ( xi , xj ) + 1 m2 m∑ i , j=1 k ( yi , yj ) − 2 nm n∑ i=1 m∑ j=1 k ( xi , yj ) . ( 1 ) Since MMD is known to be a metric on the space of probability distributions under certain conditions , Eq . 1 is often treated as a metric as well ( Liao et al. , 2019 ; Niu et al. , 2020 ; You et al. , 2018 ) , even though it would be more correct to only use it for hypothesis/two-sample testing , since as the authors of MMD themselves note , the empirical estimate of MMD can even take negative values ( Gretton et al. , 2012a ) , which we also observed in our experiments.2 Nevertheless , MMD has been adopted by the community and requires ( i ) choosing a descriptor function f as described above , and ( ii ) choosing a kernel on Rd such as an RBF kernel . One then evaluates dMMD ( G , G∗ ) : = MMD ( f ( G ) , f ( G∗ ) ) for a distribution G. Given multiple distributions { G1 , G2 , . . . } , the values dMMD ( Gi , G∗ ) can be used to rank models : smaller values are assumed to indicate a larger agreement with the original distribution G∗ . We will now describe this procedure in more detail and highlight some of its pitfalls . 3.1 KERNELS & DESCRIPTOR FUNCTIONS . Before one can calculate the MMD distance between two distributions , it is necessary to define both the kernel function k and the descriptor function f that will convert a graph G to a representation in Z = Rd , for use in the MMD calculation in Eq . 1 . We observed a variety of kernel choices in the existing literature . In fact , in three of the most popular graph generative models in use today , which we explore in detail in this paper , a different kernel was chosen for each one . These include a kernel using the first Wasserstein distance ( EMD ) , total variation distance ( TV ) , and the radial basis function kernel ( RBF ) , and are listed in Table 1 . In the current use of MMD , a descriptor function f is used to create a vectorial representation of a graph for use in the kernel computation . We find that several descriptor functions are commonly employed , either based on summary statistics of a graph , such as degree distribution histogram and clustering coefficient histogram , or based on spectral properties of the graph , such as the Laplacian spectrum histogram . While several papers also consider the orbit as a descriptor function , we do not consider it in depth here due to its computational complexity , which violates the “ efficiency ” property from our desiderata . We will now provide brief explanations of these prominent descriptor functions . Degree distribution histogram . Given a graph G = ( V , E ) , we obtain a histogram by evaluating deg ( v ) for v ∈ V , where position i of the resulting histogram is the number of vertices with degree i . Assuming a maximum degree d and extending the histogram with zeros whenever necessary , we obtain a mapping f : G → Rd . This representation has the advantage of being easy to calculate and easy to compare ; by normalising it ( so that it sums to 1 ) , we obtain a size-invariant descriptor . Clustering coefficient . The ( local ) clustering coefficient of a vertex v is defined as the fraction of edges within its neighbourhood divided by the number of all possible edges between neighbours , i.e . C ( v ) : = 2| { ( vi , vj ) ∈ E | vi or vj ∈ N ( v ) } | deg ( v ) ( deg ( v ) − 1 ) . ( 2 ) 2We will follow this convention in this paper and refer to Eq . 1 as a distance . The value of C ( v ) ∈ [ 0 , 1 ] measures to what extent a vertex v forms a clique ( Watts & Strogatz , 1998 ) . The collection of all clustering coefficients of a graph can be binned and converted into a histogram in order to obtain a graph-level descriptor . This function is also easy to calculate but is inherently local ; a graph consisting of disconnected cliques or a fully-connected graph can not be distinguished , for example . Laplacian spectrum histogram . The basic idea of spectral methods involves assigning a matrix to a graph G , whose spectrum , i.e . its eigenvalues and eigenvectors , is subsequently used as a characterisation of G. Let A refer to the adjacency matrix of G , with Aij = 1 if and only if vertices vi and vj are connected by an edge in G ( since G is undirected , A is symmetric ) . The normalised graph Laplacian is defined as L : = I −D− 12AD− 12 , where I denotes the identity matrix and D refers to the degree matrix , i.e . Dii = deg ( vi ) for a vertex vi and Dij = 0 for i 6= j . The matrix L is real-valued and symmetric , so it is diagonalisable with a full set of eigenvalues and eigenvectors . Letting λ1 ≤ λ2 ≤ . . . refer to the eigenvalues of L , we have 0 ≤ λi ≤ 2 ( Chung , 1997 , Chapter 1 , Lemma 1.7 ) . This boundedness lends itself naturally to a histogram representation ( regardless of the size of G ) , making it possible to bin the eigenvalues and use the resulting histogram as a simplified descriptor of a graph . The expressivity of such a representation is not clear a priori ; the question of whether graphs are fully determined by their spectrum is still open ( van Dam & Haemers , 2003 ) and has only been partially answered in the negative for certain classes of graphs ( Schwenk , 1973 ) .

Summary: This paper describes desiderata (expressivity, robustness, and efficiency) for metrics for comparing graph generative models and details the various ways that recent work has used maximum mean discrepancy (MMD) to evaluate graph generative models.  Several limitations of MMD are described, as well as the consequences of the various choices in using MMD (kernels, descriptor functions, and hyper-parameters) and how these choices have impacted experiments in recent work.  The paper concludes with recommendations to help ameliorate several of these limitations and impacts.