Understanding Isomorphism Bias in Graph Data Sets 

1 INTRODUCTION . Recently there has been an increasing interest in the development of machine learning models that operate on graph structured data . Such models have found applications in chemoinformatics ( Ralaivola et al . ( 2005 ) ; Rupp & Schneider ( 2010 ) ; Ferré et al . ( 2017 ) ) and bioinformatics ( Borgwardt et al . ( 2005 ) ; Kundu et al . ( 2013 ) ) , neuroscience ( Sharaev et al . ( 2018 ) ; Jie et al . ( 2016 ) ; Wang et al . ( 2016 ) ) , computer vision ( Stumm et al . ( 2016 ) ) and system security ( Li et al . ( 2016 ) ) , natural language processing ( Glavaš & Šnajder ( 2013 ) ) , and others ( Kriege et al . ( 2019 ) ; Nikolentzos et al . ( 2019 ) ) . One of the popular tasks that encompasses these applications is graph classification problem for which many graph kernels and graph neural networks have been developed . One of the implicit assumptions that many practitioners adhere to is that models that can distinguish isomorphic instances from non-isomorphic ones possess higher expressiveness in classification problem and hence much efforts have been devoted to incorporate efficient graph isomorphism methods into the classification models . As the problem of computing complete graph invariant is GI-hard ( Gärtner et al . ( 2003 ) ) , for which no known polynomial-time algorithm exists , other heuristics have been proposed as a proxy for deciding whether two graphs are isomorphic . Indeed , from the early days topological descriptors such Wiener index ( Wiener ( 1947a ; b ) ) attempted to find a single number that uniquely identifies a graph . Later , graph kernels that model pairwise similarities between graphs utilized theoretical developments in graph isomorphism literature . For example , graphlet kernel ( Shervashidze et al . ( 2009 ) ) is based on the Kelly conjecture ( see also Kelly ( 1957 ) ) , anonymous walk kernel ( Ivanov & Burnaev ( 2018 ) ) derives insights from the reconstruction properties of anonymous experiments ( see also Micali & Allen Zhu ( 2016 ) ) , and WL kernel ( Shervashidze et al . ( 2011a ) ) is based on an efficient graph isomorphism algorithm . For sufficiently large k , kdimensional WL algorithm includes all combinatorial properties of a graph ( Cai et al . ( 1992a ) ) , so one may hope its power is enough for the data set at hand . Since only for k = Ω ( n ) WL algorithm is guaranteed to distinguish all graphs ( for which the running time becomes exponential ; see also Fürer ( 2017 ) ) , in the general case WL algorithm can be used only as a strong baseline for graph isomorphism . In similar fashion , graph neural networks exploit graph isomorphism algorithms and have been shown to be as powerful as k-dimensional WL algorithm ( see for example Maron et al . ( 2019 ) ; Xu et al . ( 2018 ) ; Morris et al . ( 2019 ) ) . Experimental evaluation reveals that models based on the theoretical constructions with high combinatorial power such as WL algorithm performs better than the models without them such as Vertex histogram kernel ( Vishwanathan et al . ( 2010 ) ) on a commonly used data sets . This could add additional bias to results of comparison of classification algorithms since the models could simply apply a graph isomorphism method ( or an efficient approximation ) to determine a target label at the inference time . However , purely judging on the accuracy of the algorithms in such cases would imply an unfair comparison between the methods as it does not measure correctly generalization ability of the models on the new test instances . As we discover , indeed many of the data sets used in graph classification have isomorphic instances so much that in some of them the fraction of the unique non-repeating graphs is as low as 20 % of the total size . This challenges previous experimental results and requires understanding of how influential isomorphic instances on the final performance of the models . Our contributions are : • We analyze the quality of 54 graph data sets which are used ubiquitously in graph classification comparison . Our findings suggest that in the most of the data sets there are isomorphic graphs and their proportion varies from as much as 100 % to 0 % . Surprisingly , we also found that there are isomorphic instances that have different target labels suggesting they are not suitable for learning a classifier at all . • We investigate the causes of isomorphic graphs and show that node and edge labels are important to identify isomorphic graphs . Other causes include numerical attributes of nodes and edges as well as the sizes of the data set . • We express an upper bound for the generalization gap through the Radamacher complexity of a classifier and the number of isomorphic graphs in a data set . This bound presents theoretical evidence on how weightning of each graph in the training influences classification accuracy . • We evaluate a classification model ’ s performance on isomorphic instances and show that even strong models do not achieve optimal accuracy even if the instances have been seen at the training time . Hence we show a model-agnostic way to artificially increase performance on several widely used data sets . • We open-source new cleaned data sets that contain only non-isomorphic instances with no noisy target labels . We give a set of recommendations regarding applying new models that work with graph structured data . 2 RELATED WORK . Measuring quality of data sets . A similar issue of duplicates instances in commonly used data sets was recently discovered in computer vision domain . Recht et al . ( 2019 ) ; Barz & Denzler ( 2019 ) ; Birodkar et al . ( 2019 ) discover that image data sets CIFAR and ImageNet contain at least 10 % of the duplicate images in the test test invalidating previous performance and questioning generalization abilities of previous successful architectures . In particular , evaluating the models in new test sets shows a drop of accuracy by as much as 15 % ( Recht et al. , 2019 ) , which is explained by models ’ incapability to generalize to unseen slightly ” harder ” instances than in the original test sets . In graph domain , a fresh look into understanding of expressiveness of graph kernels and the quality of data sets has been considered in Kriege et al . ( 2019 ) , where an extensive comparison of existing graph kernels is done and a few insights about models ’ behavior are suggested . In contrast , we conduct a broader study of isomorphism metrics , revealing all isomorphism pairs in proposed 54 data sets , and propose new cleaned data . Additionally we also consider graph neural network performance and argue that current data sets present isomorphism bias which can artificially boost evaluation metrics in a model-agnostic way . Explaining performance of graph models . Graph kernels ( Kriege et al . ( 2019 ) ) and graph neural networks ( Wu et al . ( 2019 ) ) are two competing paradigms for designing graph representations and solving graph classification and have significantly advanced empirical results due to more efficient algorithms , incorporating graph invariance into the models , and end-to-end training . Several papers have tried to justify performance of different families of methods by studying different statistical properties . For example , in Ying et al . ( 2019 ) by maximizing mutual information between explanation variables and predicted label distribution , the model is trained to return a small subgraph and the graph-specific attributes that are the most influential on the decision made by a GNN , which allows inspection of single- and multi-level predictions in an agnostic manner for GNNs . In another work ( Scarselli et al . ( 2018 ) ) , the VC dimension of GNNs models has been shown to grow as O ( p4N2 ) , where p is the number of network parameters and N is the number of nodes in a graph , which is comparable to RNN models . Furthermore , stability and generalization properties of convolutional GNNs have been shown to depend on the largest eigenvalue of the graph filter and therefore are attained for properly normalized graph convolutions such as symmetric normalized graph Laplacian ( Verma & Zhang ( 2019 ) ) . Finally , expressivity of graph kernels has been studied from statistical learning theory ( Oneto et al . ( 2017b ) ) and property testing ( Kriege et al . ( 2018b ) ) , showing that graph kernels can capture certain graph properties such as planarity , girth , and chromatic number ( Johansson et al . ( 2014 ) ) . Our approach is complementary to all of the above as we analyze if the data sets used in experiments have any effect on the final performance . 3 PRELIMINARIES . In this work we analyze 54 graph data sets from Kersting et al . ( 2016 ) that are commonly used in graph classification task . Examples of popular graph data sets are presented in Table 1 and statistics of all 54 data sets can be found in Table 5 , see Section A in the appendix . All data sets represent a collection of graphs and accompanying categorical label for each graph in the data sets . Some data sets also include node and/or edge labels that graph classification methods can use to improve the scoring . Most of the data sets come either from biological domain or from social network domain . Biological data sets such as MUTAG , ENZYMES , PROTEINS are graphs that represent small or large molecules , where edges of the graphs are chemical bonds or spatial proximity between different atoms . Graph labels in these cases encode different properties like toxicity . In social data sets such as IMDB-BINARY , REDDIT-MULTI-5K , COLLAB the nodes represent people and edges are relationships in movies , discussion threads , or citation network respectively . Labels in these cases denote the type of interaction like the genre of the movie/thread or a research subfield . For completeness we also include synthetic data sets SYNTHETIC ( Morris et al . ( 2016 ) ) that have continuous attributes and computer vision data sets MSRC ( Neumann et al . ( 2016 ) ) , where images are encoded as graphs . The origin of all data sets can be found in the Table 5 . Graph isomorphism . Isomorphism between two graphs G1 = ( V1 , E1 ) and G2 = ( V2 , E2 ) is a bijective function φ : V1 7→ V2 such that any edge ( u , v ) ∈ E1 if and only if ( φ ( u ) , φ ( v ) ) ∈ E2 . Graph isomorphism problem asks if such function exists for given two graphs G1 and G2 . We denote isomorphic graphs asG1 ∼= G2 . The problem has efficient algorithms in P for certain classes of graphs such as planar or bounded-degree graphs ( Hopcroft & Wong ( 1974 ) ; Luks ( 1980 ) ) , but in the general case admits only quasi-polynomial algorithm ( Babai ( 2015 ) ) . In practice many GI solvers are based on individualization-refinement paradigm ( Mckay & Piperno ( 2014 ) ) , which for each graph iteratively updates a permutation of the nodes such that the resulted permutations of two graphs are identical if an only if they are isomorphic . Importantly , while finding such canonical permutation of a graph is at least as hard as solving GI problem , state-of-the-art solvers tackle majority of pairs of graphs efficiently , only taking exponential time on the specific hard instances of graphs that possess highly symmetrical structures ( Cai et al . ( 1992b ) ) .

Summary: This work probes graph classification datasets for isomorphism bias. They find substantial amount of bias in some datasets and show that they suffer from data leakage. They further perform a more fine-grained evaluation taking into consideration the node/edge types which reduce the perceived effects. They also provide some recommendations for measuring the 'right metrics' and release clean versions of the considered datasets.