Graphon based Clustering and Testing of Networks: Algorithms and Theory

1 INTRODUCTION . Machine learning on graphs has evolved considerably over the past two decades . The traditional view towards network analysis is limited to modelling interactions among entities of interest , for instance social networks or world wide web , and learning algorithms based on graph theory have been commonly used to solve these problems ( Von Luxburg , 2007 ; Yan et al. , 2006 ) . However , recent applications in bioinformatics and other disciplines require a different perspective , where the networks are the quantities of interest . For instance , it is of practical interest to classify protein structures as enzyme or non-enzyme ( Dobson & Doig , 2003 ) or detect topological changes in brain networks caused by Alzheimer ’ s disease ( Stam et al. , 2007 ) . In this paper , learning from network-valued data refers to clustering where each network is treated as an entity , as opposed to the traditional network analysis problems that involve a single network of interactions ( Newman , 2003 ) . Machine learning on network-valued data has been an active area of research in recent years , although most works focus on the network classification problem . The generic approach is to convert the network-valued data into a standard representation . Graph neural networks are commonly used for network embedding , that is , finding Euclidean representations of each network that can be further used in standard machine learning models ( Narayanan et al. , 2017 ; Xu et al. , 2019 ) . In contrast , graph kernels capture similarities between pairs of networks that can be used in kernel based learning algorithms ( Shervashidze et al. , 2011 ; Kondor & Pan , 2016 ; Togninalli et al. , 2019 ) . In particular , the graph neural tangent kernel defines a graph kernel that corresponds to infinitely wide graph neural networks , and typically outperforms neural networks in classification tasks ( Du et al. , 2019 ) . A more classical equivalent for graph kernels is to define metrics that characterise the distances between pairs of graphs ( Bunke & Shearer , 1998 ) , but there has been limited research on designing efficient graph distances and developing algorithms for clustering network-valued data . The motivation for this paper stems from two shortcomings in the literature on network-valued data analysis : first , the efficacy of existing kernels or embeddings have not been studied beyond network classification , and second is the lack of theoretical analysis of these methods , particularly in the small sample setting . Generalisation error bounds for graph kernel based learning exist ( Du et al. , 2019 ) , but these bounds , based on learning theory , are meaningful only when many networks are available . However , in many applications , one needs to learn from a small population of large networks and , in such cases , an informative statistical analysis should consider the small sample , large graph regime . To address this issue , we take inspiration from the recent statistics literature on graph two-sample testing—given two ( populations of ) large graphs , the goal is to decide if they are from same statistical model or not . Although most theoretical studies in graph two-sample testing focus on graph with vertex correspondence ( Tang et al. , 2017a ; Ghoshdastidar & von Luxburg , 2018 ) , some works address the problem of testing graphs on different vertex sets either by defining distances between graphs ( Tang et al. , 2017b ; Agterberg et al. , 2020 ) or by representing networks in terms of pre-specified network statistics ( Ghoshdastidar et al. , 2017 ) . The use of network statistics for clustering network-valued data is studied in Mukherjee et al . ( 2017 ) . Another fundamental approach for dealing with graphs of different sizes is graph matching , where the objective is to determine the vertex correspondence . Graph matching is often solved by formulating it as an optimization problem ( Zaslavskiy et al. , 2008 ; Guo et al. , 2019 ) or defining graph edit distance between the graphs ( Riesen & Bunke , 2009 ; Gao et al. , 2010 ) . Although there is extensive research on graph matching , the efficacy of these methods in clustering network-valued data remains unexplored . Contribution and organisation . In this work , we follow the approach of defining meaningful graph distances based on statistical models , and use the proposed graph distance in the context of learning from networks without vertex correspondence . In particular , we propose graph distances based on graphons . Graphons are symmetric bivariate functions that represent the limiting structure for a sequence of graphs with increasing number of nodes ( Lovász & Szegedy , 2006 ) , but can be also viewed as a nonparametric statistical model for exchangeable random graphs ( Diaconis & Janson , 2007 ; Bickel & Chen , 2009 ) . The latter perspective is useful for the purpose of machine learning since it allows us to view the multiple graphs as random samples drawn from one or more graphon models . This perspective forms the basis of our contributions , which are listed below : 1 ) In Section 2 , we propose a distance between two networks , that do not have vertex correspondence and could have different number of vertices . We view the networks as random samples from ( unknown ) graphons , and propose a graph distance that estimates theL2-distance between the graphons . The distance is inspired by the sorting-and-smoothing graphon estimator ( Chan & Airoldi , 2014 ) . 2 ) In Section 3 , we present two algorithms for clustering network-valued data based on the proposed graph distance : a distance-based spectral clustering algorithm , and a similarity based semi-definite programming ( SDP ) approach . We derive performance guarantees for both algorithms under the assumption that the networks are sampled from graphons satisfying certain smoothness conditions . 3 ) We empirically compare the performance of our algorithms with other clustering strategies based on graph kernels , graph matching , network statistics etc . and show that , on both simulated and real data , our graph distance-based spectral clustering algorithm outperforms others while the SDP approach also shows reasonable performance , and they also scale to large networks ( Section 3.3 ) . 4 ) Inspired by the success of the proposed graph distance in clustering , we use the distance for graph two-sample testing . In Section 4 , we show that the proposed two-sample test is statistically consistent for large graphs , and also demonstrate the efficacy of the test through numerical simulation . We provide further discussion in Section 5 and present the proofs of theoretical results in Appendix . 2 GRAPH DISTANCE BASED ON GRAPHONS . Clustering or testing of multiple networks requires a notion of distance between the networks . In this section , we present a transformation that converts graphs of different sizes into a fixed size representation , and subsequently , propose a graph distance inspired by the theory of graphons . We first provide some background on graphons and graphon estimation . Graphon has been studied in the literature from two perspectives : as limiting structure for infinite sequence of growing graphs ( Lovász & Szegedy , 2006 ) , or as exchangeable random graph model . In this paper , we follow the latter perspective . A random graph is said to be exchangeable if its distribution is invariant under permutation of nodes . Diaconis & Janson ( 2007 ) showed that any statistical model that generates exchangeable random graphs can be characterised by graphons , as introduced by Lovász & Szegedy ( 2006 ) . Formally , a graphon is a symmetric measurable continuous function w : [ 0 , 1 ] 2 → [ 0 , 1 ] where w ( x , y ) can be interpreted as the link probability between two nodes of the graph that are assigned values x and y , respectively . This interpretation propounds the following two stage sampling procedure for graphons . To sample a random graph G with n nodes from a graphon w , in the first stage , one samples n variables U1 , . . . , Un uniformly from [ 0 , 1 ] and constructs a latent mapping between the sampled points and the node labels . In the second stage , edges between any two nodes i , j are randomly added based on the link probability w ( Ui , Uj ) . Mathematically , if we abuse notation to denote the adjacency matrix by G ∈ { 0 , 1 } n×n , we have U1 , . . . , Un iid∼ Uniform [ 0 , 1 ] and Gij |Ui , Uj ∼ Bernoulli ( w ( Ui , Uj ) ) for all i < j . We consider problems involving multiple networks sampled independently from the same ( or different ) graphons . We make the following smoothness assumptions on the graphons . Assumption 1 ( Lipschitz continuous ) A graphon w is Lipschitz continuous with constant L if |w ( u , v ) − w ( u′ , v′ ) | ≤ L √ ( u− u′ ) 2 + ( v − v′ ) 2 for every u , v , u′ , v′ ∈ [ 0 , 1 ] . Assumption 2 ( Two-sided Lipschitz degree ) A graphon w has two-sided Lipschitz degree with constants λ1 , λ2 > 0 if its expected degree function g , defined by g ( u ) = ∫ 1 0 w ( u , v ) dv , satisfies λ2|u− u′| ≤ |g ( u ) − g ( u′ ) | ≤ λ1|u− u′| for every u , u′ ∈ [ 0 , 1 ] . One of the challenges in graphon estimation is due to the issue of non-identifiability , that is , different graphon functions w can generate the same random graph model . In particular , two graphons w and w′ generate the same random graph model if they are weakly isomorphic—there exist two measure preserving transformations φ , φ′ : [ 0 , 1 ] → [ 0 , 1 ] such that w ( φ ( u ) , φ ( v ) ) = w′ ( φ′ ( u ) , φ′ ( v ) ) . Moreover , the converse also holds meaning that such transformations are known to be the only source of non-identifiability ( Diaconis & Janson , 2007 ) . This weak isomorphism induces equivalence classes on the space of graphons . Since our goal is only to cluster graphs belonging to random graph models , we simply make the following assumption on our graphons . Assumption 3 ( Equivalence classes ) Any reference toK graphons , w1 , . . . , wK , assumes that , for every i , j , either wi = wj or wi and wj belong to different equivalence classes . Furthermore , without loss of generality , we assume that every graphon wi is represented such that the corresponding degree function gi is non-decreasing . Remark on the necessity of Assumptions 1–3 . Assumption 1 is standard in graphon estimation literature ( Klopp et al. , 2017 ) since it avoids graphons corresponding to inhomogeneous random graph models . It is known that two graphs from widely separated inhomogeneous models ( in L2distance ) are statistically indistinguishable ( Ghoshdastidar et al. , 2020 ) , and hence , it is essential to ignore such models to derive meaningful guarantees . Assumption 2 ensures that , under a measurepreserving transformation , the graphon has strictly increasing degree function , which is a canonical representation of an equivalence class of graphons ( Bickel & Chen , 2009 ) . Assumption 3 is needed since graphons can only be estimated up to measure-preserving transformation . As noted above , it is inconsequential for all practical purposes but simplifies the theoretical exposition . Graph transformation . In order to deal with multiple graphs and measure distances among pairs of graphs , we require a transformation that maps all graphs into a common metric space—the space of all n0 × n0 symmetric matrices for some integer n0 . While the graphon estimation literature provides several consistent estimators ( Klopp et al. , 2017 ; Zhang et al. , 2017 ) , only the histogram based sorting-and-smoothing graphon estimator of Chan & Airoldi ( 2014 ) can be adapted to meet the above requirement . We use the following graph transformation , inspired by Chan & Airoldi ( 2014 ) . The adjacency matrix G of size n× n is first reordered based on a non-unique permutation σ , such that the empirical degree based on this permutation is monotonically increasing . The degree sorted adjacency matrix is denoted by Gσ . It is then transformed to a ‘ histogram ’ A ∈ Rn0×n0 as Aij = 1 h2 h∑ i1=1 h∑ j1=1 Gσ ( i−1 ) h+i1 , ( j−1 ) h+j1 , where h = ⌊ n n0 ⌋ and b·c is the floor function . ( 1 ) Proposed graph distance . Given two directed or undirected graphs G1 and G2 with n1 and n2 nodes , respectively , we apply the transformation ( 1 ) to both the graphs with n0 ≤ min { n1 , n2 } . We propose to use the graph distance d ( G1 , G2 ) = 1 n0 ‖A1 −A2‖F , ( 2 ) where A1 and A2 denote the transformed matrices and ‖ · ‖F denotes the matrix Frobenius norm . Proposition 1 shows that , if G1 and G2 are sampled from two graphons , then the graph distance ( 2 ) consistently estimates the L2-distance between the two graphons , which is defined as ‖w1 − w2‖2L2 = ∫ 1 0 ∫ 1 0 ( w1 ( x , y ) − w2 ( x , y ) ) 2 dx dy . Proposition 1 ( Graph distance is consistent ) Let w1 and w2 satisfy Assumptions 1–3 . Let G1 and G2 be random graphs with at least n nodes sampled from the graphons w1 and w2 , respectively . If n→∞ and n0 is chosen such that n 2 0 logn n → 0 , then with high probability ( w.h.p . ) , ∣∣‖w1 − w2 ‖L2 − d ( G1 , G2 ) ∣∣ = O ( 1n0 ) . ( 3 ) Proof sketch . We define a novel technique for approximating the graphon . The proof in Appendix A.1 first establishes that the approximation error is bounded using Assumption 1 . Consequently , a relation between approximated graphons and transformed graphs is derived using lemmas from Chan & Airoldi ( 2014 ) . Proposition 1 is subsequently proved using the above two results . Remark on Proposition 1 for sparse graphs . The defined sampling procedure for graphon generates dense graphs , deviating from the real world sparse networks . To adapt it to sparse graphs , one may modify the sampling procedure to Gij |Ui , Uj ∼ Bernoulli ( ρw ( Ui , Uj ) ) where ρ depends only on n ( Olhede & Wolfe , 2014 ) . Under this process , the consistency result in Proposition 1 remains unchanged for ρ = Ω ( √ log n/n ) ( proof in Appendix A.2 ) . This bound can not be improved to the expected real world measure where ρ = Ω ( log n/n ) , because of the degree sorting step in ( 1 ) . Nevertheless , our analysis allows for relatively sparser graphs with strong consistency result . Notation . For ease of exposition , Proposition 1 as well as main results are stated asymptotically using the standard O ( · ) and Ω ( · ) notations , which subsume absolute and Lipschitz constants . We use “ with high probability ” ( w.h.p . ) to state that the probability of an event converges to 1 as n→∞ .

Summary: Motivated by the sort-and-smooth graphon estimator of Chan and Airoldi, 2014, this paper proposes two new clustering algorithms (graph distance based spectral clustering and similarity based semidefinite programming) for multiple graphs observed without vertex correspondence. The idea is to use the graphon approximation method to obtain a histogram estimator with the same number of bins for each graph and subsequently apply an existing spectral clustering approach on the graphon based distance matrix. Theoretical properties of their approach are studied (under reasonable smoothness assumptions on the generating process i.e. graphon) and consistency is established. The graph distance metric is applied to test for similarity of two collections of graphs.