Graphon based Clustering and Testing of Networks: Algorithms and Theory

1 INTRODUCTION . Machine learning on graphs has evolved considerably over the past two decades . The traditional view towards network analysis is limited to modelling interactions among entities of interest , for instance social networks or world wide web , and learning algorithms based on graph theory have been commonly used to solve these problems ( Von Luxburg , 2007 ; Yan et al. , 2006 ) . However , recent applications in bioinformatics and other disciplines require a different perspective , where the networks are the quantities of interest . For instance , it is of practical interest to classify protein structures as enzyme or non-enzyme ( Dobson & Doig , 2003 ) or detect topological changes in brain networks caused by Alzheimer ’ s disease ( Stam et al. , 2007 ) . In this paper , learning from network-valued data refers to clustering where each network is treated as an entity , as opposed to the traditional network analysis problems that involve a single network of interactions ( Newman , 2003 ) . Machine learning on network-valued data has been an active area of research in recent years , although most works focus on the network classification problem . The generic approach is to convert the network-valued data into a standard representation . Graph neural networks are commonly used for network embedding , that is , finding Euclidean representations of each network that can be further used in standard machine learning models ( Narayanan et al. , 2017 ; Xu et al. , 2019 ) . In contrast , graph kernels capture similarities between pairs of networks that can be used in kernel based learning algorithms ( Shervashidze et al. , 2011 ; Kondor & Pan , 2016 ; Togninalli et al. , 2019 ) . In particular , the graph neural tangent kernel defines a graph kernel that corresponds to infinitely wide graph neural networks , and typically outperforms neural networks in classification tasks ( Du et al. , 2019 ) . A more classical equivalent for graph kernels is to define metrics that characterise the distances between pairs of graphs ( Bunke & Shearer , 1998 ) , but there has been limited research on designing efficient graph distances and developing algorithms for clustering network-valued data . The motivation for this paper stems from two shortcomings in the literature on network-valued data analysis : first , the efficacy of existing kernels or embeddings have not been studied beyond network classification , and second is the lack of theoretical analysis of these methods , particularly in the small sample setting . Generalisation error bounds for graph kernel based learning exist ( Du et al. , 2019 ) , but these bounds , based on learning theory , are meaningful only when many networks are available . However , in many applications , one needs to learn from a small population of large networks and , in such cases , an informative statistical analysis should consider the small sample , large graph regime . To address this issue , we take inspiration from the recent statistics literature on graph two-sample testing—given two ( populations of ) large graphs , the goal is to decide if they are from same statistical model or not . Although most theoretical studies in graph two-sample testing focus on graph with vertex correspondence ( Tang et al. , 2017a ; Ghoshdastidar & von Luxburg , 2018 ) , some works address the problem of testing graphs on different vertex sets either by defining distances between graphs ( Tang et al. , 2017b ; Agterberg et al. , 2020 ) or by representing networks in terms of pre-specified network statistics ( Ghoshdastidar et al. , 2017 ) . The use of network statistics for clustering network-valued data is studied in Mukherjee et al . ( 2017 ) . Another fundamental approach for dealing with graphs of different sizes is graph matching , where the objective is to determine the vertex correspondence . Graph matching is often solved by formulating it as an optimization problem ( Zaslavskiy et al. , 2008 ; Guo et al. , 2019 ) or defining graph edit distance between the graphs ( Riesen & Bunke , 2009 ; Gao et al. , 2010 ) . Although there is extensive research on graph matching , the efficacy of these methods in clustering network-valued data remains unexplored . Contribution and organisation . In this work , we follow the approach of defining meaningful graph distances based on statistical models , and use the proposed graph distance in the context of learning from networks without vertex correspondence . In particular , we propose graph distances based on graphons . Graphons are symmetric bivariate functions that represent the limiting structure for a sequence of graphs with increasing number of nodes ( Lovász & Szegedy , 2006 ) , but can be also viewed as a nonparametric statistical model for exchangeable random graphs ( Diaconis & Janson , 2007 ; Bickel & Chen , 2009 ) . The latter perspective is useful for the purpose of machine learning since it allows us to view the multiple graphs as random samples drawn from one or more graphon models . This perspective forms the basis of our contributions , which are listed below : 1 ) In Section 2 , we propose a distance between two networks , that do not have vertex correspondence and could have different number of vertices . We view the networks as random samples from ( unknown ) graphons , and propose a graph distance that estimates theL2-distance between the graphons . The distance is inspired by the sorting-and-smoothing graphon estimator ( Chan & Airoldi , 2014 ) . 2 ) In Section 3 , we present two algorithms for clustering network-valued data based on the proposed graph distance : a distance-based spectral clustering algorithm , and a similarity based semi-definite programming ( SDP ) approach . We derive performance guarantees for both algorithms under the assumption that the networks are sampled from graphons satisfying certain smoothness conditions . 3 ) We empirically compare the performance of our algorithms with other clustering strategies based on graph kernels , graph matching , network statistics etc . and show that , on both simulated and real data , our graph distance-based spectral clustering algorithm outperforms others while the SDP approach also shows reasonable performance , and they also scale to large networks ( Section 3.3 ) . 4 ) Inspired by the success of the proposed graph distance in clustering , we use the distance for graph two-sample testing . In Section 4 , we show that the proposed two-sample test is statistically consistent for large graphs , and also demonstrate the efficacy of the test through numerical simulation . We provide further discussion in Section 5 and present the proofs of theoretical results in Appendix . 2 GRAPH DISTANCE BASED ON GRAPHONS . Clustering or testing of multiple networks requires a notion of distance between the networks . In this section , we present a transformation that converts graphs of different sizes into a fixed size representation , and subsequently , propose a graph distance inspired by the theory of graphons . We first provide some background on graphons and graphon estimation . Graphon has been studied in the literature from two perspectives : as limiting structure for infinite sequence of growing graphs ( Lovász & Szegedy , 2006 ) , or as exchangeable random graph model . In this paper , we follow the latter perspective . A random graph is said to be exchangeable if its distribution is invariant under permutation of nodes . Diaconis & Janson ( 2007 ) showed that any statistical model that generates exchangeable random graphs can be characterised by graphons , as introduced by Lovász & Szegedy ( 2006 ) . Formally , a graphon is a symmetric measurable continuous function w : [ 0 , 1 ] 2 → [ 0 , 1 ] where w ( x , y ) can be interpreted as the link probability between two nodes of the graph that are assigned values x and y , respectively . This interpretation propounds the following two stage sampling procedure for graphons . To sample a random graph G with n nodes from a graphon w , in the first stage , one samples n variables U1 , . . . , Un uniformly from [ 0 , 1 ] and constructs a latent mapping between the sampled points and the node labels . In the second stage , edges between any two nodes i , j are randomly added based on the link probability w ( Ui , Uj ) . Mathematically , if we abuse notation to denote the adjacency matrix by G ∈ { 0 , 1 } n×n , we have U1 , . . . , Un iid∼ Uniform [ 0 , 1 ] and Gij |Ui , Uj ∼ Bernoulli ( w ( Ui , Uj ) ) for all i < j . We consider problems involving multiple networks sampled independently from the same ( or different ) graphons . We make the following smoothness assumptions on the graphons . Assumption 1 ( Lipschitz continuous ) A graphon w is Lipschitz continuous with constant L if |w ( u , v ) − w ( u′ , v′ ) | ≤ L √ ( u− u′ ) 2 + ( v − v′ ) 2 for every u , v , u′ , v′ ∈ [ 0 , 1 ] . Assumption 2 ( Two-sided Lipschitz degree ) A graphon w has two-sided Lipschitz degree with constants λ1 , λ2 > 0 if its expected degree function g , defined by g ( u ) = ∫ 1 0 w ( u , v ) dv , satisfies λ2|u− u′| ≤ |g ( u ) − g ( u′ ) | ≤ λ1|u− u′| for every u , u′ ∈ [ 0 , 1 ] . One of the challenges in graphon estimation is due to the issue of non-identifiability , that is , different graphon functions w can generate the same random graph model . In particular , two graphons w and w′ generate the same random graph model if they are weakly isomorphic—there exist two measure preserving transformations φ , φ′ : [ 0 , 1 ] → [ 0 , 1 ] such that w ( φ ( u ) , φ ( v ) ) = w′ ( φ′ ( u ) , φ′ ( v ) ) . Moreover , the converse also holds meaning that such transformations are known to be the only source of non-identifiability ( Diaconis & Janson , 2007 ) . This weak isomorphism induces equivalence classes on the space of graphons . Since our goal is only to cluster graphs belonging to random graph models , we simply make the following assumption on our graphons . Assumption 3 ( Equivalence classes ) Any reference toK graphons , w1 , . . . , wK , assumes that , for every i , j , either wi = wj or wi and wj belong to different equivalence classes . Furthermore , without loss of generality , we assume that every graphon wi is represented such that the corresponding degree function gi is non-decreasing . Remark on the necessity of Assumptions 1–3 . Assumption 1 is standard in graphon estimation literature ( Klopp et al. , 2017 ) since it avoids graphons corresponding to inhomogeneous random graph models . It is known that two graphs from widely separated inhomogeneous models ( in L2distance ) are statistically indistinguishable ( Ghoshdastidar et al. , 2020 ) , and hence , it is essential to ignore such models to derive meaningful guarantees . Assumption 2 ensures that , under a measurepreserving transformation , the graphon has strictly increasing degree function , which is a canonical representation of an equivalence class of graphons ( Bickel & Chen , 2009 ) . Assumption 3 is needed since graphons can only be estimated up to measure-preserving transformation . As noted above , it is inconsequential for all practical purposes but simplifies the theoretical exposition . Graph transformation . In order to deal with multiple graphs and measure distances among pairs of graphs , we require a transformation that maps all graphs into a common metric space—the space of all n0 × n0 symmetric matrices for some integer n0 . While the graphon estimation literature provides several consistent estimators ( Klopp et al. , 2017 ; Zhang et al. , 2017 ) , only the histogram based sorting-and-smoothing graphon estimator of Chan & Airoldi ( 2014 ) can be adapted to meet the above requirement . We use the following graph transformation , inspired by Chan & Airoldi ( 2014 ) . The adjacency matrix G of size n× n is first reordered based on a non-unique permutation σ , such that the empirical degree based on this permutation is monotonically increasing . The degree sorted adjacency matrix is denoted by Gσ . It is then transformed to a ‘ histogram ’ A ∈ Rn0×n0 as Aij = 1 h2 h∑ i1=1 h∑ j1=1 Gσ ( i−1 ) h+i1 , ( j−1 ) h+j1 , where h = ⌊ n n0 ⌋ and b·c is the floor function . ( 1 ) Proposed graph distance . Given two directed or undirected graphs G1 and G2 with n1 and n2 nodes , respectively , we apply the transformation ( 1 ) to both the graphs with n0 ≤ min { n1 , n2 } . We propose to use the graph distance d ( G1 , G2 ) = 1 n0 ‖A1 −A2‖F , ( 2 ) where A1 and A2 denote the transformed matrices and ‖ · ‖F denotes the matrix Frobenius norm . Proposition 1 shows that , if G1 and G2 are sampled from two graphons , then the graph distance ( 2 ) consistently estimates the L2-distance between the two graphons , which is defined as ‖w1 − w2‖2L2 = ∫ 1 0 ∫ 1 0 ( w1 ( x , y ) − w2 ( x , y ) ) 2 dx dy . Proposition 1 ( Graph distance is consistent ) Let w1 and w2 satisfy Assumptions 1–3 . Let G1 and G2 be random graphs with at least n nodes sampled from the graphons w1 and w2 , respectively . If n→∞ and n0 is chosen such that n 2 0 logn n → 0 , then with high probability ( w.h.p . ) , ∣∣‖w1 − w2 ‖L2 − d ( G1 , G2 ) ∣∣ = O ( 1n0 ) . ( 3 ) Proof sketch . We define a novel technique for approximating the graphon . The proof in Appendix A.1 first establishes that the approximation error is bounded using Assumption 1 . Consequently , a relation between approximated graphons and transformed graphs is derived using lemmas from Chan & Airoldi ( 2014 ) . Proposition 1 is subsequently proved using the above two results . Remark on Proposition 1 for sparse graphs . The defined sampling procedure for graphon generates dense graphs , deviating from the real world sparse networks . To adapt it to sparse graphs , one may modify the sampling procedure to Gij |Ui , Uj ∼ Bernoulli ( ρw ( Ui , Uj ) ) where ρ depends only on n ( Olhede & Wolfe , 2014 ) . Under this process , the consistency result in Proposition 1 remains unchanged for ρ = Ω ( √ log n/n ) ( proof in Appendix A.2 ) . This bound can not be improved to the expected real world measure where ρ = Ω ( log n/n ) , because of the degree sorting step in ( 1 ) . Nevertheless , our analysis allows for relatively sparser graphs with strong consistency result . Notation . For ease of exposition , Proposition 1 as well as main results are stated asymptotically using the standard O ( · ) and Ω ( · ) notations , which subsume absolute and Lipschitz constants . We use “ with high probability ” ( w.h.p . ) to state that the probability of an event converges to 1 as n→∞ .

Summary: The paper proposes a graph distance based on graphons that can operate on the small sample size regime; this is possible by exploiting a large number of nodes. The proposed paper is meant to address two shortcomings that the authors have found in the literature:  - a lack of study of current graph processing methods beyond graph-level classification, and  - a lack of theoretically sound methods.    The paper also shows theoretical guarantees associated with two existing clustering methods and a two-sample statistical tests when operating with the proposed distance.