Sharp Impossibility Results for Hyper-graph Testing

1 Introduction . The hypergraph is a useful representation of social relationships beyond pairwise interactions [ 5 , 11 ] . For example , the co-authorship hypergraph is often used to analyze the co-authorship topology of authors , and it provides more information than a co-authorship graph ( where an m-author paper is treated as an m-clique ) . The community detection on a hypergraph [ 10 ] is a problem of great interest ( communities in a hypergraph are clusters of nodes that have more hyperedges within than across ) . It has many applications in social network analysis [ 15 ] and machine learning [ 1 , 17 , 18 , 23 ] . We are interested in the problem of global testing , where we test whether the hypergraph has one community or multiple communities . It has applications in measuring co-authorship and citation diversity [ 12 ] and discovering non-obvious social groups and patterns [ 4 ] . It is also useful for understanding other problems such as community detection and change-point detection in dynamic hypergraphs . For instructional purpose only , we start with the 3-uniform hypergraphs ( i.e. , each hyperedge consists of 3 nodes ) , but our results cover both higher-order hypergraphs and non-uniform hypergraphs . Let A be the adjacency tensor of a uniform and symmetric order-3 hypergraph with n nodes , where Ai1i2i3 equals 1 if i1 , i2 , i3 share a hyperedge and equals 0 otherwise , ( 1.1 ) for 1 ≤ i1 , i2 , i3 ( distinct ) ≤ n. Since the tensor is symmetric , Ai1i2i3 = Aj1j2j3 for two sets of indices { i1 , i2 , i3 } and { j1 , j2 , j3 } if one is a permutation of the other . We do not consider self hyperedges , hence , Ai1i2i3 = 0 whenever i1 , i2 , i3 are non-distinct . Real world hypergraphs have several noteworthy features . First , there may be severe degree heterogeneity ( i.e. , the degree of one node is many times higher than that of another ) . Second , the overall sparsity levels may vary significantly from one hypergraph to another . Last , a node may have mixed-memberships across multiple communities ( i.e. , nonzero weights on more than one community ) . To accommodate these features , we adopt the Degree-Corrected Mixed-Membership ( tensor-DCMM ) model . The notations below are frequently used in tensor analysis . Definition 1.1 . ( matricization , slicing , and slice aggregation ) . Let A be a 3-symmetric tensor of ndimension . First , we call the n×n2 matrix A the matricization of A , defined by Ai , j+n ( k−1 ) = Aijk , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1 ≤ i , j , k ≤ n. Second , for 1 ≤ k ≤ n , we use A : :k to denote the n× n matrix whose row-i-andcolumn-j is Aijk , 1 ≤ i , j ≤ n , and call it the k-th slice of A . Last , for any n× 1 vector x , we use ( Ax ) to denote the matrix ∑n k=1 xkA : :k , which is an aggregation of the slices . Now , suppose the tensor has K perceivable communities . Let P be a symmetric ( 3-uniform ) tensor of K-dimension that models the community structure , let θ = ( θ1 , θ2 , . . . , θn ) ′ be positive parameters that model the degree heterogeneity of nodes , and π1 , π2 , . . . , πn be K-dimensional membership vectors where πi ( k ) = the weight node i puts on community k , 1 ≤ k ≤ K. We assume for all 1 ≤ i1 , i2 , i3 ≤ n that are three distinct indicies , Ai1i2i3 are independent Bernoulli random variables with P ( Ai1i2i3 = 1 ) = θi1θi2θi3 ∑K k1 , k2 , k3=1 Pk1 , k2 , k3πi1 ( k1 ) πi2 ( k2 ) πi3 ( k3 ) , where by Definition 1.1 , the right hand side equals to θi1θi2θi3π ′ i1 ( Pπi3 ) πi2 . Introduce a non-stochastic 3-uniform tensor Q of n-dimension where Qi1i2i3 = θi1θi2θi3π′i1 ( Pπi3 ) πi2 for 1 ≤ i1 , i2 , i3 ≤ n. Let diag ( Q ) be the tensor with the same size of Q where ( diag ( Q ) ) i1i2i3 = Qi1i2i3 if i1 , i2 , i3 are non-distinct , and ( diag ( Q ) ) i1i2i3 = 0 otherwise . It follows that E ( A ) = Q− diag ( Q ) , where Qi1i2i3 = θi1θi2θi3π′i1 ( Pπi3 ) πi2 . ( 1.2 ) For identifiability , let P ∈ RK , K2 be the matricization of P ( see Definition 1.1 ) . We assume rank ( P ) = K , and Piii = 1 for all 1 ≤ i ≤ K. ( 1.3 ) Definition 1.2 . We call ( 1.1 ) - ( 1.3 ) the tensor-DCMM model for 3-uniform hypergraphs . We call Q and P the Bernoulli probability tensor and community structure tensor for DCMM , respectively . Later in Section 3 , we introduce the non-uniform tensor-DCMM as a more sophisticated model . In tensor-DCMM , if we require all πi to be degenerate ( i.e. , one entry is 1 , all other entries are 0 ) , then tensor-DCMM reduces to the Degree-Corrected Block Model ( tensor-DCBM ) [ 15 ] . If we further require θ1 = . . . = θn ( but the second condition in ( 1.3 ) can be removed ) , then tensor-DCBM further reduces to the Stochastic Block Model ( tensor-SBM ) [ 9 ] . For simplicity , we may drop “ tensor '' in these terms if there is no confusion . The global testing problem above is then to test H0 : K = 1 vs. H1 : K > 1 . ( 1.4 ) Our primary goals are ( a ) to find a sharp information lower bound for 3-uniform DCMM , and especially , to fully characterize the lower bound by a simple quantity to be discovered , and ( b ) extend the results to more sophisticated non-uniform hypergraphse ( see Section 3 ) . A good understanding of the problem greatly helps us understand the fundamental limits of many other problems ( e.g. , community detection , determining the number of communities K , dynamic hypergraphs ) . For example , for community detection ( e.g. , [ 10 ] ) , we either assume K as known or estimate it first . Note that in parameter regions where we can not tell whether K = 1 or K > 1 , we can not estimate K consistently , so we can not have consistent community detection either . Therefore , a lower bound for global testing is always a valid lower bound for estimating K and for community detection . To facilitate the lower bound study , we frequently adopt a Random Mixed-Membership ( RMM ) model . Introduce a subset V0 = { x ∈ RK : xk ≥ 0 , ∑K k=1 xk = 1 } , and let F be a K-variate distribution with support contained in V0 . We assume πi iid∼ F ; ( let h = EF [ πi ] ) . ( 1.5 ) Moreover , let V ∗0 = { e1 , e2 , . . . , eK } ⊂ V0 , where ek is the k-th basis vector of RK . Similarly , RMM-DCMM reduces to RMM-DCBM if we require supp ( F ) = V ∗0 , and reduces to RMM-SBM if we further require θ1 = θ2 = . . . = θn ( but the second condition in ( 1.3 ) can be removed ) . Let θ = ( θ1 , θ2 , . . . , θn ) ′ . We allow ( θ , P , h , F ) to vary with n to cover a variety of settings where we allow for severe degree heterogeneity , mixed-memberships , flexible sparsity levels , and weak signals . Example 1 ( 2-parameter SBM [ 2 , 22 , 19 ] ) . This model is a special case of DCMM , where θ1 = . . . = θn = αn ( no degree heterogeneity ) , all πi are degenerate ( no mixed membership ) , and Pijk = 1 if i = j = k and Pijk = ρn otherwise , for two parameters ( αn , ρn ) . Also , Lin et al . [ 20 ] studied a 3-parameter SBM , which is the same as above except that they assume a different form of P , where Pijk equals to 1 , ρn , or τn if i , j , k take 1 , 2 , or 3 distinct values , respectively , for three parameters ( αn , ρn , τn ) . Compared to DCMM , these models are much narrower : they do not accommodate severe degree heterogeneity or mixed-memberships , and P is parametrized by 2 or 3 different values . How to derive a sharp lower bound for global testing ( and especially , to identify a simple quantity that fully characterizes the lower bound ) in our setting is a rather challenging problem . Our model is a non-uniform hypergraph model ( see Section 3 ) , which consists of hypergraphs of order 2 , 3 , . . . , M , and each layer consists of many unknown parameters ( θ , P , h , F ) . Existing works on lower bounds have been focused on uniform hypergraphs , and non-uniform hypergraphs are much less studied . Even for uniform hypergraphs , existing works have been focused on the the special SBM as in Example 1 , not the more general DCMM model . For example , Yuan et al . [ 22 ] derived the lower bounds for global testing with the 2-parameter SBM , focusing on the extremely sparse case . Ahn et al . [ 2 ] provided lower bound results for exactly recovering the communities ( see Liang et al . [ 19 ] and Kim et al . [ 16 ] for related settings ) with a similar model . Lin et al . [ 20 ] and Chien et al . [ 7 ] used the 3-parameter SBM in Example 1 for study of the lower bounds for community detection . While these papers are very interesting , their lower bounds are characterized by only 2 or 3 parameters ( i.e. , αn , ρn , τn ) assumed in their models . For the much broader tensor-DCMM model considered here , we have many parameters θ , P , h , F ( θ is an n-vector and P is a tensor ) , and how to extend existing results to the tensor-DCMM setting here is a challenging problem . Our contributions . The main challenge in lower bound study is that , since each DCMM model ( no matter what K is ) has a large number of unknown parameters , so given an alternative hypothesis ( K > 1 ) , it is hard to identify the null hypothesis ( K = 1 ) that is most difficult to distinguish from the alternative hypothesis . As our main contribution , we approach this by proposing a degree matching strategy , where for any given DCMM model with K > 1 , we pair it with a DCMM model with K = 1 in a way so that for each node , the expected degree under the null matches with that under the alternative . This way , it is hard to separate the two hypotheses by a naive degree-based statistic . We show ( a ) the degree matching is always possible by using a tensor scaling technique [ 3 , 8 ] , and ( b ) the pair of hypotheses we construct this way lead to sharp results on lower bounds . See Section 2 . We have the following results . Consider the 3-uniform hypergraph first . We first present the standard minimax theory . Define a class of RMM-DCMM models with K > 1 and µ22∥θ∥2∥θ∥1 → 0 ( µ2 is the second singular value of P ) . In this class , we can find an RMM-DCMM model ( the alternative ) , and pair it with a DCMM model with K = 1 ( the null ) , such that the χ2-divergence between the pair converges to 0 as n → ∞ . Therefore , in this class , there exists an alternative model that is asymptotically inseparable from the null . The standard minimax theory only claims that there exists an alternative model within a specified class that is inseparable from the null . It is desirable to show a much stronger result where for any alternative in the class , we can pair it with a null so that the χ2-divergence of the pair goes to 0 as n → ∞ . In detail , we show that in the parameter space ( θ , P , h , F ) of RMM-DCBM , there is a Region of Impossibility defined by µ22∥θ∥2∥θ∥1 → 0 ; for any alternative in Region of Impossibility , we can pair it with a null such that the χ2-divergence between the pair goes 0 as n→ ∞ . Compared with existing results on minimax lower bounds , these results are more informative and theoretically more satisfactory . The proof is also different from the proof of minimax lower bounds : we have used the tensor scaling theory [ 3 , 8 ] and the “ degree matching strategy '' aforementioned . We also extend such results to the broader RMM-DCMM case ( Section 2.3 ) and discuss some major differences on the Region of Impossibility between DCBM and the more restrictive SBM ( Section 2.4 ) . Next , we generalize the results to higher-order and non-uniform hypergraphs . Fix M ≥ 2 and consider a non-uniform hypergraph ( e.g. , see [ 10 ] ) that consists of m-uniform hypergraphs for all m = 2 , . . . , M , each following a DCMM model with individual ( θ ( m ) , P ( m ) ) but the common π1 , . . . , πn . Let ℓm = ∥θ ( m ) ∥m−21 ∥θ ( m ) ∥2 ( µ ( m ) 2 ) 2 ( to be defined in Section 3 ) . We show that ( a ) for the m-uniform hypergraph case , the Region of Impossibility for the hypothesis testing ( 1.4 ) is fully characterized by the condition of ℓm → 0 , and ( b ) for the non-uniform hypergraph case , the Region of Impossibility for the hypothesis testing ( 1.4 ) is fully characterized by the condition of max2≤m≤M { ℓm } → 0 . Last , we show that our lower bounds are tight . Consider the non-uniform hypergraph above . We propose a new test statistic and show that the sum of Type I error and Type II error → 0 if max2≤m≤M { ℓm } ≥ log ( n ) 1+δ for a constant δ > 0 ( taking δ = 0.1 will work ) . Therefore , except for a logarithmic factor here , our lower bounds are tight . In summary , existing results on lower bounds are largely focused on more restrictive settings ( e.g. , uniform hypergraphs without degree heterogeneity or mixed membership ) . We provide sharp lower bounds for a much broader setting . Our study is highly non-trivial because we need ( i ) a novel degree matching strategy to construct least favorable hypothesis pairs , ( ii ) to identify a simple quantity that is able to fully characterize the lower bounds , ( iii ) delicate analysis of the χ2-divergence between the null and alternative , ( iv ) a carefully designed test that leads to tight upper bounds . 2 Sharp Lower Bounds for 3-Uniform Hypergraphs For notational simplicity , we focus on 3-uniform hypergraphs in this section . The study of higherorder and non-uniform hypergraphs is deferred to Section 3 . Consider a ( 3-uniform ) DCMM model . Recall that h = EF [ πi ] and that P ∈ RK , K 2 is the matricization of the tensor P . In this paper , we use C > 0 as a generic constant which may vary from occurrence to occurrence . We assume ∥P∥ ≤ C , θmax ≡ max { θ1 , . . . , θn } ≤ C , max 1≤k≤K { hk } ≤ C min 1≤k≤K { hk } , ∥θ∥2∥θ∥1 → ∞ . ( 2.6 ) The first condition is mild , because the model identifiability already requires that all diagonal entries of P are 1 . The second one is also mild ( note that while the largest possible value of θmax is O ( 1 ) , θmax is allowed to tend to 0 relatively fast ) . The third one assumes that the community memberships are balanced , which is also mild . For the last condition , we will see soon that if ∥θ∥2∥θ∥1 → 0 , then the signal is so weak that successful global testing is impossible , so this condition is also mild .

Summary: This paper studies a binary hypothesis testing problem that considers whether a given hypergraph has one or more than one community. Under a general class of degree-corrected mixed-membership model for hypergraphs, it identifies conditions under which it is impossible for any test to distinguish between the null and alternative hypothesis. The above results, which are first derived for uniform hypergraphs, are then extended to non-uniform hyper-graphs by viewing a non-uniform hypergraph as a collection of uniform hypergraphs. The paper also proposes a statistical test that successfully solves the above-mentioned hypothesis testing problem under conditions that can get logarithmically close to the impossibility regime. Experiments on simulated data validate some of the theoretical findings.