Title: Exact recovery and Bregman hard clustering of node-attributed Stochastic Block Model
Venue: NeurIPS 2023
Accepted: yes
Keywords: community detection

stochastic block model

bregman divergence
Abstract: Classic network clustering tackles the problem of identifying sets of nodes (communities) that have similar connection patterns. However, in many scenarios nodes also have attributes that are correlated and can also be used to identify node clusters. Thus, network information (edges) and node information (attributes) can be jointly leveraged to design high-performance clustering algorithms. Under a general model for the network and node attributes, this work establishes an information-theoretic criteria for the exact recovery of community labels and characterizes a phase transition determined by the Chernoff-Hellinger divergence of the model. The criteria shows how network and attribute information can be exchanged in order to have exact recovery (e.g., more reliable network information requires less reliable attribute information). This work also presents an iterative clustering algorithm that maximizes the joint likelihood, assuming that the probability distribution of network interactions and node attributes belong to exponential families. This covers a broad range of possible interactions (e.g., edges with weights) and attributes (e.g., non-Gaussian models) while also exploring the connection between exponential families and Bregman divergences. Extensive numerical experiments using synthetic and real data indicate that the proposed algorithm outperforms algorithms that leverage only network or only attribute information as well as recently proposed algorithms that perform clustering using both sources of information. The contributions of this work provide insights into the fundamental limits and practical techniques for inferring community labels on node-attributed networks.

Exact recovery and Bregman hard clustering of node-attributed Stochastic Block Model

Abstract

Network clustering tackles the problem of identifying sets of nodes (communities) that have similar connection patterns. However, in many scenarios, nodes also have attributes that are correlated with the clustering structure. Thus, network information (edges) and node information (attributes) can be jointly leveraged to design high-performance clustering algorithms. Under a general model for the network and node attributes, this work establishes an information-theoretic criterion for the exact recovery of community labels and characterizes a phase transition determined by the Chernoff-Hellinger divergence of the model. The criterion shows how network and attribute information can be exchanged in order to have exact recovery (e.g., more reliable network information requires less reliable attribute information). This work also presents an iterative clustering algorithm that maximizes the joint likelihood, assuming that the probability distribution of network interactions and node attributes belong to exponential families. This covers a broad range of possible interactions (e.g., edges with weights) and attributes (e.g., nonGaussian models), as well as sparse networks, while also exploring the connection between exponential families and Bregman divergences. Extensive numerical experiments using synthetic data indicate that the proposed algorithm outperforms classic algorithms that leverage only network or only attribute information as well as state-of-the-art algorithms that also leverage both sources of information. The contributions of this work provide insights into the fundamental limits and practical techniques for inferring community labels on node-attributed networks.

1 Introduction

Community detection or network clustering-the task of identifying sets of similar nodes in a networkis a fundamental problem in network analysis [4, 1, 18], with applications in diverse fields such as digital humanities, data science and biology. In the classic formulation, a set of communities must be determined from the connection patterns among the nodes of a single network. A simple random graph model with community structure, the Stochastic Block Model (SBM), has been the canonical model to characterise theoretical limitations and evaluate different community detection algorithms [1].

However, nodes of many real-world networks have attributes or features that can reveal their identity as an individual or within a group. For example, the age, gender and ethnicity of individuals in a social network [27], the title, keywords and co-authors of papers in a citation network [32], or the longitude and latitude of meteorological stations in weather forecast networks [8]. In some scenarios, such attributes can be leveraged alone to identify node communities (clusters) without even using the network.

Thus, a modern formulation for community detection must consider network information (edges) and node information (attributes). Indeed, recent works have designed community detection algorithms that can effectively leverage both sources of information to improve performance. Various methods have been proposed for clustering node-attributed networks, including modularity optimisation [14], belief propagation [15], expected-maximisation algorithms [24, 34], information flow compression [33], semidefinite programming [37], spectral algorithms [2, 7, 25] and iterative likelihood based methods [8].

A fundamental problem in this new formulation is fusing both sources of information: how important is network information in comparison to node information given a problem instance? Intuitively, this depends on the noise associated with network edges and node attributes. For example, if edges are reliable then the clustering algorithm should prioritize them when determining the communities. However, most prior approaches adopt some form of heuristic when merging the two sources of information [14, 16, 37]. A rigorous approach to this problem requires a mathematical model, and one has been recently proposed.

The Contextual Stochastic Block Model (CSBM) is a generalization of the SBM where each node has a random attribute that depends on its community label. While the model formulation is general (in terms of distribution for edges and attributes), CSBM has only been rigorously studied in the restrictive setting where the pairwise interactions are binary (edges are present or not) and the node attributes are Gaussian [2, 8, 15]. In this scenario, the phase transitions for exactly recovering the community labels and for detecting them better than a random guess has been established. Moreover, the comparison with the respective phase transitions in SBM [1] and in Gaussian mixture model [11, 26] (with no network) highlight the value of jointly leveraging network and node information in recovering community labels.

However, real networks often depart from binary edges and Gaussian attributes. Indeed, in many scenarios network edges have weights that reveal information about that interaction and nodes have discrete or non-Gaussian attributes. This work tackles this scenario by considering a CSBM where edges have weights and nodes have attributes that follow some arbitrary distributions. Under this general model, this work is the first to characterise the phase transition for the exact recovery of community labels. In particular, the Chernoff-Hellinger divergence , initially defined just for binary networks [3], is extended to this more general model. This divergence effectively captures the difficulty of distinguishing different communities and thus plays a crucial role in determining the limits of exact recovery. The analysis reveals an additional term in the divergence that quantifies the information provided by the attributes of the nodes. Moreover, it quantifies the trade-off between network and node information in meeting the threshold for exact recovery.

The CSBM generates weighted networks that are complete (all possible edges are present) when edge weights follow a continuous distribution. However, most weighted real networks are sparse. To model sparse weighted networks and to provide a practical community detection algorithm, we consider a CSBM whose weights belong to zero-inflated distributions . More precisely, we suppose that conditioned on observing an edge, the distribution of the weight of this edge belongs to an exponential family. Similarly, the node attribute distributions are also assumed to belong to an exponential family. Working with exponential families is motivated by two factors. Firstly, exponential families encompass a broad range of parametric distributions, including the commonly used Bernoulli, Poisson, Gaussian, or Gamma distributions. Secondly, there exists an intricate connection between exponential families of distributions and Bregman divergences, which has proven to be a powerful tool for developing algorithms across a variety of problems such as clustering, classification, and dimensionality reduction [6, 13].

This connection between Bregman divergences and exponential families has been previously explored in the context of clustering dense networks (all possible edges are present) [23]. In contrast, this work proposes an iterative algorithm that maximizes the log-likelihood of the model, for both dense and sparse networks. This is a key difference with many previous works which either study only dense

weighted networks [9, 24] or binary networks with Gaussian attributes [2, 15, 34]. Simulations on synthetic networks demonstrate that our algorithm outperforms state-of-the-art approaches in various settings, providing practical techniques for achieving accurate clustering results.

The article is structured as follows. The relevant related work is discussed in Section 2. Section 3 introduces the model under consideration along with the main theoretical contributions on exact recovery. Section 4 focuses on sparse networks with edge weights and node attributes drawn from exponential families and introduces an iterative algorithm for clustering such networks. Numerical results and comparisons to prior works are presented in Section 5, and Section 6 concludes the paper.

Notations Let Ber( p ) denote a Bernoulli random variable (r.v.) with parameter p , Nor( m,σ ) a Gaussian r.v. with mean m and standard deviation σ , and Exp( λ ) an exponential r.v. with mean λ -1 . The notation [ K ] refers to the set { 1 , · · · , K } , while A i · stands for the i -th row of matrix A .

2 Related work

2.1 Exact recovery in SBM with edge weights and node attributes

Community detection in classic SBM (binary edges) is a well-understood problem with strong theoretical results concerning exact recovery and efficient algorithms with guaranteed accuracy [1, 39]. However, extending the classic SBM to weighted networks (non-binary edges) with arbitrary distributions is an ongoing research area. Most existing work in this scenario has been restricted to the homogeneous model 1 , where edge weights within and across communities are determined by two respective distributions. Moreover, existing works often restrict to categorical or real-valued weights [22, 36], or to multiplex networks (multiple edge types) with independent and identically distributed layers [29]. However, a recent work has provided a strong theoretical foundation the homogeneous model with arbitrary distributions [5], highlighting the role of the Rényi divergence as the key information-theoretic quantity for the homogeneous model.

In non-homogeneous models, a more complex divergence called the Chernoff-Hellinger divergence is the appropriate information-theoretic quantity for exact community recovery [3]. However, the expression of the Chernoff-Hellinger divergence as originally defined in [3] for binary networks does not have an intuitive interpretation, and its extension to non-binary (weighted) networks is challenging. For example, the exact recovery threshold for non-homogeneous SBM whose edges are categorical random variables has been established [38], but this threshold is expressed as a condition involving the minimization of a mixture of Kullback-Leibler divergences over the space of probability distributions. Although the relationship between Kullback-Leibler and Chernoff divergences are known (see for example [35, Theorem 30-32]), the specific technical lemma required to link them to the Chernoff-Hellinger divergence is not straightforward (see [38, Claim 4]).

Another generalization of the SBM allows for nodes to have attributes that provide information about their community, such as the Contextual SBM (CSBM) [15]. The CSBM has only been rigorously studied in the setting where edges are binary and node attributes follow a Gaussian distribution. In this scenario, the phase transition for exact recovery for the community labels has been established [2, 8, 15]. A natural generalization is to investigate the model where network edges have weights and nodes have attributes that follow arbitrary distributions. Indeed, this is one of the main contributions of this work: Expression (3.4) gives a straightforward yet crucial formula for the phase transition for exact recovery, also providing a natural interpretation for the influence of both the network and node attributes. Moreover, Expression (3.4) also applies when no node attribute is available, thus providing the exact recovery threshold for a non-homogeneous model and arbitrary edge weight distribution, a significant advancement in the state of the art.

2.2 Algorithms for clustering weighted networks with node attributes

Algorithms leveraging different approaches have been proposed to tackle community detection in networks with edge weights and node attributes. A common principled approach is to determine the community assignment that maximizes the likelihood function of a model for the data. However, optimizing the likelihood function is computationally intractable even for binary networks. Thus,

approximation schemes such as variational inference and pseudo-likelihood methods are often adopted. For instance, [24] introduced a variational-EM algorithm for clustering non-homogeneous weighted SBM with arbitrary distributions. Another approach for clustering node-attributed SBM whose edge weights and attribute distributions belong to exponential families is [23]. These two approaches assume that the network is dense (all edges are present and have non-zero edge weight). However, most real networks are very sparse (most node pairs do not have an edge) and this work focuses on this scenario. Another very recent work tackling sparse networks is the IR_sLs algorithm from [8], although its theoretical guarantees assume binary networks with Gaussian attributes.

The iterative clustering algorithm presented in this work maximizes the pseudo-likelihood likelihood by assuming that the probability distribution of network edges and node attributes belong to exponential families. This yields a direct connection with Bregman divergences and establishes an elegant expression for the likelihood function. This connection has also been leveraged in [23], however, their model is restricted to dense weighted networks (all edges are present). This work (more specifically, Lemma 2), demonstrates that this connection can also be applied to sparse weighted networks (using zero-inflated distributions to model the weights). This extension enhances the applicability of pseudo-likelihood algorithms using Bregman divergence to a broader class of scenarios, namely weighted sparse networks with node attributes.

3 Model and exact recovery in node-attributed SBM

3.1 Model definition

Consider a population of n objects, called nodes , partitioned into K ≥ 2 disjoint sets, called blocks or communities. A node-labelling vector z = ( z 1 , · · · , z n ) ∈ [ K ] n represents this partitioning so that z i indicates the block of node i . The labels (blocks) of nodes are random variables assumed to be independent and identically distributed such that P ( z i = k ) = π k for some vector π ∈ (0 , 1) K verifying that ∑ k π k = 1 . The nodes interact in unordered pairs giving rise to undirected edges, and X is the measurable space of all possible pairwise interactions. Additionally, each node has an attribute that is an element of a measurable space Y . Let X ∈ X N × N denote the symmetric matrix such that X ij represents the interaction between node pair ( ij ) , and by Y = ( Y 1 , · · · , Y n ) ∈ Y n the node attribute vector.

Assume that interactions and attributes are independent conditionally on the community labels of the nodes. Let f kℓ ( x ) denote the probability that two nodes in blocks k and ℓ have an interaction x ∈ X , and h k ( y ) denote the probability that a node in block k ∈ [ K ] has an attribute y ∈ Y . Thus,

In the following, the interaction spaces X , Y might depend on n , as well as the respective interaction probabilities f, h . The number of nodes n will increase to infinity while K and π are constant. For an estimator ˆ z ∈ [ K ] n of z , we define the classification error as

where S K is the set of permutations of [ K ] and Ham( · , · ) is the hamming distance between two vectors. An estimator ˆ z = ˆ z ( X,Y ) achieves exact recovery if P (loss( z, ˆ z ) ≥ 1) = o (1) .

3.2 Exact recovery threshold in node-attributed SBM

The difficulty of classifying empirical data in one of K possible classes is traditionally measured by the Chernoff information [12]. More precisely, in the context of network clustering, let CH( a, b ) = CH( a, b, π, f, h ) denote the hardness of distinguishing nodes that belong to block a from block b . This quantity is defined by

where

is the Chernoff coefficient of order t across blocks a and b , and D t ( f ∥ g ) = 1 t -1 log ∫ f t ( x ) g 1 -t ( x ) dx is the Rényi divergence of order t between two probability densities f, g [35]. The key quantity assessing the possibility or impossibility of exact recovery in SBM is then the minimal Chernoff information across all pairs of clusters. We denote it by I = I ( π, f, h ) , and it is defined by

̸

The following Theorem provides the information-theoretic threshold for exact recovery in nodeattributed SBM.

Theorem 1. Consider model (3.1) with π a > 0 for all a ∈ [ K ] . Denote by a ∗ , b ∗ the two hardest blocks to estimate, that is CH( a ∗ , b ∗ ) = I . Suppose that t ∈ (0 , 1) ↦→ lim n →∞ n log n CH t ( a ∗ , b ∗ ) exists and is strictly concave. Then the following holds:

(i) exact recovery is information-theoretically impossible if lim n →∞ n log n I < 1 ;

(ii) exact recovery is information-theoretically possible if lim n →∞ n log n I > 1 .

̸

The proof for Theorem 1 is provided in the supplemental material. The main ingredient of the proof is the asymptotic study of log-likelihood ratios. More precisely, let L ( z ) = P ( X,Y | z ) . The application of Chernoff bounds provides an upper bound on -log P (log L ( z ) ≥ log L ( z ′ )) , where z denotes the correct block structure and z ′ ∈ [ K ] n is another node-labelling vector (see Lemma 1 in the supplement). We can use this upper bond to prove that the maximum likelihood estimator (MLE) achieves exact recovery if I > n -1 log n . Reciprocally, if I < n -1 log n , we show that whp there exist some "bad" nodes i for which z i = arg max a ∈ [ k ] P ( X,Y | z -i , z i = a ) . In other words, even in a setting where an oracle would reveal z -i ( i.e., the correct block assignment of all nodes except node i ), the MLE would fail at recovering z i . Establishing this fact requires to lower bound -log P (log L ( z ) ≥ log L ( z ′ )) where z ′ ∈ [ K ] n is such that Ham( z, z ′ ) = 1 ( i.e., z ′ correctly labels all nodes except one). This lower bound is obtained using large deviation results for general random variables [10, 30]. To apply these results, the strict concavity of the limit n (log n ) -1 I is needed. In most practical settings, this assumption is verified, except in some edge cases (see Examples 1 and 2). Let us now provide some examples of applications of Theorem 1.

Example 1 (Binary SBM with no attributes) . Suppose that f ab ∼ Ber( α ab n -1 log n ) where α ab are constants. A Taylor-expansion of the Rényi divergence between Bernoulli distributions leads to

̸

which indeed coincides with the expression of the Chernoff-Hellinger divergence defined in [3]. We also note that the limit n (log n ) -1 I is strictly concave as long as the α ab are not all equals 2 .

Example 2 (Binary SBM with Gaussian attributes) . Suppose that f ab ∼ Ber( α ab n -1 log n ) and h a ∼ Nor( µ a log n, σ 2 I d ) , where α ab and µ a are independent of n . Then,

̸

In particular, the technical conditions of Theorem 1 are verified if we rule out the uninformative case where all the α ab 's and the µ a 's are equal to each other. Thus, exact recovery is possible if

̸

̸

Further assuming that α ab = α 1( a = b ) + β 1( a = b ) (homogeneous interactions) and π = ( 1 K , · · · , 1 K ) (uniform block probabilities), the expression of I simplifies to

̸

̸

̸

where ∆ = min a = b ∥ µ a -µ b ∥ 2 . This last scenario recovers the recently established threshold for exact recovery in the Contextual SBM [8].

Example 3 (Semi-supervised clustering in SBM) . Consider binary interactions given by

where α, β are independent of n . Consider a semi-supervised model in which the vector of attributes Y is a noisy oracle of the true community labels z . More precisely, for a node i such that z i = k , let

with η 0 + η 1 = η . A bit of algebra shows that exact recovery is possible if

When η 0 = 0 (perfect oracle), the condition simplifies to ( √ a -√ b ) 2 -2 log(1 -η ) log n > K . Note that the oracle term is non-negligible only if -log (1 -η ) ≳ log n , as previously established [31]. This last condition is very strong since it implies η ≳ 1 -1 /n , and hence the oracle must provide the correct label for almost all nodes.

4 Bregman hard clustering of sparse weighted node-attributed networks

In this section, we will propose an algorithm for clustering sparse, weighted networks with node attributes. When present, the weights are sampled from an exponential family, and the node attributes also belong to an exponential family. In Section 4.1, we provide some reminder of exponential families. We derive the likelihood of the model in Section 4.2, and present the algorithm in Section 4.3.

4.1 Exponential family

An exponential family E ψ is a parametric class of probability distributions whose densities can be canonically written as p θ,ψ ( x ) = e <θ,x> -ψ ( θ ) , where the density is taken with respect to an appropriate measure, θ ∈ Θ is a function of the parameters of the distribution that must belong to an open convex space Θ , and ψ is a convex function.

We consider the model defined in (3.1), such that f ab are zero-inflated distributions and are given by

where p ab ∈ [0 , 1] is the interaction probability between blocks a and b , δ 0 ( x ) is the Dirac delta at zero, and ˜ f ab is a probability density with no mass at zero. Note that this model can represent sparse weighted networks, as edges between nodes in blocks a and b are absent with probability 1 -p ab .

Finally, suppose that the distributions { ˜ f ab } and { h a } belong to exponential families. More precisely,

for some parameters θ ab , η a and functions ψ, ϕ . The following lemma provides the expression of the Chernoff divergence of this model.

Lemma 1. Let f ab and h a be defined as in (4.1) -(4.2) . Suppose that p ab = α ab δ where α ab is constant and δ ≪ 1 . We have

̸

In combination with Theorem 1, Lemma 1 provides the expression for the exact recovery threshold for node-attributed networks with interaction and attribute distributions given by (4.1)-(4.2).

4.2 Log-likelihood

Given a convex function ψ , the Bregman divergence d ψ : R m × R m → R + is defined by

The log-likelihood of the density p ψ,θ of an exponential family distribution is linked to the Bregman divergence by the following relationship (see for example [6, Equation 13])

where µ = E p ψ,θ ( X ) is the mean of the distribution, and ψ ∗ denotes the Legendre transform of ψ , defined by ψ ∗ ( t ) = sup θ { < θ, t > -ψ ( θ ) } . The following Lemma provides an expression for the log-likelihood of f ab when f ab is a distribution belonging to a zero-inflated exponential family.

Lemma 2. Let f ab be a probability density as defined in (4.1) -(4.2) . For x, y ∈ (0 , 1) , let d KL ( x, y ) be the Kullback-Leibler divergences between Ber( x ) and Ber( y ) , and let H ( x ) = x log x +(1 -x ) log(1 -x ) , with the usual convention 0 log 0 = 1 . Then,

̸

where s = 1( x = 0) .

Proof. To express log f ab ( x ) , we first note that

̸

where c = 1( x = 0) . The result follows by adding and subtracting H ( b ) in the previous expression and expressing log ˜ f ab ( x ) with a Bregman divergence as in (4.3).

̸

Suppose that X,Y follow the model (3.1) with probability distributions given by (4.1)-(4.2). Let A be a binary matrix such that A ij = 1( X ij = 0) . We have

̸

where the additional term c is a function of X,Y but does not depend on z . Denoting Z ∈ { 0 , 1 } n × K the one-hot membership matrix such that Z ik = 1( z i = k ) , observe that p z i z j = ( ZpZ T ) ij where p is a symmetric matrix with the interaction probabilities between different blocks, µ z i z j = ( ZµZ T ) ij where µ is a symmetric matrix with the expected value of the interaction between different blocks (edge weights), and ν z i = ( Z T ν ) i where ν is a vector with the expected value of the attribute for different blocks. Thus, up to some additional constants, the negative log-likelihood -log P ( X,Y | Z ) is equal to

̸

where d ′ ψ ∗ ( B,C ) = ∑ n j =1 1( B j = 0) d ψ ∗ ( B j , C j ) for two vectors B,C ∈ R n .

4.3 Clustering by iterative likelihood maximisation

Following the log-likelihood expression derived in (4.4), we propose an iterative clustering algorithm that places each node in the block maximising P ( X,Y | z -i , z i = a ) for 1 ≤ a ≤ K , the likelihood that node i is in community a given the community labels of the other nodes, z -i . Let Z ( ia ) denote the membership matrix obtained from Z by placing node i in block a , and let L ia ( Z ( ia ) ) denote the contribution of node i to the negative log-likelihood when node i is placed in block a . Equation (4.4) shows that

where the p , µ and ν in the equation above must be estimated from X , Y , and the community membership matrix Z . Let ˆ p = ˆ p ( A,Z ) , ˆ µ = ˆ µ ( X,Z ) , and ˆ ν = ˆ ν ( Y, Z ) denote the estimators for p , µ and ν , respectively. Their values can be computed as follows:

Note that the matrix inverse ( Z T Z ) -1 can be easily computed since Z T Z is a K -byK diagonal matrix. This approach is described in Algorithm 1.

Algorithm 1: Bregman hard clustering for node-attributed SBM.

A fundamental aspect of many likelihood maximization iterative algorithms such as Algorithm 1 is the initial membership assignment, Z 0 . This initial assignment often has a profound influence on the final membership assignment, and thus, it is important to have an adequate initialization. In the numerical section, we proceed as follows. We construct the matrix W ∈ R n × 2 K such that the first K columns of W are the first K eigenvectors of the graph normalised Laplacian, while the last K columns of W are the first K eigenvectors of the Gram matrix Y Y T .

5 Numerical experiments

5.1 Performance of Algorithm 1

We first compare in Figure 1 the performance of Algorithm 1 in terms of exact recovery (fraction of times the algorithm correctly recovers the community of all nodes) with the theoretical threshold for exact recovery proved in the paper (red curve in the plots) in two settings: Figure 1a shows binary weight with Gaussian attributes, and Figure 1b shows zero-inflated Gaussian weights with Gaussian attributes. A solid black (resp., white) square means that over 50 trials, the algorithms failed 50 times (resp., succeeded 50 times) at exactly recovering the block structure.

5.2 Comparison with other algorithms

In this section, we compare Algorithm 1 with other algorithms presented in the literature. We used the Adjusted Rand Index (ARI) [21] between the predicted clusters and the ground truth ones to evaluate the performance of each algorithm.

In Figure 2, we compare Algorithm 1 with the variational-EM algorithm of [24] and the algorithm of [23] (which is also based on Bregman divergences, but tailored for dense networks). Because both of these algorithms are designed for dense networks, we observe that Algorithm 1 has overall better performance on sparse networks.

We also compare Algorithm 1 with the IR_sLs algorithm from [8]. This is one of the most recent algorithms for node-attributed SBM and it comes with theoretical guarantees (for binary networks

1

2

3

4

5

6

alpha

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

r

0.0

0.2

0.4

0.6

0.8

1.0

(a) Binary weights with Gaussian attributes

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

mu

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

r

0.0

0.2

0.4

0.6

0.8

1.0

(b) zero-inflated Gaussian weights with Gaussian attributes.

(a) n = 500 , K = 2 , f in = Ber( αn log n ) , f out = Ber( n log n ) . The attributes are 2dspherical Gaussian with radius ( ± r √ log n, 0) and identity covariance matrix.

(b) n = 600 , K = 3 , f in = (1 -ρ ) δ 0 + ρ Nor( µ, 1) , f out = (1 -ρ ) δ 0 + ρ Nor(0 , 1) with ρ = 5 n -1 log n . The attributes are 2d-spherical Gaussian whose means are the vertices of a regular polygon on the circle of radius r √ log n .

0.03

0.05

0.07

0.09

0.11

p

0.00

0.25

0.50

0.75

1.00

ARI

Algorithm 1

bregman [23]

vem [24]

initialisation

1

3

5

7

9

11

muin

0.00

0.25

0.50

0.75

1.00

ARI

Algorithm 1

bregman [23]

vem [24]

initialisation

with Gaussian attributes). We also compare with the EM algorithm of [34], attSBM , which is designed for binary networks with Gaussian attributes. Finally, we compare with baseline methods for clustering using the network or the attributes alone. EM-GMM refers to fitting a Gaussian Mixture Model via EM on attribute data Y , and sc refers to spectral clustering on network data X .

Figure 3 shows the results for binary networks with Gaussian attributes. Algorithm 1 successfully learns from both the signal coming from the network and the attributes, even in scenarios where one of them is non-informative. Moreover, Algorithm 1 has better performance than the two other node-attributed clustering algorithms, and those algorithms also show a large variance 3 . We also note that IR_sLs and attSBM are both tailor-made for binary edges and Gaussian attributes. Even in such a setting, Algorithm 1 outperforms these two algorithms. We show in the supplement material that when the network is weighted and the attributes non-Gaussian, IR_sLs and attSBM perform poorly.

5.3 Evaluation using real datasets

The following three benchmark datasets were used to evaluate and compare the proposed algorithm: CiteSeer ( n = 3279 , m = 9104 , K = 6 , d = 3703 ), Cora ( n = 2708 , m = 10556 , K = 7 ,

(a) Varying edge distribution.

(b) Varying attribute distributions.

1.2

1.0

0.8

ARI

0.6

EM-GMM

0.4

SC

Algo1

0.2

0.0

IR_sLS

5

7

9

11

13

15

a

1.0

0.8

ARI

0.6

EM-GMM

0.4

SC

Algo1

0.2

IR_sLS

0.0

0

1

2

3

4

5

r

d = 1433 ), and Cornell ( n = 183 , m = 298 , K = 5 , d = 1703 ) (all available in Pytorch Geometric). For each network, the original node attribute vector was reduced to have dimension d = 10 by selecting the 10 best features according to the chi-square test. Algorithm 1 assumed a multivariate Gaussian distribution with d = 10 for node attributes and Bernoulli edges (these networks have no edge weights). The initialization for Algorithm 1 and attSBM used spectral clustering of both the node similarity matrix (using node attributes) and network edges.

Table 1 shows that Algorithm 1 outperformed the other three algorithms. Spectral clustering ( sc ) has near zero performance, indicating that the network structure in these data sets has little information concerning the clusters of the nodes. Moreover, both Algorithm 1 and attSBM (that leverage network and node attributes) outperform EM-GMM that use only node attributes. These preliminary results indicate that Algorithm 1 is promising even in real data sets with little pre-processing.

6 Conclusion

This work made the following contributions to community detection in node-attributed networks: i) extended the known thresholds for exact recovery in binary SBM to non-binary (weighted) networks with node attributes, providing a clean expression for a new information-theoretic quantity, known in the binary setting as the Chernoff-Hellinger divergence; ii) proposed an iterative algorithm based on the likelihood function that can infer community memberships from a problem instance. The algorithm leverages the framework of Bregman divergences and is simple and computationally efficient. Numerical experiments indicate the superiority of this algorithm when compared to recent state-of-the-art approaches.

References

Emmanuel Abbe. Community detection and stochastic block models: recent developments. The Journal of Machine Learning Research , 18(1):6446-6531, 2017.

Emmanuel Abbe, Jianqing Fan, and Kaizheng Wang. An ℓ p theory of pca and spectral clustering. The Annals of Statistics , 50(4):2359-2385, 2022.

Emmanuel Abbe and Colin Sandon. Community detection in general stochastic block models: Fundamental limits and efficient algorithms for recovery. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science , pages 670-688, Los Alamitos, CA, USA, 2015. IEEE Computer Society.

Konstantin Avrachenkov and Maximilien Dreveton. Statistical Analysis of Networks . Now Publishers, 2022.

Konstantin Avrachenkov, Maximilien Dreveton, and Lasse Leskelä. Community recovery in non-binary and temporal stochastic block models. arXiv preprint arXiv:2008.04790 , 2022.

Arindam Banerjee, Srujana Merugu, Inderjit S Dhillon, Joydeep Ghosh, and John Lafferty. Clustering with Bregman divergences. Journal of machine learning research , 6(10), 2005.

Norbert Binkiewicz, Joshua T Vogelstein, and Karl Rohe. Covariate-assisted spectral clustering. Biometrika , 104(2):361-377, 2017.

Guillaume Braun, Hemant Tyagi, and Christophe Biernacki. An iterative clustering algorithm for the contextual stochastic block model with optimality guarantees. In International Conference on Machine Learning , pages 2257-2291. PMLR, 2022.

Andressa Cerqueira and Elizaveta Levina. A pseudo-likelihood approach to community detection in weighted networks. arXiv preprint arXiv:2303.05909 , 2023.

Narasinga Rao Chaganty and Jayaram Sethuraman. Strong large deviation and local limit theorems. The Annals of Probability , pages 1671-1690, 1993.

Xiaohui Chen and Yun Yang. Cutoff for exact recovery of Gaussian mixture models. IEEE Transactions on Information Theory , 67(6):4223-4238, 2021.

Herman Chernoff. A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. The Annals of Mathematical Statistics , pages 493-507, 1952.

Michael Collins, Sanjoy Dasgupta, and Robert E Schapire. A generalization of principal components analysis to the exponential family. Advances in neural information processing systems , 14, 2001.

David Combe, Christine Largeron, Mathias Géry, and El˝ od Egyed-Zsigmond. I-Louvain: An attributed graph clustering method. In Advances in Intelligent Data Analysis XIV: 14th International Symposium, IDA 2015, Saint Etienne. France, October 22-24, 2015. Proceedings 14 , pages 181-192. Springer, 2015.

Yash Deshpande, Subhabrata Sen, Andrea Montanari, and Elchanan Mossel. Contextual stochastic block models. Advances in Neural Information Processing Systems , 31, 2018.

Issam Falih, Nistor Grozavu, Rushed Kanawati, and Younès Bennani. Community detection in attributed network. In Companion proceedings of the the web conference 2018 , pages 1299-1306, 2018.

Cees M Fortuin, Pieter W Kasteleyn, and Jean Ginibre. Correlation inequalities on some partially ordered sets. Communications in Mathematical Physics , 22:89-103, 1971.

Santo Fortunato and Darko Hric. Community detection in networks: A user guide. Physics reports , 659:1-44, 2016.

Geoffrey Grimmett. Percolation . Springer, 1999.

Theodore E. Harris. A lower bound for the critical probability in a certain percolation process. In Mathematical Proceedings of the Cambridge Philosophical Society , volume 56, pages 13-20. Cambridge University Press, 1960.

Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification , 2:193-218, 1985.

Varun Jog and Po-Ling Loh. Recovering communities in weighted stochastic block models. In Proceedings of the 53rd Annual Allerton Conference on Communication, Control, and Computing , Los Alamitos, CA, USA, October 2015. IEEE Computer Society.

Bo Long, Zhongfei Mark Zhang, and Philip S Yu. A probabilistic framework for relational clustering. In ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD) , pages 470-479, 2007.

Mahendra Mariadassou, Stéphane Robin, and Corinne Vacher. Uncovering latent structure in valued graphs: A variational approach. The Annals of Applied Statistics , 4(2):715 - 742, 2010.

Angelo Mele, Lingxin Hao, Joshua Cape, and Carey E Priebe. Spectral inference for large stochastic blockmodels with nodal covariates. arXiv preprint arXiv:1908.06438 , 2019.

Mohamed Ndaoud. Sharp optimal recovery in the two component Gaussian mixture model. The Annals of Statistics , 50(4):2096-2126, 2022.

Mark EJ Newman and Aaron Clauset. Structure and inference in annotated networks. Nature communications , 7(1):11863, 2016.

Frank Nielsen. Chernoff information of exponential families. arXiv preprint arXiv:1102.2684 , 2011.

Subhadeep Paul and Yuguo Chen. Consistent community detection in multi-relational data through restricted multi-layer stochastic blockmodel. Electronic Journal of Statistics , 10(2):3807 - 3870, 2016.

Detlef Plachky and Joseph Steinebach. A theorem about probabilities of large deviations with an application to queuing theory. Periodica Mathematica Hungarica , 6(4):343-345, 1975.

Hussein Saad and Aria Nosratinia. Community detection with side information: Exact recovery under the stochastic block model. IEEE Journal of Selected Topics in Signal Processing , 12(5):944-958, 2018.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. AI magazine , 29(3):93-93, 2008.

Laura M Smith, Linhong Zhu, Kristina Lerman, and Allon G Percus. Partitioning networks with node attributes by compressing information flow. ACM Transactions on Knowledge Discovery from Data (TKDD) , 11(2):1-26, 2016.

Natalie Stanley, Thomas Bonacci, Roland Kwitt, Marc Niethammer, and Peter J Mucha. Stochastic block models with multiple continuous attributes. Applied Network Science , 4(1):1-22, 2019.

Tim Van Erven and Peter Harremos. Rényi divergence and Kullback-Leibler divergence. IEEE Transactions on Information Theory , 60(7):3797-3820, 2014.

Min Xu, Varun Jog, and Po-Ling Loh. Optimal rates for community estimation in the weighted stochastic block model. The Annals of Statistics , 48(1):183 - 204, 2020.

Bowei Yan and Purnamrita Sarkar. Covariate regularized community detection in sparse graphs. Journal of the American Statistical Association , 116(534):734-745, 2021.

Se-Young Yun and Alexandre Proutiere. Optimal cluster recovery in the labeled stochastic block model. Advances in Neural Information Processing Systems , 29, 2016.

Anderson Y. Zhang and Harrison H. Zhou. Minimax rates of community detection in stochastic block models. The Annals of Statistics , 44(5):2252 - 2280, 2016.

A Establishing the exact recovery threshold

The proof of Theorem 1 is structured as follows. We start by establishing some concentration results on the block sizes in Section A.1. In Section A.2, we delve into the asymptotic analysis of log-likelihood ratios, establishing fundamental results. Building upon these findings, we prove the converse statement, demonstrating the impossibility of exact recovery below the threshold, in Section A.3. Conversely, in Section A.4, we establish the positive statement, demonstrating the possibility of exact recovery above the threshold. To complement the proof, Section A.5 presents a set of technical combinatorial lemmas.

Notations In the following, we denote by z ∈ [ K ] n the true block-labelling vector, and by z ′ ∈ [ K ] n another block-labelling vector, and the conditional probabilities are denoted by P z ( · ) = P ( · | z ) .

A.1 Preliminaries on the block sizes

For any z ′ ∈ [ K ] n , we for all c ∈ [ K ] define ˆ π c ( z ′ ) , the empirical size of block c , as

For any 0 < ξ < 1 , we define

Recalling that the true block-labelling vector z is sampled from π ⊗ n , we have by concentration of multinomial distributions (see for example [5, Lemma A1])

In the following, we chose ξ = log log n √ n , and hence P ( Z ξ ) = 1 -o (1) .

A.2 Asymptotic study of log-likelihood ratios

This section studies the asymptotic behaviour of P z ( L ( z ) ≥ L ( z ′ )) , where L ( z ′ ) = P z ′ ( X,Y ) the likelihood of the block-labelling vector z ′ given the observed data X,Y .

A.2.1 Upper-bound

The following lemma provides an upper bound on P z ( L ( z ) ≥ L ( z ′ )) .

Lemma 3. Let z, z ′ ∈ Z be two block labelling vectors such that Ham( z, z ′ ) = m ≥ 1 , and let L ( z ) = P ( X,Y | z ) be the likelihood of labelling z . We have

̸

Moreover, let d = sup t ∈ (0 , 1) min a = b ∈ [ K ] c ∈ [ K ] (1 -t ) D t ( f ac ∥ f bc ) . Then we also have

Proof. Using Chernoff's bound, we have for all t ≥ 0 ,

where the probability measure P z is defined on X × Y by (3.1) in the main text. The linearity of the Rényi divergence with respect to product distributions implies

̸

where

Moreover, Lemma 6 ensures that

Therefore,

̸

̸

Using N cc = ∣ ∣ z -1 ( c ) ∣ ∣ -∑ d = c N cd and that ∑ c ∑ d = c N cd = m , we obtain that

̸

̸

Hence,

̸

Using the concentration of ˆ z , we have ˆ π c ( z ) = (1 + o (1)) π c for all c , and hence

̸

where we used ∑ a = b N ab = m and I ≤ CH( a, b, ˆ π (ˆ z )) . The first upper bound on P z ( L ( z ′ ) ≥ L ( z )) follows by taking ϵ = 0 and the supremum over t ∈ (0 , 1) in (A.2).

̸

Finally, let d = sup t ∈ (0 , 1) min a = b ∈ [ K ] c ∈ [ K ] (1 -t ) D t ( f ac ∥ f bc ) . Combining (A.3) and (A.4) leads to

̸

and this ends the proof.

A.2.2 Lower bound

We now focus on lower-bounding P z ( L ( z ) ≥ L ( z ′ )) when Ham( z, z ′ ) = 1 . In particular, the condition Ham( z, z ′ ) = 1 implies that there exists a unique node u ∈ [ n ] such that z u = z ′ u . Let a, b ∈ [ K ] such that z u = a and z ′ u = b . By definition of the likelihood, we have L ( z ) -L ( z ′ ) = ∆ ub ( X,Y,z ) where

̸

̸

Before studying the large deviation rates of likelihood ratios such as P z (∆ ub > 0) and P z (∆ ub > 0 , ∆ vb > 0) , we first recall some large deviation result for arbitrary sequences of random variables [30, 10].

̸

̸

̸

̸

̸

̸

Proposition 1 (Strong large deviations for arbitrary sequences of random variables - Theorem 3.3 of [10]) . Let T n be a sequence of random variables whose moment generating function ϕ n ( t ) = E [ e tT n ] is non-vanishing and analytic in the region Ω = { z ∈ C : | z | < a } for some a > 0 . Let k n be a sequence of real numbers, and let ψ n ( t ) = k -1 n log ϕ n ( t ) . Let ( m n ) be a bounded sequence of real numbers such that there exists a sequence ( τ n ) verifying ψ ′ n ( τ n ) = m n and 0 < τ n < α 0 < α . Suppose that:

there exists b > 0 such that | ψ n ( z ) | ≤ b for all z ∈ Ω ;

there exists a > 0 such that ψ ′′ n ( τ n ) > c ;

there exists δ 0 > 0 such that sup δ< | t | <λτ n ∣ ∣ ∣ ϕ n ( τ n + it ) ϕ n ( τ n ) ∣ ∣ ∣ = o ( k -1 / 2 n ) for any given δ, λ such that 0 < δ < δ 0 < λ .

Then,

Proposition 1 is an extension of Cramer's large deviation theorem for sums of iid r.v. to sequences of arbitrary random variables (see [10, Remark 3.6]). We will apply it to the study of P (∆ ub > 0) .

Lemma 4. Let z ∈ Z ξ (where Z ξ is defined in (A.1) ) with ξ = o (1) . Let u ∈ [ n ] such that z u = a and let ∆ ub be given in (A.5) . Suppose that there exists δ n = ω (1) such that β : t ∈ (0 , 1) ↦→-lim n →∞ nδ -1 n CH t ( a, b ) exists and is strictly convex. Then, for any sequence ϵ n such that δ n ϵ n → ϵ ∈ { β ′ ( t ) , t ∈ (0 , 1) } we have

where β ∗ ( x ) = sup t> 0 { tx -β ( t ) } is the Legendre transform of β , and t ϵ is such that β ′ ( t ϵ ) = ϵ . In particular 0 ∈ { β ′ ( t ) , t ∈ (0 , 1) } , and therefore for any sequence ϵ n = o ( δ -1 n ) we have

Proof of Lemma 4. We will apply Proposition 1 with Ω = { z ∈ C : | z | < 1 } and k n = δ -1 n to obtain the stated results.

Let us first compute ϕ n ( t ) = E z [ e t ∆ ub ] and ψ n ( t ) = δ -1 n log ϕ n ( t ) for t ∈ (0 , 1) . We first notice that for all v ∈ [ n ] ,

by definition of the Rényi divergence. Similar computations show that

Therefore, where

with ˆ π c = n -1 ∣ ∣ z -1 ( k ) ∣ ∣ . We note that since z ∈ Z ξ we have CH t ( a, b, z ) = (1 + O ( ξ ))CH t ( a, b ) . Thus, ψ n ( t ) = -(1 + o (1)) n CH t ( a, b ) . Moreover, the assumption CH( a, b ) = Θ ( n -1 δ n ) implies

that CH t ( a, b ) = Θ ( n -1 δ n ) for all t ∈ (0 , 1) since the Rényi divergences of orders t ∈ (0 , 1) are equivalent [35, Theorem 16]. Hence, δ -1 n ψ n ( t ) = (1+ o (1)) β ( t ) . Since β is well-defined and strictly convex on (0 , 1) , this ensures that Assumptions 1 and 2 of Proposition 1 are verified. Moreover, we notice that e β ( t ) is the m.g.f of some r.v. X , and let φ ( t ) = E e itX be the characteristic function of X . Then, ϕ n ( t ∗ + it ) ϕ n ( t ∗ ) = ( φ ( t ∗ + it ) φ ( t ∗ ) ) δ n . Since φ ( t ∗ + it ) φ ( t ∗ ) is the characteristic function of some r.v., its module is strictly less than 1 on an interval not containing 0 , and hence Assumption 3 of Proposition 1 is verified.

Finally, Lemma 8 ensures that β ′ (0) ≤ 0 and β ′ (1) ≥ 0 and hence 0 ∈ { β ′ ( t ) , t ∈ (0 , 1) } .

A.2.3 Asymptotic independence

Finally, the following lemma shows that the events ∆ ub > 0 and ∆ vb > 0 are asymptotically independent.

Lemma 5. Let z ∈ Z ξ , u, v ∈ z -1 ( a ) , and b ∈ [ K ] \{ a } . Then,

Proof. Let ξ ( X uv ) = log f aa f ab ( X uv ) . We have

Therefore,

where U u and U v are iid. Therefore,

Conditioning on U u and U v , we observe that

where ϕ t ( ξ ) = 1( t > ξ ) and µ = Law( ξ ) . Because ϕ u and ϕ u ′ are monotonous, the Fortuin-Kasteleyn-Ginibre (FKG) inequality [20, 17] (see also [19, Section 2.2]) implies that Cov( ϕ u ( ξ ) , ϕ u ′ ( ξ )) ≥ 0 . Therefore,

Let us now derive an upper bound for this ratio. First, Lemma 4 with ϵ n = 0 and δ n = log n implies

and the same relation holds for P (∆ vb > 0) . Moreover, conditionally on X uv we have

where γ ( x ) = P z ( U > ξ ( x )) . Since CH t ( a, b ) = o (1) , then f ac and f bc are mutually contiguous 4 for all c (see [35, Theorem 25]). In particular, this implies that ξ ( x ) < ∞ for all x ∈ X such that

f aa ( x ) > 0 . Thus, we can apply Lemma 4 with ϵ n = ξ ( x ) and δ n = log n , verifying δ n ϵ n → 0 to obtain

and this ends the proof.

A.3 Impossibility of exact recovery

̸

For a, b ∈ [ K ] two block indexes, we recall the quantity CH( a, b ) defined in (3.2) of the main text denotes the hardness to distinguish a node in block a from a node in block b . We suppose that I = min a = b CH( a, b ) = Θ ( log n n ) with lim n →∞ n log n I < 1 . We prove the failure of the MAP estimator for exact recovery in three steps:

(i) We start by showing that we can restrict the study to the node labelling vector z for which the relative size of the communities are close to their expectations, i.e., | z -1 ( a ) | n = (1 + o (1)) π a for all a ∈ [ K ] ;

(ii) Let M ab ( z ) be the number of nodes in block a for which changing their community label to block b results in an increase of likelihood. We show that E M ab ( z ) ≫ 1 when a, b are the two hardest blocks to distinguish.

(iii) We show that M ab ( z ) > 0 almost surely using the second-moment method.

(i) Conditioning on well-behaving community sizes. Recall the definition of Z ξ in (A.1). We established in Section A.1 that P ( Z ξ ) = 1 -o (1) for ξ = log log n √ n . Hence,

̸

̸

̸

The rest of the proof is devoted to show that P z ( z MAP = z ) = 1 -o (1) for all z ∈ Z ξ .

̸

(ii) Expected number of bad nodes. Given a block structure z ∈ Z ξ , an arbitrary node u ∈ [ n ] such that z u = a and a block b = a . We define by ˜ z the block structure obtained from z by swapping the block of node u to b , i.e.,

Suppose that X,Y are generating from a true block structure z , and let

the change in the MAP estimation of z obtained by swapping the label of node u from its true block a to a wrong block b . By the definition of ˜ z , we have using Bayes' law

̸

̸

Moreover,

where C does not depend on b . Hence

̸

The number of nodes in block a for which updating the label to b would cause the change of likelihood to be strictly positive is given by

In the following, we select a, b as the indexes of the two hardest blocks to distinguish. Hence, I = CH( a, b, π, f, h ) and we have using Lemma 4 that

Since z ∈ Z ξ we have that CH( a, b, ˆ π, f, h ) = (1+ o (1)) I . Moreover, by assumption I < (1 -ϵ ) log n n for some ϵ > 0 and for any n large enough. Hence E z M ab ( z ) ≫ 1 .

(iii) Conclusion. We will conclude that M ab ( z ) > 0 almost surely using the second-moment method. Denote by α = P z (∆ ub > 0) and β = P (∆ ub > 0 , ∆ vb > 0) for two arbitrary distinct nodes u, v ∈ z -1 ( a ) . We have E z M ab = ∣ ∣ z -1 ( k ) ∣ ∣ α and

Hence, the second-moment method implies that

We end the proof using Lemma 5.

A.4 Possibility of exact recovery

To show that the MAP estimator achieves exact recovery up to the desired threshold, we need to show that there is no possibility of reducing the likelihood by swapping m vertices from each community. In all the following, we recall that z denotes the true community labelling, and for any z ′ ∈ [ K ] we denote by L ( z ′ ) = P ( X,Y | z ′ ) the likelihood of labelling z ′ . For any m ≥ 1 , let us denote the set of node-labelling at a distance m of z by

The probability that there exists a labelling z ′ ∈ Γ m with higher likelihood than z is

̸

Let ˆ z be the block labelling estimated by the MAP. Lemma 7 ensures that | Γ m | = 0 if m ≥ n K -1 K . Thus, we have, using union bounds,

Moreover, [39, Proposition 5.2] show that

while Lemma 3 provides an upper bounds on max z ′ ∈ Γ m P z ( L ( z ′ ) ≥ L ( z )) . Combining these upper bounds ensure that for some sequence η = o (1) , we have

̸

where s m = enK m min ( e -(1 -η ) ( 1 -nπ m min ) nI ; e -( n -m ) d ) with d = sup t ∈ (0 , 1) min a = b ∈ [ K ] c ∈ [ K ] (1 - t ) D t ( f ac ∥ f bc ) .

Let ϵ > 0 such that I > (1 + ϵ ) log n n . We will now show that the sum on the right-hand side of Equation (A.9) goes to zero, by considering the cases (i) m ≤ π min 2 n and (ii) π min 2 n ≤ m ≤ 1 2 n . (i) First of all, suppose that m ≤ π min 2 n . Then,

Let f ( m ) = log m m -1+ ϵ π min log n n . We will show that f ( m ) ≥ log m 2 m for m ≥ 3 . Indeed, for x ≥ 3 , the function x ↦→ log x x is decreasing. Therefore, for 3 ≤ m ≤ π min 2 n we have

for n large enough. Hence s m ≤ eKn -ϵ/ 2 e -1 2 log m and ∑ π min 2 n m =1 s m m = o (1) .

(ii) Next, suppose that 1 2 π min n ≤ m ≤ n K -1 K . Then, the assumption D t ( f ac ∥ f bc ) = Θ ( log n n ) implies that d > κ log n n for some positive constant κ . Hence,

Hence, (A.9) shows that P z (loss( z, ˆ z ) ≥ 1) = o (1) , and thus the MAP estimator exactly recovers the true community structure z .

A.5 Additional lemmas

Lemma 6. Let z, z ′ ∈ [ K ] n and define for all a, b, c, d ∈ [ K ] :

We have

Proof. Denote by C a = z -1 ( a ) and C ′ a = z ′-1 ( a ) , and let C ab = C a ∩ C ′ b . Then, A ab = | C ab | and M ( ab )( cd ) = | { 1 ≤ i < j ≤ n : i ∈ C ab and j ∈ C cd } | . This proves the lemma.

Lemma 7. For any z, z ′ ∈ [ K ] n , we have loss( z, z ′ ) ≤ n (1 -1 K ) .

Proof. Without loss of generality, suppose that loss( z, z ) = Ham( z, z ′ ) (that is, the optimal permutation in the definition of the loss is simply the identity). For any a ∈ [ K ] , let us denote by C a = z -1 ( a ) , C ′ a = z -1 ( b ) . The confusion matrix of the two node labellings is the K -byK matrix having entries N ab = | C a ∩ C b | . In particular, we have n = ∑ a,b N ab and

̸

[5, Lemma B.2] show that N ab ≤ N aa + N bb -N ba , and therefore

̸

̸

We notice that and similarly,

̸

and this ends the proof.

Lemma 8. Let f, g be two probability distributions and denote c ( t ) = (1 -t ) D t ( f ∥ g ) the Chernoff coefficient of order t between f and g . We have c ′ (0) = -d KL ( g ∥ f ) and c ′ (1) = d KL ( f ∥ g ) .

Proof. From c ( t ) = log ∫ f t g 1 -t , we notice that c ′ ( t ) = ∫ f t g 1 -t log f g ∫ f t g 1 -t , and the result follows.

Finally, ∑ a ∑ b = a N ba = loss( z, z ′ ) . Thus, going back to (A.10) leads to

̸

B Proof of Lemma 1

Proof of Lemma 1. Using a Taylor expansion, we have

and we finish the proof using ∫ ( ˜ f ac ) t ( ˜ f bc ) 1 -t = e -J ψ ( θ ab ∥ θ bc ) and (1 -t ) D t ( h a ∥ h b ) = J ϕ ( η a ∥ η b ) (see for example [28]).

C Additional numerical results

We present in Table 2 and 3 the numerical results that are drawn in Figure (4a) and (4b), respectively. We observe that the variance of Algorithm 1 is very low, while all other algorithms (excepted EM-GMM ) exhibit a very large variance.

We also provide in Figure 4 comparison of Algorithm 1 with IR_sLs and attSBM on weighted networks with attributes exponentially distributed. We observe that contrary to Algorithm 1, IR_sLs and attSBM do not perform well in that setting (which is expected as they are designed for binary network and Gaussian attributes).

1.0

0.8

0.6

0.4

0.2

0.0

-0.2

0.8

0.6

ARI

0.4

EM-GMM

SC

0.2

Algo1

0.0

IR_sLS

1

2

3

4

5

attributes_avg

(b) Varying

1

3

6

9

weights_avg

(a) Varying

λ

-

1

w

, with

λ

12

a

= 1

/

2

.

λ

-

1

a

, with

λ

w

= 1

/

2

.

ARI

EM-GMM

SC

Algo1

IR_sLS

C.1 Robustness to the choice of d ψ ∗ and d ϕ ∗

We show in Figure 5 that using a divergence (distribution) for edge weights (Figure 5a and node attributes (Figure 5b different from the distribution used to generate the data does not impact the performance of Algorithm 1. We note that a similar observation was done in previous papers using Bregman divergence for clustering [6, 23].

1.0

0.8

0.6

0.4

0.2

euclidean manhattan

Itakura-Saito poisson

9

5

7

11

13

15

muin

(a) Various d ψ ∗ . Poisson is the correct model.

3

5

7

9

11

0.0

0.2

0.4

0.6

0.8

1.0

ARI

euclidean

manhattan

Itakura-Saito

poisson

(a) n = 400 , K = 4 , f in = (1 -p ) δ 0 ( x ) + p Poi( µ in ) and f out = (1 -q ) δ 0 ( x ) + q Poi(5) , with p = 0 . 04 and q = 0 . 01 . Attributes are 2d-Gaussians with unit variances and mean equally spaced the circle of radius r = 2 .

(b) n = 400 , K = 2 , f in = (1 -p ) δ 0 ( x ) + p Nor(2 , 1) and f out = (1 -q ) δ 0 ( x ) + q Nor(0 , 1) , with p = 0 . 04 and q = 0 . 01 . Attributes are Poisson with means ν 1 (for nodes in cluster 1) and 3 (for nodes in cluster 2).

ARI

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the analysis of exact recovery of the Stochastic Block model to node attributed graphs (CSBM). This opens up a new dimension as the desired clustering must not only be optimal in the sense of the SBM but also respect the node attributes (which are assumed to be distributed based on the block of the SBM). First, a binary setting is presented, where edges are either present or not. For this setting, the authors present a tight bound on the information theoretical (im-)possibility of recovering the blocks of the SBM using the Chernoff information of the model. Second, a setting is analysed where the graph structure is sampled from an SBM and node attributes and edge weights are sampled from a distribution from the exponential family. For this setting, the authors present the Chernoff information of the model given its parameters allowing for a concrete characterization of the phase transition between possibility and impossibility of recovery. An iterative likelihood maximization algorithm is then proposed and evaluated on synthetic data.\n\nSTRENGTHS:\n- This paper provides a nice perspective on the relevance of information gained from the network structure and that gained from the node attributes.\n- The claims of the paper seem sound. \n- The sections themselves have clear statements.\n\nWEAKNESSES:\n- The experiments are not convincing. \n\t- All experiments are carried out on synthetic graphs generated in a way that benefits this algorithm. \n\t- The presented algorithm is compared with 2 baseline algorithms that only optimize for one of the two dimensions and thus fail at the other. Instead, one could have compared with other approaches to node-attributed community detection, which should pose more of a challenge. [19,20] could have been used as a comparison.\n- The related work section is quite sparse and is used to distinguish the contribution rather than present related work. E.g. section 2.2 \"Algorithms for clustering node-attributed networks\" contains 2 citations. This should be augmented. \n- In the presentation of the results it should be made much clearer that the developments presented here, heavily depend on [3]; with the current writing the reader gets a wrong impression of the novelty of the here presented ideas.\n- The structure and the presentation of this paper are sometimes confusing. \t\n\t- The reader is often left guessing where we are headed next. E.g. section 4, which is framed as the main contribution (extending [19] by allowing for sparse networks) simply starts by introducing zero-inflated distributions from the exponential family. Or 4.2 gives the derivation for the negative log-likelihood without previously mentioning that this will be used in the iterative likelihood maximization algorithm. \n\t- The numbering in the appendix overwrites the numbering in the paper (e.g. Theorem 1 in the paper is not Theorem 1 in the appendix) which makes referencing the paper from the appendix or the appendix from the paper very hard.\n\nQUESTIONS:\n- Can you find any guarantees for your algorithm such as recovery with high probability in a certain regime? \n- In the experiments, why don't you compare your approach to the other papers [19,20] that you mentioned in your related work?\n- In the experiments, how close is the performance of your algorithm to the theoretical bound you presented?\n- Why did you not compare the algorithms on real-world data?\n\nLIMITATIONS:\nLimitations of the setting or the approach are not discussed -- this should be rectified.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies clustering of node-attributed Stochastic Block Model (SBM). The authors provide information-theoretic threshold for exact recovery under generic distributions for both edge weights and node attributes. In addition, the authors propose a clustering algorithm based on iterative likelihood maximization when edge weights and node attributes are drawn from exponential family distributions. The authors carry out experiment on synthetic data and show that the proposed algorithm outperforms existing state-of-the-art methods.\n\nSTRENGTHS:\n- The theoretical setting (i.e. clustering of node attributed SBM) considered in this work is fairly general. The threshold for exact recovery is presented in a general setting without assuming any particular family of distributions for either edge weights or node attributes. The examples and implications of Theorem 1 provided on page 5 are nice. \n\n- The threshold which involves Equation (3.3) captures an intuitive additive signal from edge connections and node attributes.\n\n- The overall writing and presentation is very clear and generally easy to follow (although there are many typos here and there). In particular, the presentation of background and related works seem reasonably thorough.\n\n- The empirical results, though limited to synthetic data, are promising.\n\n- Overall, I think this paper studies a well-known problem in a more general setting than those have been considered in prior work. The new results and algorithm can be a nice addition.\n\nWEAKNESSES:\n- Algorithm 1 requires knowing the functions $\\psi^*$ and $\\phi^*$. This requirement often does not hold in practice. The authors should discuss the practical implications for such requirement and what if these functions are unknown.\n\n- Though not necessary for a theory paper, it would be nice to have at least some elementary experiments on real data.\n\n- There are many typos throughout the paper. Here are some:\n  - Line 95: standard deviation $\\sigma^2$ -> standard deviation $\\sigma$\n  - Line 124: while his work -> while this work\n  - Line 129: is find the community -> is finding the community\n  - Equation (4.2): subscripts for $\\theta$ do not match, similarly, subscripts for $\\eta$ do not match\n  - Lemma 2: $p_{k\\ell}$ in the Equation should be $p_{ab}$?\n  - Line 284: in Nor(...) I think the placements for mean and variance should be reversed\n\nQUESTIONS:\n- In practice, how should one use Algorithm 1 when $\\psi^*$ and $\\phi^*$ are unknown?\n\n- Does Example 2 imply that an oracle to the true community is almost useless unless the oracle gives almost full access to all labels?\n\nLIMITATIONS:\nI could not find where the authors discuss the limitations or potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies community recovery in sparse, weighted networks, which is an important setting that is more general than the commonly-studied undirected, unweighted networks. The authors' first main contribution is to establish the information-theoretic conditions for exact community recovery in this setting, which is a form of the Chernoff information (and cleanly reduces to well-known info-theoretic thresholds in unweighted settings). Next, assuming that edge and node attributes belong to exponential families, an expression for the likelihood in terms of Bregman divergences is given, which then leads to a natural clustering procedure. Extensive numerical experiments are provided comparing the proposed procedure to other approaches.\n\nSTRENGTHS:\nDeriving the information-theoretic threshold for exact recovery via the Chernoff information is perhaps not surprising, but it is a solid contribution as it encompasses a wide range of important examples. The clustering procedure derived, based on a connection between exponential families and Bregman divergences, is clean and intuitive. The simulations on synthetic data show that the clustering procedure has favorable performance in terms of the ARI, compared to other methods, which is a good validation of the theory.\n\nWEAKNESSES:\nThe paper could be improved if the authors filled in some gaps regarding a few central topics. \n- Please discuss in more detail how similar / different your approach is to [19], since both leverage the connection between exponential families and Bregman divergences to come up with clustering procedures. \n- The information-theoretic results are not too surprising. Does the proof require new techniques to handle the more general setting under consideration? If so, they should be discussed in the main text, as it would be of interest to theoreticians in this field. \n- Please provide additional discussion around the accuracy and sample complexity of the clustering algorithm. Is it efficient? Is it guaranteed to output the MLE or similar?\n\nQUESTIONS:\n- You use the term \"homogeneous\" a lot: can you define precisely what you mean by this? \n- Section 2: You can also consider citing work on Censored Block Models, in which edges take on 3 values (present, absent, unknown)\n- Theorem 1: How crucial is the assumption on the convexity of CH_t (a^*, b^*) in t?\n- A clarification question: if the attribute distributions were not from an exponential family, would the likelihood be intractable to compute? \n- For clarity, I would suggest defining ARI. \n- In the simulations, is the ARI measured between the estimated and ground-truth communities? If so, please clarify in the main text.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies community detection in node-attributed stochastic block models. Although these models have been studied before, this work has two main contributions: (i) the edge weights in the model are now not necessarily binary, but can also be weighted. The node attributes don’t have to be Gaussian, and can come from a more general exponential family. (ii) An iterative clustering algorithm which maximizes the likelihood by placing nodes in correct cluster based on attributes and edge weights.\nExperiments on synthetic datasets are also presented\n\nSTRENGTHS:\n•\tThe weakening of the assumptions in this work are more general than what previous work has considered. This is a significant strength of this work. The assumptions in this model (weighted edges, non Gaussian node attributes) are more realistic than what previous work has considered, and I consider this a nice contribution.\n\n•\tTo the best of my knowledge the theoretical work is sound and contains some non-trivial insights. Some of the technical techniques presented might have impact on future work.\n\n•\tExperimental results on synthetic data suggest that the proposed algorithm 1 performs well compared to previous work.\n\nWEAKNESSES:\n•\tGiven that a motivation for this paper is to provide an algorithm for a more realistic setting, I would have expected to see experiments on real world datasets. In general, stochastic block models enjoy a very symmetric structure, which in some sense makes it nice to cluster. It would be interesting to see how the performance of this algorithm generalises to real-world graphs. \n\n•\tExperimental results on synthetic data is only performed when $k=2$. Higher values of $k$ should be considered \n\n\n## Minor:\n\nLine 73: conditioned of --> conditioned on\n\nLine 129: is find --> is to find\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nnone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}