SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Maximilien Dreveton
maximilien.dreveton@epfl.ch
Felipe S. Fernandes
Systems Engineering and Computer Science
felipesc@cos.ufrj.br
Daniel R. Figueiredo
daniel@cos.ufrj.br
Network clustering tackles the problem of identifying sets of nodes (communities) that have similar connection patterns. However, in many scenarios, nodes also have attributes that are correlated with the clustering structure. Thus, network information (edges) and node information (attributes) can be jointly leveraged to design high-performance clustering algorithms. Under a general model for the network and node attributes, this work establishes an information-theoretic criterion for the exact recovery of community labels and characterizes a phase transition determined by the Chernoff-Hellinger divergence of the model. The criterion shows how network and attribute information can be exchanged in order to have exact recovery (e.g., more reliable network information requires less reliable attribute information). This work also presents an iterative clustering algorithm that maximizes the joint likelihood, assuming that the probability distribution of network interactions and node attributes belong to exponential families. This covers a broad range of possible interactions (e.g., edges with weights) and attributes (e.g., non-Gaussian models), as well as sparse networks, while also exploring the connection between exponential families and Bregman divergences. Extensive numerical experiments using synthetic data indicate that the proposed algorithm outperforms classic algorithms that leverage only network or only attribute information as well as state-of-the-art algorithms that also leverage both sources of information. The contributions of this work provide insights into the fundamental limits and practical techniques for inferring community labels on node-attributed networks.
Community detection or network clustering—the task of identifying sets of similar nodes in a network—is a fundamental problem in network analysis [4, 1, 18], with applications in diverse fields such as digital humanities, data science and biology. In the classic formulation, a set of communities must be determined from the connection patterns among the nodes of a single network. A simple random graph model with community structure, the Stochastic Block Model (SBM), has been the canonical model to characterise theoretical limitations and evaluate different community detection algorithms [1].
However, nodes of many real-world networks have attributes or features that can reveal their identity as an individual or within a group. For example, the age, gender and ethnicity of individuals in a social network [27], the title, keywords and co-authors of papers in a citation network [32], or the longitude and latitude of meteorological stations in weather forecast networks [8]. In some scenarios, such attributes can be leveraged alone to identify node communities (clusters) without even using the network.
Thus, a modern formulation for community detection must consider network information (edges) and node information (attributes). Indeed, recent works have designed community detection algorithms that can effectively leverage both sources of information to improve performance. Various methods have been proposed for clustering node-attributed networks, including modularity optimisation [14], belief propagation [15], expected-maximisation algorithms [24, 34], information flow compression [33], semidefinite programming [37], spectral algorithms [2, 7, 25] and iterative likelihood based methods [8].
A fundamental problem in this new formulation is fusing both sources of information: how important is network information in comparison to node information given a problem instance? Intuitively, this depends on the noise associated with network edges and node attributes. For example, if edges are reliable then the clustering algorithm should prioritize them when determining the communities. However, most prior approaches adopt some form of heuristic when merging the two sources of information [14, 16, 37]. A rigorous approach to this problem requires a mathematical model, and one has been recently proposed.
The Contextual Stochastic Block Model (CSBM) is a generalization of the SBM where each node has a random attribute that depends on its community label. While the model formulation is general (in terms of distribution for edges and attributes), CSBM has only been rigorously studied in the restrictive setting where the pairwise interactions are binary (edges are present or not) and the node attributes are Gaussian [2, 8, 15]. In this scenario, the phase transitions for exactly recovering the community labels and for detecting them better than a random guess has been established. Moreover, the comparison with the respective phase transitions in SBM [1] and in Gaussian mixture model [11, 26] (with no network) highlight the value of jointly leveraging network and node information in recovering community labels.
However, real networks often depart from binary edges and Gaussian attributes. Indeed, in many scenarios network edges have weights that reveal information about that interaction and nodes have discrete or non-Gaussian attributes. This work tackles this scenario by considering a CSBM where edges have weights and nodes have attributes that follow some arbitrary distributions. Under this general model, this work is the first to characterise the phase transition for the exact recovery of community labels. In particular, the Chernoff-Hellinger divergence, initially defined just for binary networks [3], is extended to this more general model. This divergence effectively captures the difficulty of distinguishing different communities and thus plays a crucial role in determining the limits of exact recovery. The analysis reveals an additional term in the divergence that quantifies the information provided by the attributes of the nodes. Moreover, it quantifies the trade-off between network and node information in meeting the threshold for exact recovery.
The CSBM generates weighted networks that are complete (all possible edges are present) when edge weights follow a continuous distribution. However, most weighted real networks are sparse. To model sparse weighted networks and to provide a practical community detection algorithm, we consider a CSBM whose weights belong to zero-inflated distributions. More precisely, we suppose that conditioned on observing an edge, the distribution of the weight of this edge belongs to an exponential family. Similarly, the node attribute distributions are also assumed to belong to an exponential family. Working with exponential families is motivated by two factors. Firstly, exponential families encompass a broad range of parametric distributions, including the commonly used Bernoulli, Poisson, Gaussian, or Gamma distributions. Secondly, there exists an intricate connection between exponential families of distributions and Bregman divergences, which has proven to be a powerful tool for developing algorithms across a variety of problems such as clustering, classification, and dimensionality reduction [6, 13].
This connection between Bregman divergences and exponential families has been previously explored in the context of clustering dense networks (all possible edges are present) [23]. In contrast, this work proposes an iterative algorithm that maximizes the log-likelihood of the model, for both dense and sparse networks. This is a key difference with many previous works which either study only dense
weighted networks [9, 24] or binary networks with Gaussian attributes [2, 15, 34]. Simulations on synthetic networks demonstrate that our algorithm outperforms state-of-the-art approaches in various settings, providing practical techniques for achieving accurate clustering results.
The article is structured as follows. The relevant related work is discussed in Section 2. Section 3 introduces the model under consideration along with the main theoretical contributions on exact recovery. Section 4 focuses on sparse networks with edge weights and node attributes drawn from exponential families and introduces an iterative algorithm for clustering such networks. Numerical results and comparisons to prior works are presented in Section 5, and Section 6 concludes the paper.
Notations Let  $\mathrm{Ber}(p)$  denote a Bernoulli random variable (r.v.) with parameter  $p$ ,  $\mathrm{Nor}(m, \sigma)$  a Gaussian r.v. with mean  $m$  and standard deviation  $\sigma$ , and  $\mathrm{Exp}(\lambda)$  an exponential r.v. with mean  $\lambda^{-1}$ . The notation  $[K]$  refers to the set  $\{1, \dots, K\}$ , while  $A_i$  stands for the  $i$ -th row of matrix  $A$ .
2.1 Exact recovery in SBM with edge weights and node attributes
Community detection in classic SBM (binary edges) is a well-understood problem with strong theoretical results concerning exact recovery and efficient algorithms with guaranteed accuracy [1, 39]. However, extending the classic SBM to weighted networks (non-binary edges) with arbitrary distributions is an ongoing research area. Most existing work in this scenario has been restricted to the homogeneous model<sup>1</sup>, where edge weights within and across communities are determined by two respective distributions. Moreover, existing works often restrict to categorical or real-valued weights [22, 36], or to multiplex networks (multiple edge types) with independent and identically distributed layers [29]. However, a recent work has provided a strong theoretical foundation the homogeneous model with arbitrary distributions [5], highlighting the role of the Rényi divergence as the key information-theoretic quantity for the homogeneous model.
In non-homogeneous models, a more complex divergence called the Chernoff-Hellinger divergence is the appropriate information-theoretic quantity for exact community recovery [3]. However, the expression of the Chernoff-Hellinger divergence as originally defined in [3] for binary networks does not have an intuitive interpretation, and its extension to non-binary (weighted) networks is challenging. For example, the exact recovery threshold for non-homogeneous SBM whose edges are categorical random variables has been established [38], but this threshold is expressed as a condition involving the minimization of a mixture of Kullback-Leibler divergences over the space of probability distributions. Although the relationship between Kullback-Leibler and Chernoff divergences are known (see for example [35, Theorem 30-32]), the specific technical lemma required to link them to the Chernoff-Hellinger divergence is not straightforward (see [38, Claim 4]).
Another generalization of the SBM allows for nodes to have attributes that provide information about their community, such as the Contextual SBM (CSBM) [15]. The CSBM has only been rigorously studied in the setting where edges are binary and node attributes follow a Gaussian distribution. In this scenario, the phase transition for exact recovery for the community labels has been established [2, 8, 15]. A natural generalization is to investigate the model where network edges have weights and nodes have attributes that follow arbitrary distributions. Indeed, this is one of the main contributions of this work: Expression (3.4) gives a straightforward yet crucial formula for the phase transition for exact recovery, also providing a natural interpretation for the influence of both the network and node attributes. Moreover, Expression (3.4) also applies when no node attribute is available, thus providing the exact recovery threshold for a non-homogeneous model and arbitrary edge weight distribution, a significant advancement in the state of the art.
2.2 Algorithms for clustering weighted networks with node attributes
Algorithms leveraging different approaches have been proposed to tackle community detection in networks with edge weights and node attributes. A common principled approach is to determine the community assignment that maximizes the likelihood function of a model for the data. However, optimizing the likelihood function is computationally intractable even for binary networks. Thus,
Also known as the planted partition model.
approximation schemes such as variational inference and pseudo-likelihood methods are often adopted. For instance, [24] introduced a variational-EM algorithm for clustering non-homogeneous weighted SBM with arbitrary distributions. Another approach for clustering node-attributed SBM whose edge weights and attribute distributions belong to exponential families is [23]. These two approaches assume that the network is dense (all edges are present and have non-zero edge weight). However, most real networks are very sparse (most node pairs do not have an edge) and this work focuses on this scenario. Another very recent work tackling sparse networks is the  $IR\_sLs$  algorithm from [8], although its theoretical guarantees assume binary networks with Gaussian attributes.
The iterative clustering algorithm presented in this work maximizes the pseudo-likelihood likelihood by assuming that the probability distribution of network edges and node attributes belong to exponential families. This yields a direct connection with Bregman divergences and establishes an elegant expression for the likelihood function. This connection has also been leveraged in [23], however, their model is restricted to dense weighted networks (all edges are present). This work (more specifically, Lemma 2), demonstrates that this connection can also be applied to sparse weighted networks (using zero-inflated distributions to model the weights). This extension enhances the applicability of pseudo-likelihood algorithms using Bregman divergence to a broader class of scenarios, namely weighted sparse networks with node attributes.
3 Model and exact recovery in node-attributed SBM
3.1 Model definition
Consider a population of  $n$  objects, called nodes, partitioned into  $K \geq 2$  disjoint sets, called blocks or communities. A node-labelling vector  $z = (z_{1},\dots ,z_{n}) \in [K]^{n}$  represents this partitioning so that  $z_{i}$  indicates the block of node  $i$ . The labels (blocks) of nodes are random variables assumed to be independent and identically distributed such that  $\mathbb{P}(z_i = k) = \pi_k$  for some vector  $\pi \in (0,1)^K$  verifying that  $\sum_{k}\pi_{k} = 1$ . The nodes interact in unordered pairs giving rise to undirected edges, and  $\mathcal{X}$  is the measurable space of all possible pairwise interactions. Additionally, each node has an attribute that is an element of a measurable space  $\mathcal{V}$ . Let  $X \in \mathcal{X}^{N\times N}$  denote the symmetric matrix such that  $X_{ij}$  represents the interaction between node pair  $(ij)$ , and by  $Y = (Y_{1},\dots ,Y_{n}) \in \mathcal{V}^{n}$  the node attribute vector.
Assume that interactions and attributes are independent conditionally on the community labels of the nodes. Let  $f_{k\ell}(x)$  denote the probability that two nodes in blocks  $k$  and  $\ell$  have an interaction  $x \in \mathcal{X}$ , and  $h_k(y)$  denote the probability that a node in block  $k \in [K]$  has an attribute  $y \in \mathcal{Y}$ . Thus,
$$
\mathbb {P} (X, Y \mid z) = \prod_ {1 \leq i <   j \leq n} f _ {z _ {i} z _ {j}} \left(X _ {i j}\right) \prod_ {i = 1} ^ {n} h _ {z _ {i}} \left(Y _ {i}\right). \tag {3.1}
$$
In the following, the interaction spaces  $\mathcal{X},\mathcal{Y}$  might depend on  $n$  as well as the respective interaction probabilities  $f,h$ . The number of nodes  $n$  will increase to infinity while  $K$  and  $\pi$  are constant. For an estimator  $\hat{z}\in [K]^n$  of  $z$ , we define the classification error as
$$
\operatorname {l o s s} (z, \hat {z}) = \min _ {\tau \in \mathcal {S} _ {K}} \operatorname {H a m} (z, \tau \circ \hat {z}),
$$
where  $S_K$  is the set of permutations of  $[K]$  and  $\mathrm{Ham}(\cdot, \cdot)$  is the hamming distance between two vectors. An estimator  $\hat{z} = \hat{z}(X,Y)$  achieves exact recovery if  $\mathbb{P}(\mathrm{loss}(z,\hat{z}) \geq 1) = o(1)$ .
3.2 Exact recovery threshold in node-attributed SBM
The difficulty of classifying empirical data in one of  $K$  possible classes is traditionally measured by the Chernoff information [12]. More precisely, in the context of network clustering, let  $\mathrm{CH}(a,b) = \mathrm{CH}(a,b,\pi ,f,h)$  denote the hardness of distinguishing nodes that belong to block  $a$  from block  $b$ . This quantity is defined by
$$
\mathrm {C H} (a, b) = \sup  _ {t \in (0, 1)} \mathrm {C H} _ {t} (a, b), \tag {3.2}
$$
$$
\mathrm {C H} _ {t} (a, b) = (1 - t) \left[ \sum_ {c = 1} ^ {K} \pi_ {c} \mathrm {D} _ {t} \left(f _ {b c} \| f _ {a c}\right) + \frac {1}{n} \mathrm {D} _ {t} \left(h _ {b} \| h _ {a}\right) \right] \tag {3.3}
$$
is the Chernoff coefficient of order  $t$  across blocks  $a$  and  $b$ , and  $D_t(f\|g) = \frac{1}{t-1} \log \int f^t(x) g^{1-t}(x) dx$  is the Rényi divergence of order  $t$  between two probability densities  $f, g$  [35]. The key quantity assessing the possibility or impossibility of exact recovery in SBM is then the minimal Chernoff information across all pairs of clusters. We denote it by  $I = I(\pi, f, h)$ , and it is defined by
$$
I = \min_{\substack{a,b\in [K]\\ a\neq b}}\mathrm{CH}(a,b). \tag{3.4}
$$
The following Theorem provides the information-theoretic threshold for exact recovery in node-attributed SBM.
Theorem 1. Consider model (3.1) with  $\pi_a > 0$  for all  $a \in [K]$ . Denote by  $a^*, b^*$  the two hardest blocks to estimate, that is  $\mathrm{CH}(a^*, b^*) = I$ . Suppose that  $t \in (0,1) \mapsto \lim_{n \to \infty} \frac{n}{\log n} \mathrm{CH}_t(a^*, b^*)$  exists and is strictly concave. Then the following holds:
(i) exact recovery is information-theoretically impossible if  $\lim_{n\to \infty}\frac{n}{\log n} I < 1$ ;
(ii) exact recovery is information-theoretically possible if  $\lim_{n\to \infty}\frac{n}{\log n} I > 1$
The proof for Theorem 1 is provided in the supplemental material. The main ingredient of the proof is the asymptotic study of log-likelihood ratios. More precisely, let  $L(z) = \mathbb{P}(X,Y\mid z)$ . The application of Chernoff bounds provides an upper bound on  $-\log \mathbb{P}(\log L(z)\geq \log L(z'))$ , where  $z$  denotes the correct block structure and  $z^{\prime}\in [K]^{n}$  is another node-labelling vector (see Lemma 1 in the supplement). We can use this upper bound to prove that the maximum likelihood estimator (MLE) achieves exact recovery if  $I > n^{-1}\log n$ . Reciprocally, if  $I < n^{-1}\log n$ , we show that whp there exist some "bad" nodes  $i$  for which  $z_{i}\neq \arg \max_{a\in [k]}\mathbb{P}(X,Y|z_{-i},z_{i} = a)$ . In other words, even in a setting where an oracle would reveal  $z_{-i}$  (i.e., the correct block assignment of all nodes except node  $i$ ), the MLE would fail at recovering  $z_{i}$ . Establishing this fact requires to lower bound  $-\log \mathbb{P}(\log L(z)\geq \log L(z'))$  where  $z^{\prime}\in [K]^{n}$  is such that  $\mathrm{Ham}(z,z^{\prime}) = 1$  (i.e.,  $z^{\prime}$  correctly labels all nodes except one). This lower bound is obtained using large deviation results for general random variables [10, 30]. To apply these results, the strict concavity of the limit  $n(\log n)^{-1}I$  is needed. In most practical settings, this assumption is verified, except in some edge cases (see Examples 1 and 2). Let us now provide some examples of applications of Theorem 1.
Example 1 (Binary SBM with no attributes). Suppose that  $f_{ab} \sim \mathrm{Ber}(\alpha_{ab} n^{-1} \log n)$  where  $\alpha_{ab}$  are constants. A Taylor-expansion of the Rényi divergence between Bernoulli distributions leads to
$$
I = (1 + o(1)) \frac {\log n}{n} \min _ {a \neq b} \sup _ {t \in (0, 1)} \left(\sum_ {c} \pi_ {c} \left[ t \alpha_ {b c} + (1 - t) \alpha_ {a c} - \alpha_ {b c} ^ {t} \alpha_ {a c} ^ {1 - t} \right]\right),
$$
which indeed coincides with the expression of the Chernoff-Hellinger divergence defined in [3]. We also note that the limit  $n(\log n)^{-1}I$  is strictly concave as long as the  $\alpha_{ab}$  are not all equals<sup>2</sup>.
Example 2 (Binary SBM with Gaussian attributes). Suppose that  $f_{ab} \sim \mathrm{Ber}(\alpha_{ab} n^{-1} \log n)$  and  $h_a \sim \mathrm{Nor}(\mu_a \log n, \sigma^2 I_d)$ , where  $\alpha_{ab}$  and  $\mu_a$  are independent of  $n$ . Then,
$$
I = (1 + o(1))\frac{\log n}{n}\min_{a\neq b}\sup_{t\in (0,1)}\left(\sum_{c}\pi_{c}\left[t\alpha_{b c} + (1 - t)\alpha_{a c} - \alpha_{b c}^{t}\alpha_{a c}^{1 - t}\right] + t\frac{\| \mu_{b} - \mu_{a}\|_{2}^{2}}{2\sigma^{2}}\right).
$$
In particular, the technical conditions of Theorem 1 are verified if we rule out the uninformative case where all the  $\alpha_{ab}$ 's and the  $\mu_{a}$ 's are equal to each other. Thus, exact recovery is possible if
$$
\min _ {a \neq b} \sup _ {t \in (0, 1)} \left(\sum_ {c} \pi_ {c} \left[ t \alpha_ {b c} + (1 - t) \alpha_ {a c} - \alpha_ {b c} ^ {t} \alpha_ {a c} ^ {1 - t} \right] + t \frac {\| \mu_ {b} - \mu_ {a} \| _ {2} ^ {2}}{2 \sigma^ {2}}\right) > 1.
$$
Further assuming that  $\alpha_{ab} = \alpha 1(a = b) + \beta 1(a\neq b)$  (homogeneous interactions) and  $\pi = \left(\frac{1}{K},\dots ,\frac{1}{K}\right)$  (uniform block probabilities), the expression of  $I$  simplifies to
$$
I = (1 + o (1)) \frac {\log n}{n} \left[ \frac {(\sqrt {\alpha} - \sqrt {\beta}) ^ {2}}{K} + \frac {\Delta^ {2}}{8 \sigma^ {2}} \right],
$$
Indeed, since the matrix  $\alpha = (\alpha_{ab})$  is symmetric, this implies that there exists  $a\neq b$  such that  $\exists c^{*}\in [k]\colon \alpha_{ac^{*}}\neq \alpha_{bc^{*}}$ . The function  $f(t) = \sum_{c}\pi_{c}\left[t\alpha_{bc} + (1 - t)\alpha_{ac} - \alpha_{bc}^{t}\alpha_{ac}^{1 - t}\right]$  is continuous and strictly concave, hence  $\sup_{t\in (0,1)}f(t)$  is strictly concave.
where  $\Delta = \min_{a\neq b}\| \mu_a - \mu_b\| _2$ . This last scenario recovers the recently established threshold for exact recovery in the Contextual SBM [8].
Example 3 (Semi-supervised clustering in SBM). Consider binary interactions given by
$$
f _ {a b} \sim \left\{ \begin{array}{l l} \operatorname {B e r} (\alpha n ^ {- 1} \log n) & \text {i f a = b ,} \\ \operatorname {B e r} (\beta n ^ {- 1} \log n) & \text {o t h e r w i s e ,} \end{array} \right.
$$
where  $\alpha, \beta$  are independent of  $n$ . Consider a semi-supervised model in which the vector of attributes  $Y$  is a noisy oracle of the true community labels  $z$ . More precisely, for a node  $i$  such that  $z_{i} = k$ , let
$$
h _ {k} (y) = \mathbb {P} (Y _ {i} = y)   =   \left\{ \begin{array}{l l} 1 - \eta & i f y = 0, \\ \eta_ {1} & i f y = k, \\ \frac {\eta_ {0}}{K - 1} & i f y \in [ K ] \backslash \{k \}, \end{array} \right.
$$
with  $\eta_0 + \eta_1 = \eta$ . A bit of algebra shows that exact recovery is possible if
$$
\left(\sqrt {\alpha} - \sqrt {\beta}\right) ^ {2} - \frac {2}{\log n} \log \left(1 - \eta + 2 (K - 1) ^ {- 1 / 2} \sqrt {\eta_ {0} \eta_ {1}}\right) > K.
$$
When  $\eta_0 = 0$  (perfect oracle), the condition simplifies to  $(\sqrt{a} - \sqrt{b})^2 - \frac{2\log(1 - \eta)}{\log n} > K$ . Note that the oracle term is non-negligible only if  $-\log (1 - \eta) \gtrsim \log n$ , as previously established [31]. This last condition is very strong since it implies  $\eta \gtrsim 1 - 1/n$ , and hence the oracle must provide the correct label for almost all nodes.
4 Bregman hard clustering of sparse weighted node-attributed networks
In this section, we will propose an algorithm for clustering sparse, weighted networks with node attributes. When present, the weights are sampled from an exponential family, and the node attributes also belong to an exponential family. In Section 4.1, we provide some reminder of exponential families. We derive the likelihood of the model in Section 4.2, and present the algorithm in Section 4.3.
4.1 Exponential family