SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Michael Scholkemper  
Department of Computer Science  
RWTH Aachen University  
scholkemper@cs.rwth-aachen.de
Michael T. Schaub  
Department of Computer Science  
RWTH Aachen University  
schaub@cs.rwth-aachen.de
Similar to community detection, partitioning the nodes of a network according to their structural roles aims to identify fundamental building blocks of a network. The found partitions can be used, e.g., to simplify descriptions of the network connectivity, to derive reduced order models for dynamical processes unfolding on processes, or as ingredients for various graph mining tasks. In this work, we offer a fresh perspective on the problem of role extraction and its differences to community detection and present a definition of node roles related to graph-isomorphism tests, the Weisfeiler-Leman algorithm, and equitable partitions. We study two associated optimization problems (cost functions) grounded in ideas from graph isomorphism testing and present theoretical guarantees associated with the solutions to these problems. Finally, we validate our approach via a novel "role-infused partition benchmark", a network model from which we can sample networks in which nodes are endowed with different roles in a stochastic way.
Networks are a powerful abstraction for a range of complex systems [33, 45]. To comprehend such networks we often seek patterns in their connections, e.g., core-periphery structures or densely knit communities. A complementary notion to community structure is that of a role partition of the nodes. The concept of node roles, or node equivalences, originates in social network analysis [24]. Node roles are often related to symmetries or connectivity features that can be used to simplify complex networks. Contrary to communities, even nodes that are far apart or are part of different connected components of a network can have the same role [40].
Traditional approaches to defining node roles, put forward in the context of social network analysis [8], consider exact node equivalences, based on structural symmetries within the graph structure. The earliest notion is that of structural equivalence [26], which assigns the same role to nodes if they are adjacent to the same nodes. Another definition is that of automorphic equivalence [14], which states that nodes are equivalent if they belong to the same automorphism orbits. Closely related is the idea of regular equivalent nodes [48], defined recursively as nodes that are adjacent to equivalent nodes.
However, large real-world networks often manifest in such a way that these definitions result in a vast number of different roles. What's more, the above definitions do not define a similarity metric between nodes, and it is thus not obvious how to compare two nodes that are deemed not equivalent. For example, the above definitions all have in common that nodes with different degrees also have different roles. With the aim of reducing a network's complexity, this is detrimental.
To resolve this problem in a principled way and to provide an effective partitioning of large graphs into nodes with similar roles, we present a quantitative definition of node roles in this paper. Our definition of roles is based on so-called equitable partitions (EPs), which are strongly related to the notion of regular equivalence [48]. Crucially, this not only allows us to define an equivalence, but we can also quantify the deviation from an exact equivalence numerically. Further, the notion of EPs generalizes orbit partitions induced by automorphic equivalence classes in a principled way
and thus remains tightly coupled to graph symmetries. Knowledge of EPs in a particular graph can, e.g., facilitate the computation of network statistics such as centrality measures [41]. As they are associated with certain spectral signatures, EPs are also relevant for the study of dynamical processes on networks such as cluster synchronization [34, 42], consensus dynamics [51], and network control problems [28]. They have even been shown to effectively imply an upper bound on the expressivity of Graph Neural Networks [30, 50].
Related Literature The survey by Rossi and Ahmed [38] puts forward an application-based approach to node role extraction that evaluates the node roles by how well they can be utilized in a downstream machine learning task. However, this perspective is task-specific and more applicable to node embeddings based on roles rather than the actual extraction of roles.
Apart from the already mentioned exact node equivalences originating from social network analysis, there exist numerous works on role extraction, which focus on identifying nodes with similar roles, by associating each node with a feature vector that is independent of the precise location of the nodes in the graph. These feature vectors can then be clustered to assign nodes to roles. A recent overview article [40] puts forward three categories: First, graphlet-based approaches [35, 39, 25] use the number of graph homomorphisms of small structures to create node embeddings. This retrieves extensive, highly local information such as the number of triangles a node is part of. Second, walk-based approaches [3, 11] embed nodes based on certain statistics of random walks starting at each node. Finally, matrix-factorization-based approaches [17, 20] find a rank- $r$  approximation of a node feature matrix ( $F \approx MG$ ). Then, the left side multiplicand  $M \in \mathbb{R}^{|V| \times r}$  of this factorization is used as a soft assignment of the nodes to  $r$  clusters.
Jin et al. [21] provide a comparison of many such node embedding techniques in terms of their ability to capture exact node roles such as structural, automorphic, and regular node equivalence. Detailed overviews of (exact) role extraction and its links to related topics such as block modeling are also given in [9, 10].
Contribution Our main contributions are as follows:
- We provide a principled stochastic notion of node roles, grounded in equitable partitions, which enables us to rigorously define node roles in complex networks.
- We provide a family of cost functions to assess the quality of a putative role partitioning. Specifically, using a depth parameter  $d$ , we can control how much of a node's neighborhood is taken into account when assigning roles.
- We present algorithms to minimize the corresponding optimization problems and derive associated theoretical guarantees.
- We develop a generative graph model that can be used to systematically test the recovery of roles in numerical experiments, and use this novel benchmark model to test our algorithms and compare them to well-known role detection algorithms from the literature.
2 Notation and Preliminaries
Graphs. A simple graph  $G = (V, E)$  consists of a node set  $V$  and an edge set  $E = \{uv \mid u, v \in V\}$ . The neighborhood  $N(v) = \{x \mid vx \in E\}$  of a node  $v$  is the set of all nodes connected to  $v$ . We allow self-loops  $vv \in E$  and positive edge weights  $w: E \to \mathbb{R}_+$ .
Matrices. For a matrix  $M$ ,  $M_{i,j}$  is the component in the  $i$ -th row and  $j$ -th column. We use  $M_{i, -}$  to denote the  $i$ -th row vector of  $M$  and  $M_{-j}$  to denote the  $j$ -th column vector.  $\mathbb{I}_n$  is the identity matrix and  $\mathbb{1}_n$  the all-ones vector, both of size  $n$  respectively. Given a graph  $G = (V,E)$ , we identify the node set  $V$  with  $\{1,\ldots ,n\}$ . An adjacency matrix of a given graph is a matrix  $A$  with entries  $A_{u,v} = 0$  if  $uv\notin E$  and  $A_{u,v} = w(uv)$  otherwise, where we set  $w(uv) = 1$  for unweighted graphs for all  $uv\in E$ .  $\rho (A)$  denotes the largest eigenvalue of the matrix  $A$ .
Partitions. A node partition  $C = (C_1, C_2, \dots, C_k)$  is a division of the node set  $V = C_1 \dot{\cup} C_2 \dot{\cup} \dots \dot{\cup} C_k$  into  $k$  disjoint subsets, such that each node is part of exactly one class  $C_i$ . For a node  $v \in V$ , we write  $C(v)$  to denote the class  $C_i$  where  $v \in C_i$ . We say a partition  $C'$  is coarser than  $C$  ( $C' \supseteq C$ ) if  $C'(v) \neq C'(u) \implies C(v) \neq C(u)$ . For a partition  $C$ , there exists a partition indicator matrix  $H \in \{0, 1\}^{|V| \times k}$  with  $H_{i,j} = 1 \iff i \in C_j$ .
2.1 Equitable Partitions.
An equitable partition (EP) is a partition  $C = (C_1, C_2, \dots, C_k)$  such that  $v, u \in C_i$  implies that
$$
\sum_ {x \in N (v)} [ C (x) = C _ {j} ] = \sum_ {x \in N (u)} [ C (x) = C _ {j} ] \tag {1}
$$
for all  $1 \leq j \leq k$ , where the Iverson bracket  $[C(x) = C_j]$  is 1 if  $C(x) = C_j$  and 0 otherwise. The coarsest EP (cEP) is the equitable partition with the minimum number of classes  $k$ . A standard algorithm to compute the cEP is the so-called Weisfeiler-Leman (WL) algorithm [47], which iteratively assigns a color  $c(v) \in \mathbb{N}$  to each node  $v \in V$  starting from a constant initial coloring. In each iteration, an update of the following form is computed:
$$
c ^ {t + 1} (v) = \operatorname {h a s h} \left(c ^ {t} (v), \{\{c ^ {t} (x) | x \in N (v) \} \}\right) \tag {2}
$$
where hash is an injective hash function, and  $\{\{\cdot\}\}$  denotes a multiset (in which elements can appear more than once). In each iteration, the algorithm splits classes that do not conform with eq. (1). At some point  $T$ , the partition induced by the coloring no longer changes and the algorithm terminates returning the cEP as  $\{(c^T)^{-1}(c^T(v))|v \in V\}$ . While simple, the algorithm is a powerful tool and is used as a subroutine in graph isomorphism testing algorithms [4, 29].
On a node level, the cEP carries an immense amount of information about the structure of the network surrounding each node. One can think of it as iteratively building a tree of possible paths from the start node to neighboring nodes. At the first iteration, the color  $c^1 (v)$  encodes the degree of node  $v$  in the graph. At the second iteration, it encodes the number of nodes with a certain degree that are adjacent to  $v$ . Thus, iterating this update rule (eq. (2)) encodes the structure of possible paths surrounding each node and if these trees are isomorphic for two nodes, the two remain in the same class. This tree of possible paths (also called unraveling or unrolling) is profoundly important for processes that move over the edges of a network such as dynamical systems [42, 51] and GNNs [50, 30], and even simple ones like PageRank [41] and other centrality measures [44].
The above definition is useful algorithmically but only allows us to distinguish between exactly equivalent vs. non-equivalent nodes. To obtain a meaningful quantitative metric to gauge the quality of a partition, the following equivalent algebraic characterization of an EP will be instrumental: Given a graph  $G$  with adjacency matrix  $A$  and a partition indicator matrix  $H_{\mathrm{cEP}}$  of the cEP, it holds that:
$$
A H _ {\mathrm {c E P}} = H _ {\mathrm {c E P}} \left(H _ {\mathrm {c E P}} ^ {\top} H _ {\mathrm {c E P}}\right) ^ {- 1} H _ {\mathrm {c E P}} ^ {\top} A H _ {\mathrm {c E P}} =: H _ {\mathrm {c E P}} A ^ {\pi}. \tag {3}
$$
The matrix  $AH_{\mathrm{cEP}} \in \mathbb{R}^{n \times k}$  counts in each row (for each node) the number of neighboring nodes within each class ( $C_i$  for  $i = 1, \dots, k$ ), which must be equal to  $H_{\mathrm{cEP}} A^{\pi}$  — a matrix in which each row (node) is assigned one of the  $k$  rows of the  $k \times k$  matrix  $A^{\pi}$ . Thus, from any node  $v$  of the same class  $C_i$ , the sum of edges from  $v$  to neighboring nodes of a given class  $C_k$  is equal to some fixed number — this is precisely the statement of Equation (1). The matrix  $A^{\pi}$  containing the connectivity statistics between the different classes is the adjacency matrix of the quotient graph, which can be viewed as a compressed graph with similar structural properties to the original. In particular, the adjacency matrix of the original graph inherits all eigenvalues from the quotient graph, as can be seen by direct computation. Specifically, let  $(\lambda, \nu)$  be an eigenpair of  $A^{\pi}$ , then  $AH_{\mathrm{cEP}} \nu = H_{\mathrm{cEP}} A^{\pi} \nu = \lambda H_{\mathrm{cEP}} \nu$ .
This makes EPs interesting from a dynamical systems point of view: the dominant (if unique) eigenvector is shared between the graph and the quotient graph. Hence, centrality measures such as Eigenvector Centrality or PageRank are predetermined if one knows the EP and the quotient graph [41]. For similar reasons, the cEP also provides insights into the long-term behavior of other (non)-linear dynamical processes such as cluster synchronization [42] or consensus dynamics [51]. Recently, there has been some study on relaxing the notion of "exactly" equitable partitions. One approach is to compare the equivalence classes generated by eq. (2) by computing the edit distance of the trees (the unravellings) that are encoded by these classes implicitly [19]. Another way is to relax the hash function (eq. (2)) to not be injective. This way, " buckets" of coarser equivalence classes are created [7]. Finally, using a less algorithmic perspective, one can define the problem of approximating EP by specifying a tolerance  $\epsilon$  of allowed deviation from eq. (1) and consequently asking for the minimum number of clusters that still satisfy this constraint [22]. In this paper, we adopt the opposite approach and instead specify the number of clusters  $k$  and then ask for the partition minimizing a cost function (section 4) i.e. the most equitable partition with  $k$  classes. We want to stress that while
similar, none of these relaxations coincide with our proposed approach. For clarity, we would also like to point out that the notion sometimes called almost or externally equitable partition [2] in the literature is not the same as the approximately equitable partition we propose here. The former is an exactly equitable partition (eq. (1)) that simply disregards the number of adjacent neighbors within the same class.
2.2 The Stochastic Block Model
The Stochastic Block Model (SBM) [1] is a generative network model which assumes that the node set is partitioned into blocks. The probability of an edge between a node  $i$  and a node  $j$  is then only dependent on the blocks  $B(i)$  and  $B(j)$ . Given the block labels the expected adjacency matrix of a network sampled from the SBM fulfills:
$$
\mathbb {E} [ A ] = H _ {B} \Omega H _ {B} ^ {\top} \tag {4}
$$
where  $H_{B}$  is the indicator matrix of the blocks and  $\Omega_{B(i),B(j)} = \operatorname*{Pr}((i,j)\in E(G))$  is the probability with which an edge between the blocks  $B(i)$  and  $B(j)$  occurs.
For simplicity, we allow self-loops in the network. The SBM is used especially often in the context of community detection, in the form of the planted partition model.
In this restriction of the SBM, there is only an inside probability  $p$  and an outside probability  $q$  and  $\Omega_{i,j} = p$  if  $i = j$  and  $\Omega_{i,j} = q$  if  $i \neq j$ . Usually, one also restricts  $p > q$ , to obtain a homophilic community structure i.e., nodes of the same class are more likely to connect. However,  $p < q$  (heterophilic communities) are also sometimes considered.
3 Communities vs. Roles
In this section, we more rigorously define "communities" and "roles" and their difference. To this end, we first consider certain extreme cases and then see how we can relax them stochastically. Throughout the paper, we use the term communities synonymously with what is often referred to as homophilic communities, i.e., a community is a set of nodes that is more densely connected within the set than to the outside. In this sense, one may think of a perfect community partition into  $k$  communities if the network consists of  $k$  cliques. In contrast, we base our view of the term "role" on the cEP: If  $C$  is the cEP, then  $C(v)$  is  $v$ 's perfect role. In this sense, the perfect role partition into  $k$  roles is present when the network has an exact EP with  $k$  classes. The intuition behind perfect communities and perfect roles can be seen in appendix A.
In real-world networks, such a perfect manifestation of communities and roles is rare. In fact, even if there was a real network with a perfect community (or role) structure, due to a noisy data collection process this structure would typically not be preserved. Hence, to make these concepts more useful in practice we need to relax them. For communities, the planted partition model relaxes this perfect notion of communities of disconnected cliques to a stochastic setting: The expected adjacency matrix exhibits perfect (weighted) cliques — even though each sampled adjacency matrix may not have such a clear structure. To obtain a principled stochastic notion of a node role, we argue that a planted role model should, by extension, have an exact cEP in expectation:
Definition 3.1. Two nodes  $u, v \in V$  have the same stochastic role if they are in the same class in the cEP of  $\mathbb{E}[A]$ .
The above definition is very general. To obtain a practical generative model from which we can sample networks with a planted role structure, we concentrate on the following sufficient condition. We define a probability distribution over adjacency matrices such that for a given a role partition  $C$ , for  $x, y \in C_i$  and classes  $C_j$  there exists a permutation  $\sigma : C_j \to C_j$  such that  $\operatorname{Pr}(A_{x,z} = 1) = \operatorname{Pr}(A_{y,\sigma(z)} = 1) \forall z \in C_j$ . That is: two nodes have the same role if the stochastic generative process that links them to other nodes that have a certain role is the same up to symmetry. Note that if we restrict  $\sigma$  to the identity, we recover the SBM. Therefore, we will consider the SBM as our stochastic generative process in the following.
RIP model. In line with our above discussion we propose the role-infused partition (RIP) model, to create a well-defined benchmark for role discovery, which allows us to contrast role and community
Figure 1: Example of the RIP model. It depicts (a) the expected adjacency matrix and the correct according to (b) SBM inference, (c) community detection, (d) role detection.
Input: Graph adjacency  $A\in \{0,1\}^{n\times n}$  number of classes  $k$
Output: Node assignment  $H\in \{0,1\}^{n\times k}$
1  $A = \mathrm{normalize}(A)$
2 Initialize  $H = \frac{1}{k}\mathbb{1}_n\mathbb{1}_k^T$
3 for number of steps do
4  $X = AH$
5  $H = \operatorname {cluster}(X)$
6 Return  $H$
Algorithm 1: Approximate Weisfeiler Lehman Algorithm.
structure. The RIP model is fully described by the parameters  $p\in \mathbb{R},c,k,n\in \mathbb{N},\Omega_{\mathrm{role}}\in \mathbb{R}^{k\times k}$  as follows: We sample from an SBM with parameters  $\Omega ,H_B$  (see fig. 1):
$$
\Omega_ {i, j} = \left\{ \begin{array}{l l} \Omega_ {\text {r o l e} _ {i \bmod c, j \bmod c}} & \text {i f} \left\lfloor \frac {i}{c} \right\rfloor = \left\lfloor \frac {j}{c} \right\rfloor \\ p & \text {o t h e r w i s e} \end{array} \right. \tag {5}
$$
where  $H_{B}$  corresponds to  $c \cdot k$  blocks of size  $n$ . There are  $c$  distinct communities - analogous to the planted partition model. The probability of being adjacent for nodes that are not in the same community is  $p$ . In each community, there are the same  $k$  distinct roles with their respective probabilities to attach to one another as defined by  $\Omega_{\mathrm{role}}$ . Each role has  $n$  instances in each community.
Notice that the RIP model has both a planted community structure with  $c$  communities and a planted role structure, since  $\mathbb{E}[A]$  has an exact cEP with  $k$  classes (definition 3.1). We stress that the central purpose of our model is to delineate the role recovery from community detection, i.e., community detection is not the endeavor of this paper. Rather, the planted communities within the RIP model are meant precisely as an alternative structure that can be found in the data and serve as a control mechanism to determine what structure an algorithm finds. To showcase this, consider Figure 1, which shows an example of the RIP model for  $c = 2, k = 3, n = 10, p = 0.1$ . It shows a graph that has 2 communities, each of which can be subdivided into the same 3 roles. In standard SBM inference, one would like to obtain 6 blocks - each combination of community and role being assigned its own block (1.b). In community detection, with the objective of obtaining 2 communities, the target clustering would be to merge the first 3 and the second 3 into one cluster respectively (1.c). However, the target clustering for this paper - aiming for 3 roles - is the one on the far right (1.d), combining from each community the nodes that have stochastically the same neighborhood structure.
4 Extracting Roles by Approximating the cEP
In this section, we define a family of cost functions (eq. 6, 7) that frame role extraction as an optimization problem. That is, we try to answer the question: Given a desired number  $k$  of role classes, what is the partition that is most similar to an  $EP$ ? As discussed above, searching for an exact equitable partition with a small number of classes is often not possible: It returns the singleton partition on almost all random graphs [5]. Already small asymmetries, inaccuracies, or noise in data collection can lead to a trivial cEP made up of singleton classes. As such, the cEP is neither a robust nor a particularly useful choice for noisy or even just slightly asymmetric data. Our remedy to the problem is to search for coarser partitions that are closest to being equitable.
Considering the algebraic definition of cEP (eq. 1), intuitively one would like to minimize the difference between the left- and the right-hand side (throughout the paper, we use the  $\ell_2$  norm by default and the  $\ell_1$  norm where specified):
$$
\Gamma_ {\mathrm {E P}} (A, H) = \left| \left| A H - H D ^ {- 1} H ^ {\top} A H \right| \right| \tag {6}
$$
Here  $D = \mathrm{diag}(\mathbb{1}H)$  is the diagonal matrix with the sizes of the classes on its diagonal. We note that  $HD^{-1}H^{\top} = H(H^{\top}H)^{-1}H^{\top} = HH^{\dagger}$  is the projection onto the column space of  $H$ . However, eq. (6) disregards an interesting aspect of the exact cEP. By its definition, the cEP is invariant under multiplication with  $A$ . That is,
$$
A ^ {t} H _ {\mathrm {c E P}} = H _ {\mathrm {c E P}} \left(A ^ {\pi}\right) ^ {t} \quad \text {f o r a l l} t \in \mathbb {N}
$$
This is especially interesting from a dynamical systems point of view, since dynamics cannot leave the cEP subspace once they are inside it. Indeed, even complex dynamical systems such as Graph Neural Networks suffer from this restriction [50, 30]. To address this, we put forward the following family of cost functions.
$$
\Gamma_ {d - \mathrm {E P}} (A, H) = \sum_ {t = 1} ^ {d} \frac {1}{\rho (A) ^ {t}} \Gamma_ {\mathrm {E P}} \left(A ^ {t}, H\right) \tag {7}
$$
The factor of  $\frac{1}{\rho(A)^t}$  is to rescale the impacts of each matrix power and not disproportionately enhance larger matrix powers. This family of cost functions measures how far the linear dynamical system  $t \mapsto A^t H$  diverges from a corresponding equitable dynamical system after  $d$  steps. Equivalently, it takes the  $d$ -hop neighborhood of each node into account when assigning roles. The larger  $d$ , the deeper it looks into the surroundings of a node. Note that all functions of this family have in common that if  $H_{\mathrm{cEP}}$  indicates the exact cEP, then  $\Gamma_{d - \mathrm{EP}}(A, H_{\mathrm{cEP}}) = 0$  for any choice of  $d$ .