SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the following, we consider the two specific cost functions with extremal values of  $d$  for our theoretical results and our experiments: For  $d = 1$ ,  $\Gamma_{1\text{-EP}}$  is a measure of the variance of each node's adjacencies from the mean adjacencies in each class (and equivalent to eq. (6)). As such, it only measures the differences in the direct adjacencies and disregards the longer-range connections. We call this the short-term cost function. The other extreme we consider is  $\Gamma_{\infty\text{-EP}}$ , where  $d \to \infty$ . This function takes into account the position of a node within the whole graph. It takes into account the long-range connectivity patterns around each node. We call this function the long-term cost.
In the following sections, we aim to optimize these objective functions to obtain a clustering of the nodes according to their roles. However, when optimizing this family of functions, in general, there exist examples where the optimal assignment is not isomorphism equivariant (see appendix B). As isomorphic nodes have exactly the same global neighborhood structure, arguably, they should be assigned the same role. To remedy this, we restrict ourselves to partitions compatible with the cEP when searching for the minimizer of these cost functions.
4.1 Optimizing the Long-term Cost Function
In this section, we consider optimizing the long-term objective eq. (7). This is closely intertwined with the dominant eigenvector of  $A$ , as the following theorem shows:
Theorem 4.1: Let  $\mathcal{H}$  be the set of indicator matrices  $H\in \{0,1\}^{n\times k}$  s.t.  $H\mathbb{1}_k = \mathbb{1}_n$  . Let  $A\in \mathbb{R}^{n\times n}$  be an adjacency matrix. Assume the dominant eigenvector to the eigenvalue  $\rho (A)$  of  $A$  is unique. Using the  $\ell_1$  norm in eq. (6), the optimizer
$$
O P T = \arg \min  _ {H \in \mathcal {H}} \lim  _ {d \rightarrow \infty} \Gamma_ {d - E P} (A, H)
$$
can be computed in  $\mathcal{O}(a + nk + n\log (n))$ , where  $a$  is the time needed to compute the dominant eigenvector of  $A$ .
The proof of the theorem directly yields a simple algorithm that efficiently computes the optimal assignment for the long-term cost function. Simply compute the dominant eigenvector  $v$  and then cluster it using 1-dimensional  $k$ -means. We call this EV-based clustering.
4.2 Optimizing the Short-term Cost Function
In contrast to the previous section, the short-term cost function is more challenging. In fact,
Theorem 4.2: Optimizing the short-term cost is NP-hard.
In this section, we thus look into optimizing the short-term cost function by recovering the stochastic roles in the RIP model. Given  $s$  samples  $A^{(s)}$  of the same RIP model, asymptotically, the sample mean  $\frac{1}{s}\sum_{i=1}^{s} A^{(i)} \to \mathbb{E}[A]$  converges to the expectation as  $s \to \infty$ . Thus, recovering the ground truth partition is consistent with minimizing the short-term cost in expectation.
To extract the stochastic roles, we consider an approach similar to the WL algorithm which computes the exact cEP. We call this the Approximate WL algorithm (Algorithm 1). A variant of this without the clustering step was proposed in [23]. Starting with one class encompassing all nodes, the algorithm
iteratively computes an embedding vector  $x = (x_{1},\dots,x_{k})$  for each node  $v\in V$  according to the adjacencies of the classes:
$$
x _ {i} = \sum_ {u \in N (v)} [ C ^ {(t)} (u) = C _ {i} ^ {(t)} ]
$$
The produced embeddings are then clustered to obtain the partition into  $k$  classes  $H$  of the next iteration. The clustering routine can be chosen freely. This is the big difference to the WL algorithm, which computes the number of classes on-the-fly — without an upper bound to the number of classes. Another difference is that the assignment from the previous iteration is not taken into account. This is a design choice and clustering  $X$  concatenated with  $H$  works similarly. In practice, we have found that omitting  $H$  makes it easier to reassign nodes that were assigned to the wrong cluster in a previous iteration, thus making the algorithm more robust. It also does not take away from the power of the algorithm as Morris et al. [30] show this variant to be equivalent to the original WL algorithm. The main theoretical result of this section uses average linkage for clustering:
Theorem 4.3: Let  $A$  be sampled from the RIP model with parameters  $p \in \mathbb{R}, c \in \mathbb{N}, 3 \leq k \in \mathbb{N}, n \in \mathbb{N}, \Omega_{role} \in \mathbb{R}^{k \times k}$ . Let  $H_{role}^{(0)}, \ldots, H_{role}^{(T')}$  be the indicator matrices of each iteration when performing the exact WL algorithm on  $\mathbb{E}[A]$ . Let  $\delta = \min_{0 \leq t' \leq T'} \min_{i \neq j} ||(\Omega H_{role}^{(t')})_{i,-} - (\Omega H_{role}^{(t'})})_{j,-}||$ . Using average linkage in algorithm 1 in the clustering step and assuming the former correctly infers  $k$ , if
$$
n > - \frac {9 \mathcal {W} _ {- 1} ((q - 1) \delta^ {2} / 9 k ^ {2})}{2 \delta^ {2}} \tag {8}
$$
where  $\mathcal{W}$  is the Lambert  $W$  function, then with probability at least  $q$ : Algorithm 1 finds the correct role assignment using average linkage for clustering.
The proof hinges on the fact that the number of links from each node to the nodes of any class concentrates around the expectation. Given a sufficient concentration, the correct partitioning can then be determined by the clustering step. Notice that even though we allow for the SBM to have more blocks than there are roles, the number of roles (and the number of nodes therein) is the delimiting factor here — not the overall number of nodes. Notice also that theorem 4.3 refers to exactly recovering the partition from only one sample. Typically, concentration results refer to a concentration of multiple samples from the same model. Such a result can be derived as a direct consequence of Theorem 4.3 and can be found in appendix E. The bound given in the theorem is somewhat crude in the sense that it scales very poorly as  $\delta$  decreases. This is to be expected as the theorem claims exact recovery for all nodes with high probability.
Fractional Assignments In a regime, where the conditions of Theorem 4.3 do not hold, it may be beneficial to relax the problem. A hard assignment in intermediate iterations, while possible, has shown to be empirically unstable (see experiments). Wrongly assigned nodes heavily impact the next iteration of the algorithm. As a remedy, a soft assignment — while not entirely different — has proven more robust. We remain concerned with finding the minimizer  $H$  of eq. (6) However, we no longer constrain  $H_{i,j} \in \{0,1\}$ , but relax this to  $0 \leq H_{i,j} \leq 1$ .  $H$  must still be row-stochastic - i.e.  $H \mathbb{1} = \mathbb{1}$ . That is, a node may now be fractionally assigned to multiple classes designating how strongly it belongs to each class. This remedies the above problems, as algorithms such as fuzzy  $c$ -means or Bayesian Gaussian Mixture Models are able to infer the number of clusters at runtime and must also not make a hard choice about which cluster a node belongs to. This also allows for Gradient Descent approaches like GNNs. We investigate these thoughts empirically in the experiments section.
5 Numerical Experiments
For the following experiments, we use two variants of the Approximate WL algorithm (1), one where the clustering is done using average linkage and one where fuzzy  $c$ -means is used. We benchmark the EV-based clustering (4.1) and the 2 variants of the Approximate WL algorithm as well as node classes obtained from the role2vec [3] and the node2vec [16], struc2vec [36] and rolX [17] node embeddings (called  $R2V$ ,  $N2V$ ,  $S2V$  and rolX in the figures). We retrieve an assignment from the baseline benchmark embeddings by  $k$ -means. The three X2vec node embedding techniques use autoencoders with skip-gram to compress information obtained by random walks. rolX uses recursive feature extraction [18] paired with non-negative matrix factorization. While node2vec is a somewhat
Figure 2: Role recovery on graphs sampled from the RIP model. (a-c) On the x-axis, we vary the number of samples  $s$  that are averaged to obtain the input  $A$ . The graphs used are randomly sampled from the planted partition model. On the y-axis, we report the long-term cost  $c_{20 - \mathrm{EP}}$  (a), the short-term cost (b), and the overlap of the clusterings with the ground-truth (c) over 100 runs along with their standard deviation. In (d), the average role assignment (rows reordered to maximize overlap) is shown for the number of samples  $s = 1$ .
universal node embedding technique also taking into account the community structure of a network, role2vec, struc2vec, and rolX are focussed on embedding a node due to its role. All embeddings are state-of-the-art node embedding techniques used for many downstream tasks. Further, we compare the above algorithms to the GIN [50], which is trained to minimize the short-term cost individually on each graph. The GIN uses features of size 32 that are uniformly initialized and is trained for 1000 epochs. To enable a fair comparison, we convert the fractional assignments into hard assignments by taking the class with the highest probability for each node. Experimental data and code is available here<sup>1</sup>.
Experiment 1: Planted Role Recovery. For this experiment, we sampled adjacency matrices  $A^{(i)}$  from the RIP model as described in Section 3 with  $c = k = 5$ ,  $n = 10$ ,  $p = 0.05$ ,  $\Omega_{\mathrm{role}} \in \mathbb{R}^{k \times k}$ . Each component of  $\Omega_{\mathrm{role}}$  is sampled uniformly at random i.i.d from the interval [0, 1]. We then sample  $s$  samples from this RIP model and perform the algorithms on the sample mean. The mean and standard deviation of long-term and short-term costs and the mean recovery accuracy of the ground truth and its variance are reported in Figure 2 over 100 trials for each value of  $s$ . The overlap score of the assignment  $C$  with the ground truth role assignment  $C^{\mathrm{gt}}$  is computed as:
$$
\operatorname {o v e r l a p} (C, C ^ {\mathrm {g t}}) = \max  _ {\sigma \in \text {p e r m u t a t i o n s} (\{1, \dots , k \})} \sum_ {i = 1} ^ {k} \frac {\left| C _ {\sigma (i)} \cap C _ {i} ^ {\mathrm {g t}} \right|}{\left| C _ {i} \right|}
$$
Figure 2 (d) shows the mean role assignments output by each algorithm. Since the columns of the output indicator matrix  $H$  of the algorithms are not ordered in any specific way, we use the maximizing permutation  $\sigma$  to align the columns before computing the average.
Discussion. In Figure 2 (a), one can clearly see that the EV-based clustering outperforms all other algorithms measured by long-term cost, validating Theorem 4.1. While both Approximate WL algorithms perform similarly in the cost function, the fuzzy variant has a slight edge in recovery accuracy. We can see that the tendencies for the short-term cost and the accuracy are directly adverse. The Approximate WL algorithms have the lowest cost and also the highest accuracy in recovery. The trend continues until all three X2vec algorithms are similarly bad in both measures. The GIN performs better than the X2vec algorithms both in terms of cost and accuracy. However, it mainly finds 2 clusters. This may be because of the (close to) uniform degree distribution in these graphs. On the contrary, the node2vec and role2vec detect the communities instead of the roles. This is surprising for role2vec since it aims to detect roles.
Experiment 2: Inferring the Number of Roles and Centrality. A prominent problem in practice that has been scarcely addressed in this paper so far is that the number of roles may not be known. Some algorithms — like fuzzy  $c$ -means or GIN — can infer the number of clusters while performing the clustering. In this experiment, we consider the protein dataset [6] and run the suite of algorithms
<sup>1</sup>https://git.rwth-aachen.de/netsci/publication-2023-an-optimization-based-approach-to-node-role-discovery-in-networks
Figure 3: Recovery of centralities on a real-world network. On the x-axis, the number of classes  $2 \leq k \leq 20$  that the algorithms are tasked to find is shown. On the y-axis, the mean short-term cost  $c_{\mathrm{EP}}$ , the average deviation from the cluster mean is then shown from left to right for PageRank, Eigenvector centrality, Closeness, and Betweenness over 10 trials on the protein dataset.
for varying  $2 \leq k \leq 20$ . The mean short-term cost of the assignments and its standard deviation is reported in Figure 3. Additionally for the PageRank, Eigenvector, Closeness, and Betweenness Centrality, the  $l_{1}$  deviations of each node from the mean cluster value are reported.
Discussion. In Figure 3, all algorithms show a similar trend. The costs decrease as the number of clusters increases. The elbow method yields  $k = 4, \dots, 12$  depending on the algorithm. The GIN performs much better than in the previous experiment. This may be due to the fact that there are some high-degree nodes in the dataset, that are easy to classify correctly. The converse is true for the fuzzy variant of Approximate WL that implicitly assumes that all clusters should have about the same size. The EV algorithm clusters the nodes well in terms of Eigenvector centrality, which is to be expected. Both struc2vec and rolX achieve the best results for Closeness, which is not captured well by the Approximate WL algorithms. The sudden dip for rolX at  $k = 12$  is due to the fact that rolX cannot be applied when the number of clusters is larger than the number of features. We use  $k$ -means to extract the role assignment in this case.
Experiment 3: Approximate EP as a Node Embedding. In this section, we diverge a little from the optimization-based perspective of the paper up to this point and showcase the effectiveness of the information content of the extracted roles in a down-stream learning task. This links our approach to the application-based role evaluation approach of [38] and provides a better comparison with state-of-the-art node embedding techniques. In this experiment, we use an embedding based on our role assignments: For a given adjacency matrix  $A$  with dominant eigenvector  $\nu$ , we compute the output of the Approximate WL algorithm  $H$  using average linkage. We also compute the output  $H^{\mathrm{EV}}$  of the EV-based clustering (section 4.1). Then the embedding is defined as follows:
$$
\mathrm {f} (i) = \left(H _ {i, -} ^ {\mathrm {E V}}, (A H ^ {\mathrm {E V}}) _ {i, -}, H _ {i, -}, (A H) _ {i, -}, \nu_ {i}\right)
$$
That is, the embedding is made up of the short- and long-term role assignments and role adjacencies as well as the component of the dominant eigenvector (used for the long-term optimization). We recreate the experimental design of Ribeiro et al. [36, Section 4.4]. The networks are flight networks where the nodes are airports and the edges are flights between them. In each graph, the airports are labeled with a class reflecting their activity, which is the target for this task. We then fit a "one-vs-all" logistic regression and perform a 5-fold cross-validation, tuning hyperparameters for each fold. The results are reported in Table 1.
Discussion. Table 1 shows that our embedding and the struc2vec embedding perform best by a margin over the other algorithms. Our embedding has a slight edge in each of the datasets over struc2vec. This shows that the extracted roles provide a good characterization of the roles of the nodes in the networks.
Experiment 4: Graph Embedding. We continue the line of thought of the previous experiment and consider graph-level classification tasks instead. Specifically, we consider a setting with only few learning examples. In this setting, the dimensionality of the embedding is extremely important. Therefore, we analyze which part of the above embedding is most important for the classification task, which leads to the following aggregated embedding:
$$
\mathrm {F} _ {\mathrm {a E P}} (G) = \left(| C _ {1} |, \dots , | C _ {k} |, \frac {1}{| C _ {1} |} \sum_ {i \in C _ {1}} v _ {i}, \dots , \frac {1}{| C _ {k} |} \sum_ {i \in C _ {k}} v _ {i}\right)
$$
Table 1: Performance as a node embedding on a classification task. Mean accuracy in % and standard deviation over a 5-fold cross-validation on the airport networks of [36].
ours	struc2vec	rolX	role2vec	node2vec	deg
brazil	90.0 ± 8.5	86.4 ± 3.3	78.0 ± 1.2	70.2 ± 11.1	62.8 ± 14.6	69.2 ± 0.6
usa	73.8 ± 0.3	72.6 ± 1.8	62.9 ± 0.5	64.6 ± 1.5	59.9 ± 1.1	53.7 ± 0.0
europe	79.8 ± 6.5	76.4 ± 1.5	60.8 ± 0.9	48.9 ± 2.8	64.3 ± 3.9	53.6 ± 0.0
Table 2: Few shot graph embedding performance. Mean accuracy in % over 10 runs of the EV embedding, Graph2Vec and the GIN. For each run, we randomly sample 10 data points for training and evaluate with the rest. As a comparison, the  $\mathrm{GIN + }$  is trained on  $90\%$  of the data points.
ours	G2Vec	GIN	GIN+
AIDS	95.0 ± 5.2	79.9 ± 4.4	80.0 ± 14.1	97.8 ± 1.4
ENZYMES	21.3 ± 1.7	20.3 ± 1.5	21.0 ± 1.7	60.3 ± 0.7
PROTEINS	66.5 ± 6.4	60.3 ± 3.5	59.6 ± 7.6	75.4 ± 1.3
NCI1	58.5 ± 4.0	53.9 ± 1.5	50.0 ± 1.4	82.0 ± 0.3
MUTAG	81.5 ± 7.7	66.9 ± 5.6	77.5 ± 11.1	94.3 ± 0.5
The value for  $k$  was found by a grid search over  $k \in \{2, \dots, 40\}$ . We benchmark this against the commonly used graph embedding Graph2Vec [32] and the GIN. We use graph classification tasks from the field of bioinformatics ranging from 188 graphs with an average of 18 nodes to 4110 graphs with an average of 30 nodes. The datasets are taken from [31] and were first used (in order of Table 2) in [37, 43, 13, 46, 12]. In each task, we use only 10 data points to train a 2-layer MLP on the embeddings and a 4-layer GIN. Each hidden MLP and GIN layer has 100 nodes. The Graph2Vec embedding is set to size 16, whereas the GIN receives as embedding the attributes of the nodes of the respective task. The GIN is thus informed. We also report the accuracy of a GIN that is trained on  $90\%$  of the respective data sets. The results over 10 independent runs are reported in Table 2.
Discussion. Experiment 4 has the EV embedding as the overall winner of few-shot algorithms. Our claim here is not that the EV embedding is a particularly powerful few-shot learning approach, but that the embedding carries a lot of structural information. Not only that but it is robust in the sense that few instances are sufficient to train a formidable classifier. However, it pales in comparison with the "fully trained" GIN, which performs better on every dataset.
This paper has brought forward a rigorous definition of "role" in networks that mirrors the notion of community established before. Based on this definition, we presented the RIP model as a generative model to sample networks with predetermined roles. We also defined quality measures for role assignments and proposed an optimization-based framework for role extraction. Each of the two proposed cost functions measures how well a certain characteristic of the cEP is upheld - long-term or short-term behavior. We proposed an algorithm for finding the optimal clustering for the long-term cost function and related the optimization of the other cost function to the retrieval of stochastic roles from the RIP model.
Limitations The proposed cost functions are sensitive to the degree of the nodes. In scale-free networks, for example, it can happen that few extremely high-degree nodes are put into singleton clusters and the many remaining low-degree nodes are placed into one large cluster. The issue is somewhat reminiscent of choosing a minimal min-cut when performing community detection, which may result in a single node being cut off from the main graph. A remedy akin to using a normalized cut may thus be a helpful extension to our optimization-based approach. Future work may thus consider correcting for the degree, and a further strengthening of our theoretic results.
This research has been supported by the DFG RTG 2236 "UnRAVeL" - Uncertainty and Randomness in Algorithms, Verification and Logic - and the Ministry of Culture and Science of North Rhine-Westphalia (NRW Rückkehrprogramm).
[1] E. Abbe. Community detection and stochastic block models: recent developments. The Journal of Machine Learning Research, 18(1):6446-6531, 2017.
[2] C. O. Aguilar and B. Gharesifard. Almost equitable partitions and new necessary conditions for network controllability. Autom., 80:25-31, 2017. URL https://api_semanticscholar.org/CorpusID:40989553.
[3] N. K. Ahmed, R. A. Rossi, J. B. Lee, T. L. Willke, R. Zhou, X. Kong, and H. Eldardiry. role2vec: Role-based network embeddings. Proc. DLG KDD, pages 1-7, 2019.
[4] L. Babai. Graph isomorphism in quasipolynomial time. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pages 684-697, 2016.
[5] L. Babai and L. Kucera. Canonical labelling of graphs in linear average time. In 20th Annual Symposium on Foundations of Computer Science, pages 39-46. IEEE, 1979.
[6] A. Barabási. Networkscience book datasets. URL http://networksciencebook.com/resources/data.html. Accessed: 2022-10-01.
[7] F. Bause and N. M. Kriege. Gradual weisfeiler-leman: Slow and steady wins the race. In Learning on Graphs Conference, pages 20–1. PMLR, 2022.
[8] U. Brandes. Network analysis: methodological foundations, volume 3418. Springer Science & Business Media, 2005.
[9] A. Browet. Algorithms for community and role detection in networks. PhD thesis, Catholic University of Louvain, Belgium, 2014.
[10] T. P. Cason. Role extraction in networks. PhD thesis, Catholic University of Louvain, Belgium, 2012.
[11] K. Cooper and M. Barahona. Role-based similarity in directed networks. arXiv preprint arXiv:1012.2726, 2010.
[12] A. K. Debnath, R. L. Lopez de Compadre, G. Debnath, A. J. Shusterman, and C. Hansch. Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. correlation with molecular orbital energies and hydrophobicity. Journal of medicinal chemistry, 34(2):786–797, 1991.
[13] P. D. Dobson and A. J. Doig. Distinguishing enzyme structures from non-enzymes without alignments. Journal of molecular biology, 330(4):771–783, 2003.
[14] M. G. Everett and S. P. Borgatti. Regular equivalence: General theory. Journal of mathematical sociology, 19(1):29-52, 1994.
[15] A. Gronlund, K. G. Larsen, A. Mathiasen, J. S. Nielsen, S. Schneider, and M. Song. Fast exact k-means, k-medians and bregman divergence clustering in 1d. arXiv preprint arXiv:1701.07204, 2017.
[16] A. Grover and J. Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855-864, 2016.
[17] K. Henderson, B. Gallagher, T. Eliassi-Rad, H. Tong, L. Akoglu, D. Koutra, L. Li, S. Basu, and C. Faloutsos. Rolx: Role extraction and mining in large networks. Technical report, Lawrence Livermore National Lab, Livermore, CA (United States), 2011.