Title: Community Detection Guarantees using Embeddings Learned by Node2Vec
Venue: NeurIPS 2024
Accepted: yes
Keywords: Network Embedding

Node2Vec

Community Detection

Networks
Abstract: Embedding the nodes of a large network into an Euclidean space is a common objective in modern
machine learning, with a variety of tools available. These embeddings can then be used as features for
tasks such as community detection/node clustering or link prediction, where they achieve state of the art
performance. With the exception of spectral clustering methods, there is little theoretical understanding
for commonly used approaches to learning embeddings. In this work we examine the theoretical
properties of the embeddings learned by node2vec. Our main result shows that the use of k-means
clustering on the embedding vectors produced by node2vec gives weakly consistent community recovery
for the nodes in (degree corrected) stochastic block models. We also discuss the use of these embeddings
for node and link prediction tasks. We demonstrate this result empirically for both
real and simulated networks, and examine how this relates
to other embedding tools for network data.

Community Detection Guarantees using Embeddings Learned by Node2Vec

Abstract

Embedding the nodes of a large network into an Euclidean space is a common objective in modern machine learning, with a variety of tools available. These embeddings can then be used as features for tasks such as community detection/node clustering or link prediction, where they achieve state of the art performance. With the exception of spectral clustering methods, there is little theoretical understanding for commonly used approaches to learning embeddings. In this work we examine the theoretical properties of the embeddings learned by node2vec. Our main result shows that the use of k-means clustering on the embedding vectors produced by node2vec gives weakly consistent community recovery for the nodes in (degree corrected) stochastic block models. We demonstrate this result empirically for both real and simulated networks, and examine how this relates to other embedding tools and machine learning procedures for network data.

1 Introduction

Within network science, a widely applicable and important inference task is to understand how the behavior of interactions between different units (nodes) within the network depend on their latent characteristics. This occurs within a wide array of disciplines, from sociological [14] to biological [33] networks.

One simple and interpretable model for such a task is the stochastic block model (SBM) [20], which assumes that nodes within the network are assigned a discrete community label. Edges between nodes in the network are then formed independently across all pairs of edges, conditional on these community assignments. While such a model is simplistic, various extensions have been proposed. These include the degree corrected SBM (DCSBM), used to handle degree heterogenity [23], and mixed-membership SBMs, used to allow for more complex community structures [4]. These extensions have seen a wide degree of empirical success [26, 28, 3].

A restriction of the stochastic block model and its generalizations is the requirement for a discrete community assignment as a latent representation of the units within the network. While the statistical community has previously considered more flexible latent representations [19], over the past decade, there have been significant advancements in general embedding methods for networks. These produce

general vector representations of units within a network, and can achieve start-of-the-art performance in downstream tasks for node classification and link prediction.

An early example of such a method is spectral clustering [37], which constructs an embedding of the nodes in the network from an eigendecomposition of the graph Laplacian. The k smallest non zero eigenvectors provides a k dimensional representation of each of the nodes in the network. This has been shown to allow consistent community recovery [30], however it may not be computationally feasible on the large networks which are now common. More recently, machine learning methods for producing vector representations have sought inspiration from NLP methods and the broader machine learning literature, such as the node2vec algorithm [16], graph convolutional networks [51], graph attention networks [46] and others. There are now a wide class of embedding methods which are available to practitioners which can be applied across a mixture of unsupervised and supervised settings. [8] provides a survey of relatively recent developments and [49] reviews the connection between the embedding procedure and the potential downstream task.

Embedding methods such as Deepwalk [38] and node2vec [16] consider random walks on the graph, where the probability of such a walk is a function of the embedding of the associated nodes. Given embedding vectors ̂ ω u , ω v ∈ R d of nodes u and v respectively, from graph G with vertex set V , the probability of a random walk from node u to node v is modeled as

where ⟨ x, y ⟩ is the inner product of x and y . This leads to a representation of each of the nodes in the network as a vector in d dimensional Euclidean space. This representation is then amenable to potential downstream tasks about the network. For example, if we wish to cluster the nodes in the network we can simply cluster their embedding vectors. Or, if we wish to classify the nodes in the network, we can use these embeddings to construct a multinomial classifier. We note that the sampling schemes introduced by DeepWalk and node2vec motivate more complex models such as GraphSAGE [17] and Deep Graph Infomax [47], which utilise similar node sampling schemes for learning embeddings of networks.

As such, one of the key goals of learning vector representations of the units within networks is to allow for easy use for a multitude of downstream tasks. However, there is little theoretical understanding to what information is carried within these representations, and whether they can be applied successfully and efficiently to downstream tasks. This paper aims to address this gap by examining whether learned embeddings can facilitate community detection tasks in an unsupervised setting.

1.1 Summary of main results

Our main contribution is to describe the asymptotic distribution of the embeddings learned by the node2vec procedure, and to then use this to give consistency guarantees when these embeddings are used for community detection. A simple and informal form of our results, in the scenario of a balanced two block stochastic block model (SBM), is given below:

Theorem 1. (Informal) Suppose we observe a sequence of graphs G n on n vertices arising from a two-dimensional stochastic block model: for each vertex u ∈ [ n ] we assign a community label c ( u ) ∈ { 0 , 1 } with equal probability, and then we form edges in the graph independently with probability

where ˜ p = ˜ q . Suppose that ( ̂ ω u ) are two-dimensional embeddings learned by node2vec on the above graph (where we hide the dependence on n ). Then there exists some distinct vectors η c ( u ) ∈ R 2 such that

Consequently, if we apply a k-means algorithm to the embeddings learned via node2vec, as n →∞ we will classify at least 100(1 -ϵ ) %of vertices to the correct community (up to permutation) with asymptotic probability 1 , for any ϵ > 0 .

̸

We give formal theorem statements, complete with full conditions, in Section 3; we note that our results extend to graph models beyond SBMs and are not limited to the dense regime. To give some brief intuition for the method of proof, we show that the probability that a pair ( u, v ) is positively or negatively sampled within node2vec concentrates around a function which depends only on the underlying communities c ( u ) and c ( v ) of u and v . With this, we are able to argue that the node2vec loss concentrates uniformly (in a neighborhood of their minima) around a function whose minima M ∗ is such that M ∗ u,v = ˜ M c ( u ) ,c ( v ) for some matrix ˜ M . This allows us to show that any set of embeddings which minimize the node2vec loss will converge (up to rotation) to vectors which depend only on the community label, which consequently allows us to give consistency guarantees for clustering algorithms such as k-means.

Wehighlight that while the theoretical properties of spectral clustering are well studied in the literature, there are relatively few theoretical guarantees provided for more modern embedding procedures such as node2vec. Our work provides some of the first theoretical results for models of this form. Our main contributions are the following:

We give convergence guarantees for embeddings learned via node2vec, under various sparsity regimes of (degree corrected) stochastic block models. We then use this to give weak consistency guarantees for community detection, when using the embeddings as features within a k-means clustering algorithm.

ii) Weverify the theoretical guarantees for simulated networks and examine the the performance of this procedure on real networks. We also empirically investigate important extensions of these theoretical results, relating to rates of recovery for community detection between node2vec and spectral clustering methods. We identify that as these networks grow the sampling parameters in node2vec have little impact on the performance of the proposed procedure.

The layout of the paper is as follows. In Section 2 we formulate the problem of constructing an embedding of the nodes in a network and state the criterion under which we consider community detection. In Section 3 we give the main result of this paper, the conditions under which k-means clustering of the node2vec embedding of a network gives consistent community recovery. In Section 4 we verify these theoretical results empirically and investigate potential further results. In Section 5 we summarize our contributions and consider potential extensions.

1.2 Related Works

Community detection for networks is a widely studied area with a large literature of existing work. Several notions of theoretical guarantees for community recovery are provided in [1], along with a survey of many existing approaches. There are many existing works which consider the embeddings obtained from the eigenvectors of the adjacency matrix of Laplacian of a network. For example, [30] considers spectral clustering using the eigenvectors of the adjacency matrix for a stochastic block model. Spectral clustering has provided such guarantees for a wide variety of network models, including [35, 12, 42, 32, 29].

With the more recent development of random walk based embeddings, several recent works have begun to examine the theoretical properties of such embeddings, however the treatment is limited compared to spectral embeddings. [40] study the global minimizers of the node2vec loss in the setting where d = n , viewing the problem as a matrix factorization problem. If M ∗ is the global minimizing matrix, we highlight that their results apply for any d ≥ rank( M ∗ ) . That said, this minimizer equals the entrywise logarithm of functions of the adjacency matrix A ; we note that entrywise logarithms of matrices typically blow up their rank, and that even when "in expectation" the adjacency matrix is of low rank, the actual adjacency matrix is of full rank with high probability [7]. This means that it is unlikely when d ≪ n that the global minimizer is the actual minimizer, which is the regime where embedding dimensions are considered in practice. We contrast that with our results, where we can take d = Ω( κ ) where κ is the number of communities, and obtain rigorous guarantees for the embeddings.

[52] then studies the concentration of the best rank d approximation (with respect to the Frobenius norm) of the matrix M ∗ about it's expected value under SBM and DCSBM models for node2vec with p = q = 1 only, to argue that the best rank d approximation can be used for strongly consistent community detection. We note that our results can be applied to node2vec without this restriction on

the hyperparameters. Otherwise, they give similar types of guarantees as our paper in similar sparsity regimes and with similar rates, but in stronger norms. The key difference between our work and that of [52] is that we are able to give guarantees for the the actual minimizers of the node2vec loss as soon as d = Ω( κ ) , whereas [52] use an approximation to the global minimizer, without studying the gap between this matrix and any minimizer of the node2vec loss (which is a cross-entropy loss, and therefore difficult to relate to a Frobenius norm approximation). [10] and [11] study node2vec with in the constrained setting (where U = V ), and focus on giving more abstract guarantees for the gram matrix in the setting of graphons. In [11] the norm guarantees extend only to the L 1 norm between the gram matrix of the embeddings and the minimizer, which is not sufficient to give guarantees on the individual embeddings. In [10] the norm guarantees are upgraded to the L 2 norm, albeit with less optimal rates of convergence than what we show here. Our results also give guarantees for node2vec in full generality (no restriction on p and q ) and give the calculation details for SBMs and DCSBMs to explicitly describe the asymptotic distribution in certain regimes.

2 Framework

We consider a network G consisting of a vertex set V of size n and edge set E . We can express this also using an n × n symmetric adjacency matrix A , where A uv = 1 indicates there is an undirected edge between node u and node v , with A uv = 0 otherwise, where u, v ∈ V . Given a realisation of such a network, we wish to examine models for community structure of the nodes in the network. We then examine the embeddings which can be obtained from node2vec and examine how they can be used for community detection.

2.1 Probabilistic models for community detection

The most widely studied statistical model for community detection is the Stochastic Block Model (SBM) [20]. The SBM specifies a distribution for the communities, placing each of the n nodes into one of κ communities, where these community assignments are drawn from some categorical distribution Categorical ( π ) . Writing c ( u ) ∈ [ κ ] for the community of u , the connection probabilities between edges are independent, conditional on these community assignments, with probability

where P is a κ × κ matrix of probabilities, and ρ n is the overall network sparsity (so that the network has O ( ρ n n 2 ) edges on average). As a special case, the planted-partition model considers P as being a matrix with ˜ p along its diagonal and the value ˜ q elsewhere, with κ equally balanced communities, so π = ( κ -1 , . . . , κ -1 ) . We will denote such a model by SBM ( n, κ, ˜ p, ˜ q, ρ n ) .

The most widely studied extension of the SBM is to incorporate a degree correction, equipping each node with a non negative degree parameter θ u drawn from some distribution independently of the community assignments [4]. This alters the previous model, instead giving

Degree corrected SBM models can be more appropriate for modeling the degree heterogeneity seen within communities in real world network data [23].

Performance of stochastic block models is assessed in terms of their ability to recover the true community assignments of the nodes in a network, from the observed adjacency matrix A . Given an estimated community assignment vector ˆ c ∈ [ κ ] n and the true communities z then we can compute the agreement between these two assignment vectors, up to a relabeling of c , as

̸

where S κ denotes the symmetric group of permutations σ : [ κ ] → [ κ ] . We can also control the worst-case misclassification rate across all the different communities. If C k is the set of nodes belonging to community k , then this is defined as

̸

Guarantees of the form L ( ̂ c , c ) = o p (1) as n →∞ are known as weak consistency guarantees in the community detection literature. Strong consistency considers the stronger setting where L ( ̂ c , c ) = 0 with asymptotic probability 1. [1] provides a review of results for guarantees of these forms. In this work we consider only the weak consistency setting; we highlight that stricter assumptions are necessary in order to give these type of guarantees.

2.2 Obtaining embeddings from node2vec

Machine learning methods such as node2vec aim to obtain an embedding of each node in a network. In general, for each node u two d -dimensional embedding vectors are learned, a centered representation ω i ∈ R d and a context representation ̂ ω i ∈ R d . node2vec modifies the simple random walk considered in DeepWalk [38], incorporating tuning parameters p, q which encourage the walk to return to previously sampled nodes or transition to new nodes. Formally, this is defined by sampling concurrent pairs of vertices in the second-order random walk ( X n ) n ≥ 1 defined via

where d u,s denotes the length of the shortest path between u and s , after selecting some initial two vertices. Here we consider the case where ( X 0 , X 1 ) is drawn uniformly from the set of edges in order to initialize the walk. We note that when p = q = 1 , corresponding to DeepWalk, this reduces down to a simple random walk, in which case the initial distribution samples a vertex proportionally to their degree.

A negative sampling approach is also used to approximate the computationally intractable loss function, replacing -log( P ( v | u )) in (1) with

where σ ( x ) = (1 + e -x ) -1 , the sigmoid function. The vertices n 1 , . . . , n L are sampled according to a negative sampling distribution, which we denote as P ns ( ·| u ) . This is usually chosen as the unigram distribution,

which does not depend on the current location of the random walk, u . This unigram distribution has parameter α , which is commonly chosen as α = 3 / 4 , as was used by word2vec [36]. Given this, and using (9), the loss considered by node2vec for a random walk of length k can be written as

Here we use E n l ∼ P ns ( ·| v i ) to denote the procedure to sample a draw from the negative sampling distribution, with W = 1 commonly chosen. Given this loss function, stochastic gradient updates are used to estimate the embedding vector for each node. This amounts to minimizing an empirical risk function (e.g [41, 45]), which we can write as

̸

where P n ( · ) := P ( · | G n ) , and P = P ( G n ) and N = N ( G n ) are sets of positive and negative samples respectively. We consider a sequence of graphs G n with |V| = n and study the behavior of this loss function when n is large. To be explicit, P n (( i, j ) ∈ P ) denotes the probability (conditional on a realization of the graph) that the vertices ( i, j ) appear concurrently within a random walk of length k , and P n (( i, j ) ∈ N ) denotes the probability that ( i, j ) is selected as a pair of edges through the negative sampling scheme (conditional on the random walk process in the first stage).

The loss depends on two matrices U, V ∈ R n × d , with u i , v j ∈ R d denoting the i -th and j -th rows of U and V respectively. The rows of U correspond to the "centered representations" of each node, while the rows of V correspond to the "context representation" (borrowing the terminology used by e.g Word2Vec). In practice we can constrain the embedding vectors u i and v i to be equal if we wish; we will consider both approaches in this paper. (If these are not constrained to be equal, the centered representation is commonly used for downstream tasks.) We highlight Equation (12) is defined only as a function of UV T . There are two potential approaches to deal with this. We can regularize the objective function to enforce U T U = V T V , which does not change the matrix UV T that we recover [53]. Alternatively, if these matrices are initialized to be balanced then they will remain balanced during the gradient descent procedure [34]. Either procedure can be used to implicitly enforce U T U = V T V , which reduces the symmetry group of ( U, V ) → UV T to the orthogonal group. Similarly, if we constrain U = V then we obtain the same reduction.

2.3 Using embeddings for community detection

Having learned embedding vectors ω i for each node, we seek to use them for a further task, such as node clustering or classification. For community detection a natural procedure is to perform k-means clustering on the embedding vectors, using the estimated cluster assignments as inferred communities. k-means clustering [18] aims to find k vectors x 1 , . . . , x k ∈ R d which minimize the within cluster sum of squares. This can be formulated in terms of a matrix X ∈ R k × d and a membership matrix Θ ∈ { 0 , 1 } n × k where each row of Θ has exactly k -1 zero entries. Then the k-means clustering objective can be written as

where ̂ Ω ∈ R n × d is the matrix whose rows are the ̂ ω i . The non-zero entries in each row of Θ gives the estimated community assignments. Finding exact minima to this minimization problem is NP-hard in general [5]. For theoretical purposes, we will give guarantees for any (1 + ϵ ) -minimizer to the above problem, which returns any pair ( ̂ Θ , ̂ X ) for which L k-means ( ̂ Θ , ̂ X ) ≤ (1 + ϵ ) min Θ ,X L k-means (Θ , X ) , and can be solved efficiently [25].

3 Results

Within this section, we give theoretical results which allow us to describe what happens when we use node2vec to learn embedding vectors for each node in the network, and then use these as features for a k-means clustering algorithm to perform community detection. Throughout, we assume that we observe a sequence of graphs ( G n ) n ≥ 1 on n vertices drawn from a probabilistic model and fit a node2vec model, according to one of the three scenarios below:

(i) We use DeepWalk ( p = q = 1 in node2vec), and the graph is drawn according to a SBM with ρ n ≫ log( n ) /n ;

(ii) We use node2vec, and the graph is drawn according to a SBM with ρ n = n -α for some α < α ′ , where α ′ depends on node2vec's hyperparameters;

(iii) We use DeepWalk and a unigram parameter of α = 1 , and the graph is drawn according to a DCSBM with ρ n ≫ log( n ) /n where the degree heterogeneity parameters θ u ∈ [ C -1 , C ] for some C < ∞ .

All probabilistic statements below are with respect to the joint law of G n and the sampling which occurs to form the node2vec loss. All proofs are deferred to the Appendix. There we also provide extensions for the tasks of node classification and link prediction.

3.1 Asymptotic distribution of the embeddings

We begin with a result which describes the asymptotic distribution of the gram matrices formed by the embeddings which minimize the loss L n ( U, V ) over matrices U, V ∈ R n × d .

Theorem 2. There exist constants ˜ A ∞ and ˜ A 2 , ∞ (depending on π, P and the sampling scheme) and a matrix M ∗ ∈ R κ × κ (also depending on π, P and the sampling scheme) such that when d ≥ rk( M ∗ ) ,

for any minimizer ( U ∗ , V ∗ ) of L ( U, V ) over X × X where

we have that

where C is a constant depending on the (DC)SBM parameters, the node2vec hyperparameters, ˜ A ∞ and ˜ A 2 , ∞ . In the case where we constrain U = V within node2vec, the same result holds under scenarios i) and ii). Moreover, under all scenarios we can allow the number of communities κ to grow with n - provided κ = o ( nρ n ) - and still maintain consistency as n →∞ .

To give some intuition, we describe the form of M ∗ when the graph arises from a SBM ( n, κ, ˜ p, ˜ q, ρ n ) model when using DeepWalk. In this case, we show in the Appendix that

for some constants α and β and δ lm is the Kronecker delta. In the unconstrained case we have that

In the constrained case we instead have that β ∗ = -α ∗ / ( κ -1) , and that α ∗ is a function of p/q which is non-negative iff p > q , and equals zero when p ≤ q . With regards to the constants ˜ A ∞ and ˜ A 2 , ∞ , we have that ∥ M ∗ ∥ ∞ ≤ O ( | log( p/q ) | ) . Additionally, it is possible to write M ∗ = U ∗ M ( V ∗ M ) T where ∥ U ∗ M ∥ 2 , ∞ and ∥ V ∗ M ∥ 2 , ∞ are upper bounded by O ( | log( p/q ) | 1 / 2 ) . In particular, this means that ˜ A ∞ and ˜ A 2 , ∞ do not have any implicit dependence on n or κ , and so the constant in Theorem 2 is not affecting the rate here.

While Theorem 2 gives guarantees from the gram matrices formed by the embeddings, in practice we want guarantees for the actual embedding vectors themselves. For convenience we suppose that the embedding dimension d is chosen exactly to be the rank of M ∗ ; upon doing so, we can then obtain guarantees for the embedding vectors themselves. We recall that in the unconstrained case, we implicitly suppose that we find embedding matrices U ∗ and V ∗ which are balanced in that ( U ∗ ) T U ∗ = ( V ∗ ) T V ∗ .

Theorem 3. Suppose that the conclusion of Theorem 2 holds, and further suppose that d equals the rank of the matrix M ∗ . Then there exists a matrix ˜ U ∗ ∈ R κ × d such that

3.2 Guarantees for community detection

With Theorem 3, we are now in a position to give guarantees for machine learning methods which use the embeddings as features for a downstream task. We only discuss using the embeddings for clustering; in Appendix D.2 we discuss what can be said for other downstream tasks.

Theorem 4. Suppose that we have embedding vectors u ∗ i ∈ R d for i ∈ [ n ] such that

for some rate function r n → 0 as n →∞ and vectors η l ∈ R d for l ∈ [ κ ] . Moreover suppose that δ := min l = k ∥ ˜ u ∗ l -˜ u ∗ k ∥ 2 > 0 . Then if ˆ c ( i ) is the community assignment of node i produced by applying a (1 + ϵ ) -approximate k-means clustering with k = κ to the matrix whose columns are the u ∗ i , we have that L ( c , ˆ c ) = O p ( δ -2 r n ) and ˜ L ( c , ˆ c ) = O p ( δ -2 r n ) . In the case where the RHS of (16) is only o p (1) instead, then instead L ( c , ˆ c ) and ˜ L ( c , ˆ c ) are δ -2 o p (1) .

̸

Within the SBM ( n, κ, ˜ p, ˜ q, ρ n ) model, we can show in the unconstrained case that δ 2 = Θ( | log(˜ p/ ˜ q ) | ) , and in the constrained case that δ 2 = Θ((˜ p/ ˜ q )) . As a result, this suggests that as ˜ p/ ˜ q approaches 1 , the task of distinguishing the communities becomes more difficult. This is inline with basic intuition, along with our experimental results in Section 4. We note that, due to the nature of the embedding vectors, for any proportion of vertices arbitrarily close to 1, the nodes will, with high probability for sufficiently large n , be separated in the embedding space according to their community assignments. This separation allows clustering methods, such as DBSCAN, to accurately recover the communities of these nodes also.

Recall that from the discussion before, we know that M ∗ equals the zero matrix in the constrained regime when ˜ p ≤ ˜ q (and therefore the embeddings asymptotically contain no information about the network). As in the case where ˜ p > ˜ q we can show that δ > 0 , we get the immediate corollary.

Corollary 5. Under scenario (i), suppose the embedding vectors learned through the node2vec loss are obtained by constraining the embedding matrices U = V . Then the embeddings can be used for weakly consistent recovery of the communities if and only if ˜ p > ˜ q .

As a result, the constrained model can be disadvantageous if used without a-priori knowledge of the network beforehand (in that within-community connections outnumber between-community connections), even though it avoids interpretability issues about which embedding vector should be used as single representation for the node.

4 Experiments

In this section we provide simulation and real data experiments to empirically validate the previous theoretical results. We demonstrate the performance, in terms of community detection, of k-means clustering of the embedding vectors learned by node2vec, for both the regular and degree corrected stochastic block model. We also investigate the role of the negative sampling parameter α and the node2vec tuning parameters p and q , before examining performance on a real network with known community structure.

We first simulate data from the planted partition stochastic block model, SBM ( n/κ, κ, ˜ p, ˜ q, ρ n ) . We consider ˜ q = ˜ pβ for a range of values of β ≪ 1 , giving varying strengths of associative community structure. In each setting we vary both the number of true communities present and the number of nodes in each community, considering n = 200 to n = 5000 and K = 2 , 3 , 4 , 5 . We use node2vec to construct an embedding of the nodes in the network. 1 We use an embedding dimension of 64 and do not modify other default tuning parameters for the embedding procedure unless specified, so that p = q = 1 . We investigate the role of these tuning parameters below, allowing them to vary as is considered in node2vec. We pass these embedding vectors into k-means clustering, where k = κ , the true number of communities present in the network. This estimates a community assignment for each of the nodes in the network.

To evaluate the performance of our procedure, we compute the proportion of nodes correctly classified, up to permutation of the community assignments. For each simulation setting we perform 10 replications. We show the resulting estimates in Figure 1(a), for the relatively sparse setting where ρ n = log( n ) /n . For all settings, the proportion of nodes assigned to the correct community by k-means clustering of the node2vec embeddings is high, particularly when the ratio of the between to within community edge probabilities, β , is small. As expected, as we increase the number of nodes in the network, a larger proportion of nodes are correctly recovered. We examine the empirical rate of convergence of this procedure in the Appendix. This appears to be approximately super-linear for dense networks and sub-linear for relatively sparse networks. Compared to the results of [50], this indicates that node2vec may be supoptimal. In the Appendix we also show community recovery using normalized mutual information (NMI) [9]. We also see good performance.

We can similarly evaluate the performance of node2vec for data generated from a degree corrected SBM (DC-SBM). To generate such networks we modify the simulation setting used by [15]. We generate the degree correction parameters θ u = | Z u | +1 -(2 π ) -1 / 2 where Z u ∼ N (0 , σ = 0 . 25) and incorporate these into the SBM ( n/κ, κ, ˜ p, ˜ q, ρ n ) considered previously. Two nodes u and v in the same community will have connection probability θ u θ v ρ n ˜ p while for nodes in different communities

K=2

K=2

1.00

1.00

0.75

0.75

0.50

0.50

Prop.Recovered

0.25

q/p

Recovered

0.25

q/p

+0.01

+0.01

0.00

+0.02

0.00

+0.02

K= 4

K=5

+0.05

K = 4

K=5

+0.05

1.00

0.1

Prop.F

1.00

0.1

0.15

0.15

0.2

+0.2

0.75

0.75

0.50

0.50

0.25

0.25

0.00

0.00

100

500

1000100

500

1000

100

500

1000

100

500

1000

Nodes per Community

Nodes per Community

(a) A relatively sparse SBM.

(b) A degree corrected relatively sparse SBM.

it will be θ u θ v ρ n ˜ q . We again learn an embedding of the nodes using a default implementation of node2vec and cluster these embedding vectors using k-means clustering. We show the corresponding results, in terms of the proportion of the nodes assigned to their correct communities under this setting in Figure 1(b). As expected, the degree corrections make community recovery somewhat more challenging however as we increase the number of nodes in the network, we are able to correctly recover a high proportion of nodes.

We next wish to examine empirically the role of the unigram parameter α of Equation (10), and how this affects community detection. While the previous theoretical results require α = 1 for weak consistency of community recovery in the DC-SBM, we investigate if good empirical performance is possible with other choices of this parameter. We consider the DC-SBM simulation described previously, where we now vary α ∈ {-1 , 0 , 0 . 25 , 0 . 5 , 0 . 75 , 1 } when learning the node embeddings. For each of these settings (with all other parameters as before) we consider the proportion of nodes correctly recovered. We show this result for networks with κ = 2 communities in Figure 2. These experiments indicate similar performance for a range of values of α . Further work is needed to confirm the guarantees do indeed extend to these alternative choices of α , and we investigate this for real networks in Section A of the appendix.

We also investigate the role of the node2vec tuning parameters p and q on performance. For κ = 2 we consider β = 0 . 01 and β = 0 . 2 , giving networks with strong and weak associative community structure respectively. We simulate from the previous relatively sparse DC-SBM with varying numbers of nodes and fit node2vec, using p, q ∈ { 0 . 5 , 1 , 2 } . As the number of nodes in the network increases all choices of p and q give similar good performance for both choices of β . This indicates that the impact of these sampling parameters becomes limited as the networks become sufficiently large. We provide further discussion and a visualization of this result in Appendix A.

Finally, we briefly examine the performance of our community detection procedure on the political blog data collected by [2]. As highlighted by [23], degree heterogeneity makes community recovery challenging for methods which do not account for this. We see similar performance if we cluster using a Gaussian mixture model rather than k-means clustering. In particular, spectral clustering struggles regardless of the graph Laplacian used. Our procedure shows excellent community recovery (average NMI of 0.75) for a range of embedding dimensions and unigram parameter settings as shown in Figure 3, with further details and an additional real network example in Appendix A.

α= -1

α=0

α=0.25

1.0

0.9

0.8

.Recovered

a/p

0.7

+0.01

1

0.02

α=0.5

α = 0.75

α=1

0.05

Prop.

1.0

+

0.1

0.15

0.9

0.2

0.8

0.7

100

500

1000100500

1000100500

1000

Nodes per Community

5 Conclusion and Future Work

In this work we consider the theoretical properties of node embeddings learned from node2vec. We show, when the network is generated from a (degree corrected) stochastic block model, that the embeddings learned from DeepWalk and node2vec converge asymptotically to vectors depending only on their community assignment. As a result, we show that K-means clustering of the node2vec embedding vectors can provide weakly consistent estimates of the true community assignments of the nodes in the network. We verify these results empirically using simulated networks.

There are several important future directions which can extend this work. One direction is in extending the recovery results within the degree corrected SBM to the full range of hyperparmaeters for node2vec, as our simulation studies indicate that a more general result may hold. There is also the matter of increasing the strength of our results to give better rates and strongly consistent community detection; one possible avenue of exploration would be to see whether our results and the results of [52] could be combined to achieve this. Another improvement would be to study the behavior of the random walk on the graph in the sparse regime, although this would require a generalization of e.g the result of [13]. We have also not considered the task of estimating κ , the number of communities in a SBM model, using the embeddings obtained by node2vec. This has been considered for alternative approaches to community detection, ([22, 27] are some recent results) but not in the context of a general embedding of the nodes. Finally, there is a desire to obtain consistency results for more recent and complex network embedding methods, such as [17] and [47].

References

Emmanuel Abbe. Community detection and stochastic block models: recent developments. The Journal of Machine Learning Research , 18(1):6446-6531, 2017.

Lada A. Adamic and Natalie Glance. The political blogosphere and the 2004 u.s. election: divided they blog. In Proceedings of the 3rd International Workshop on Link Discovery , LinkKDD '05, page 36-43, New York, NY, USA, 2005. Association for Computing Machinery. ISBN 1595932151. doi: 10.1145/1134271.1134277. URL https://doi.org/10.1145/1134271.1134277 .

Edoardo M Airoldi, David M. Blei, Stephen E. Fienberg, Eric P. Xing, and Tommi Jaakkola. Mixed membership stochastic block models for relational data with application to protein-protein interactions. In Proceedings of the international biometrics society annual meeting , volume 15, page 1, 2006.

1.00

0.75

0.50

0.25

0.00

Node2Vec

Spectral Clustering

Edoardo M Airoldi, David M. Blei, Stephen E. Fienberg, and Eric P. Xing. Mixed membership stochastic blockmodels. Advances in neural information processing systems , 21, 2008.

Daniel Aloise, Amit Deshpande, Pierre Hansen, and Preyas Popat. NP-hardness of Euclidean sum-ofsquares clustering. Machine Learning , 75(2):245-248, May 2009. ISSN 1573-0565. doi: 10.1007/ s10994-009-5103-0. URL https://doi.org/10.1007/s10994-009-5103-0 .

Béla Bollobás. Threshold functions for small subgraphs. In Mathematical Proceedings of the Cambridge Philosophical Society , volume 90, pages 197-206. Cambridge University Press, 1981.

Kevin P. Costello and Van H. Vu. The rank of random graphs. Random Structures & Algorithms , 33(3): 269-285, 2008. doi: https://doi.org/10.1002/rsa.20219. URL https://onlinelibrary.wiley.com/ doi/abs/10.1002/rsa.20219 .

Peng Cui, Xiao Wang, Jian Pei, and Wenwu Zhu. A survey on network embedding. IEEE transactions on knowledge and data engineering , 31(5):833-852, 2018.

Leon Danon, Albert Diaz-Guilera, Jordi Duch, and Alex Arenas. Comparing community structure identification. Journal of statistical mechanics: Theory and experiment , 2005(09):P09008, 2005.

Andrew Davison. Asymptotics of ℓ _ 2 regularized network embeddings. Advances in Neural Information Processing Systems , 35:24960-24974, 2022.

Andrew Davison and Morgane Austern. Asymptotics of network embeddings learned via subsampling. Journal of Machine Learning Research , 24(138):1-120, 2023.

Shaofeng Deng, Shuyang Ling, and Thomas Strohmer. Strong consistency, graph laplacians, and the stochastic block model. The Journal of Machine Learning Research , 22(1):5210-5253, 2021.

Jian Ding, Eyal Lubetzky, and Yuval Peres. Anatomy of the giant component: The strictly supercritical regime. Eur. J. Comb. , 35:155-168, January 2014. ISSN 0195-6698. doi: 10.1016/j.ejc.2013.06.004. URL https://doi.org/10.1016/j.ejc.2013.06.004 .

Linton Freeman. The development of social network analysis. A Study in the Sociology of Science , 1(687): 159-167, 2004.

Chao Gao, Zongming Ma, Anderson Y. Zhang, and Harrison H. Zhou. Community detection in degreecorrected block models. The Annals of Statistics , 46(5):2153 - 2185, 2018. doi: 10.1214/17-AOS1615. URL https://doi.org/10.1214/17-AOS1615 .

Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining , pages 855-864, 2016.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

John A Hartigan and Manchek A Wong. Algorithm as 136: A k-means clustering algorithm. Journal of the royal statistical society. series c (applied statistics) , 28(1):100-108, 1979.

Peter D Hoff, Adrian E Raftery, and Mark S Handcock. Latent space approaches to social network analysis. Journal of the american Statistical association , 97(460):1090-1098, 2002.

Paul W Holland, Kathryn Blackmond Laskey, and Samuel Leinhardt. Stochastic blockmodels: First steps. Social networks , 5(2):109-137, 1983.

Roger A. Horn and Charles R. Johnson. Matrix Analysis . Cambridge University Press, New York, NY, USA, 2nd edition, 2012. ISBN 978-0-521-54823-6.

Jiashun Jin, Zheng Tracy Ke, Shengming Luo, and Minzhe Wang. Optimal estimation of the number of network communities. Journal of the American Statistical Association , pages 1-16, 2022.

Brian Karrer and Mark EJ Newman. Stochastic blockmodels and community structure in networks. Physical review E , 83(1):016107, 2011.

Vladimir Koltchinskii and Evarist Giné. Random Matrix Approximation of Spectra of Integral Operators. Bernoulli , 6(1):113-167, 2000. ISSN 1350-7265. doi: 10.2307/3318636. URL http://www.jstor.org/ stable/3318636 . Number: 1 Publisher: International Statistical Institute (ISI) and Bernoulli Society for Mathematical Statistics and Probability.

Amit Kumar, Yogish Sabharwal, and Sandeep Sen. Linear Time Algorithms for Clustering Problems in Any Dimensions. In Luís Caires, Giuseppe F. Italiano, Luís Monteiro, Catuscia Palamidessi, and Moti Yung, editors, Automata, Languages and Programming , Lecture Notes in Computer Science, pages 1374-1385, Berlin, Heidelberg, 2005. Springer. ISBN 978-3-540-31691-6. doi: 10.1007/11523468_111.

Pierre Latouche, Etienne Birmelé, and Christophe Ambroise. Overlapping stochastic block models with application to the French political blogosphere. The Annals of Applied Statistics , 5(1):309 - 336, 2011. doi: 10.1214/10-AOAS382. URL https://doi.org/10.1214/10-AOAS382 .

Can M Le and Elizaveta Levina. Estimating the number of communities by spectral methods. Electronic Journal of Statistics , 16(1):3315-3342, 2022.

Sirio Legramanti, Tommaso Rigon, Daniele Durante, and David B Dunson. Extended stochastic block models with application to criminal networks. The Annals of Applied Statistics , 16(4):2369, 2022.

Jing Lei. Network representation using graph root distributions. The Annals of Statistics , 49(2):745 - 768, 2021.

Jing Lei and Alessandro Rinaldo. Consistency of spectral clustering in stochastic block models. The Annals of Statistics , 43(1):215 - 237, 2015. doi: 10.1214/14-AOS1274. URL https://doi.org/10. 1214/14-AOS1274 .

Jure Leskovec and Andrej Krevl. SNAP Datasets: Stanford large network dataset collection. http: //snap.stanford.edu/data , June 2014.

Keith D Levin, Fred Roosta, Minh Tang, Michael W Mahoney, and Carey E Priebe. Limit theorems for out-of-sample extensions of the adjacency and laplacian spectral embeddings. The Journal of Machine Learning Research , 22(1):8707-8765, 2021.

Feng Luo, Yunfeng Yang, Chin-Fu Chen, Roger Chang, Jizhong Zhou, and Richard H Scheuermann. Modular organization of protein interaction networks. Bioinformatics , 23(2):207-214, 2007.

Cong Ma, Yuanxin Li, and Yuejie Chi. Beyond procrustes: Balancing-free gradient descent for asymmetric low-rank matrix sensing. IEEE Transactions on Signal Processing , 69:867-877, 2021.

Shujie Ma, Liangjun Su, and Yichong Zhang. Determining the number of communities in degree-corrected stochastic block models. The Journal of Machine Learning Research , 22(1):3217-3279, 2021.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems , 26, 2013.

Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems , 14, 2001.

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 701-710, 2014.

Yannik Pitcan. A note on concentration inequalities for u-statistics, 2019.

Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Kuansan Wang, and Jie Tang. Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vec. In Proceedings of the eleventh ACM international conference on web search and data mining , pages 459-467, 2018.

Herbert Robbins and Sutton Monro. A Stochastic Approximation Method. The Annals of Mathematical Statistics , 22(3):400-407, September 1951. ISSN 0003-4851, 2168-8990. doi: 10.1214/aoms/1177729586. URL https://projecteuclid.org/journals/annals-of-mathematical-statistics/ volume-22/issue-3/A-Stochastic-Approximation-Method/10.1214/aoms/1177729586. full . Number: 3 Publisher: Institute of Mathematical Statistics.

P Rubin-Delanchy, CE Priebe, M Tang, and J Cape. A statistical interpretation of spectral embedding: the generalised random dot product graph. arxiv e-prints. arXiv preprint arXiv:1709.05506 , 2017.

Michel Talagrand. Upper and Lower Bounds for Stochastic Processes: Modern Methods and Classical Problems . Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge / A Series of Modern Surveys in Mathematics. Springer-Verlag, Berlin Heidelberg, 2014. ISBN 978-3-642-54074-5. doi: 10.1007/ 978-3-642-54075-2. URL https://www.springer.com/gp/book/9783642540745 .

Stephen Tu, Ross Boczar, Max Simchowitz, Mahdi Soltanolkotabi, and Ben Recht. Low-rank Solutions of Linear Matrix Equations via Procrustes Flow. In Proceedings of The 33rd International Conference on Machine Learning , pages 964-973. PMLR, June 2016. URL https://proceedings.mlr.press/v48/ tu16.html . ISSN: 1938-7228.

Victor Veitch, Morgane Austern, Wenda Zhou, David M Blei, and Peter Orbanz. Empirical risk minimization and stochastic gradient descent for relational data. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1733-1742. PMLR, 2019.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Petar Veliˇ ckovi´ c, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. Deep Graph Infomax. arXiv:1809.10341 [cs, math, stat] , December 2018. URL http://arxiv.org/ abs/1809.10341 . arXiv: 1809.10341.

V. H. Vu. Concentration of non-lipschitz functions and applications. Random Structures & Algorithms , 20(3):262-316, 2002. doi: https://doi.org/10.1002/rsa.10032. URL https://onlinelibrary.wiley. com/doi/abs/10.1002/rsa.10032 .

Owen G Ward, Zhen Huang, Andrew Davison, and Tian Zheng. Next waves in veridical network embedding. Statistical Analysis and Data Mining: The ASA Data Science Journal , 14(1):5-17, 2021.

Anderson Ye Zhang. Fundamental limits of spectral clustering in stochastic block models. arXiv preprint arXiv:2301.09289 , 2023.

Si Zhang, Hanghang Tong, Jiejun Xu, and Ross Maciejewski. Graph convolutional networks: a comprehensive review. Computational Social Networks , 6(1):1-23, 2019.

Yichi Zhang and Minh Tang. A theoretical analysis of deepwalk and node2vec for exact recovery of community structures in stochastic blockmodels. IEEE Transactions on Pattern Analysis and Machine Intelligence , 46(2):1065-1078, 2024. doi: 10.1109/TPAMI.2023.3327631.

Zhihui Zhu, Qiuwei Li, Gongguo Tang, and Michael B Wakin. The global optimization geometry of low-rank matrix optimization. IEEE Transactions on Information Theory , 67(2):1308-1331, 2021.

Appendix - Community Detection Guarantees using Embeddings Learned by Node2Vec

The Appendix consists of the proofs of the results stated within the paper, along with some extra discussions which would detract from the flow of the main paper. We also provide some additional simulation results relating to node classification, and further simulated and real data experiments examining community detection.

A Additional Experimental Results

Here we provide additional details describing the experimental results presented in the main paper. We also describe additional experiments. All experiments were run on a computing cluster utilising 4 cores of an Intel E5-2683 v4 Broadwell 2.1GHz CPU or similar with 2 GB of memory per core. Each individual experimental run required at most 2 hours of computing time. All experiments, including initial preliminary experiments, required approximately 25k CPU hours. All code required to reproduce all results is included in the code repository in the supplemental files.

Additional Simulation, Node Classification We provide a simple experiment to support the theoretical results on node classification demonstrated in Section D of the appendix. We simulate data from a SBM ( n/κ, κ, ˜ p, ˜ q, ρ n ) as before with ˜ q = ˜ pβ as in the main text. We learn an embedding of each node using node2vec with embedding dimension of 64 and all other parameters set at their default values. We then use the true community labels of 10% of these nodes to train a (multinomial) logistic regression classifier, and predict the class label for the remaining 90% of nodes in the network. We examine the performance of this classification tool using the node2vec embeddings in terms of classification accuracy. We show these results in Figure S1 for ρ n = log( n ) /n , with 10 simulations for each setting, with the mean across these simulations and error bars indicating one standard error. This classifier has excellent accuracy at predicting the labels of other nodes.

K=2

1.00-

0.75

Classification Accuracy

0.50

a/p

0.25

10.01

0.00

0.02

K=4

K=5

0.05

1.00

全

0.1

0.15

0.75

0.2

0.50

0.25-

0.00

100

500

1000

100

500

1000

Nodes per Community

Additional Results, Community Detection Here we include additional simulation results which were omitted from the main text. In particular, for the simulations considered in the main manuscript we now examine the community recovery performance in terms of the normalized mutual information [9]. We show the average NMI score across these simulations, along with error bars corresponding to one standard error. In each case, the NMI metric is similar to the proportion of nodes correctly recovered. As we increase the number of nodes this performance improves.

K=2

1.00

0.75-

0.50

a/p

0.25

+10.01

0.00

NMI

0.02

K=4

K=5

0.05

1.00

0.1

0.15

0.75

0.2

0.50

0.25-

0.00

100

500

1000100

500

1000

Nodes per Community

K=2

1.00

0.75

车

0.50-

a/p

0.25

+10.01

0.00

NMI

$

0.02

K=4

K=5

0.05

1.00-

0.1

0.15

0.75-

0.2

0.50

0.25

0.00

100

500

1000

100

500

1000

Nodes per Community

Rates of Convergence We can also investigate the empirical convergence of these methods. Here, we consider the same simulated SBM data as above, and examine the convergence in the proportion of nodes correctly recovered, as we increase the number of nodes in the network, for κ = 2 , 3 , 4 , 5 . We empirically investigate this convergence using a log-log plot, which is shown in Figure S5 for a relatively sparse SBM. Our node2vec procedures demonstrates empirical convergence which is super-linear for dense networks while being sub-linear for relatively sparse networks.

Varying the node2vec walk parameters We also wish to examine the performance of our proposed clustering procedure when the parameters of the random walk are varied. While p and q are both commonly chosen to be 1 , resulting in a simple random walk, other values are possible. We consider data simulated from the relatively sparse DC-SBM considered previously with κ = 2 communities and consider the within between community probability ratio β = . 01 and β = 0 . 2 , corresponding to an easier and harder setting to recover the communities respectively. We then consider p, q ∈ { 0 . 5 , 1 , 2 } , the common possible values and vary the number of nodes in each community as before. For each

α= -1

α=0

α = 0.25

0.75-

0.50

a/p

0.25-

+

0.01

NMI

0.02

α = 0.5

α = 0.75

α= 1

0.05

0.1

0.15

0.75

0.2

0.50

0.25-

100

500

1000100500

1000100

500

1000

Nodes per Community

0.100

a/p

1

0.01

Error

0.02

0.010-

0.05

0.1

0.15

0.2

0.001

300

1000

3000

Number of Nodes

of these settings we perform community detection using node2vec and spectral clustering. When β = 0 . 01 weobtain excellent community recovery for all values of p and q , as shown in Figure S6(a). When β = 0 . 2 community recovery is more challenging for small networks for all values of p and q . As the number of nodes increases, Figure S6(b) shows that all choices of p and q result in good performance.

A.1 Performance on Real Networks

We wish to further examine the performance of this community detection procedure for real networks, with known community structure. We also wish to compare this procedure to spectral clustering, which is widely used in practice for community detection. We use two publicly available networks containing known community structure. We first consider a network of emails between 1005 members

(a)

β

= 0

.

01

(b)

β

= 0

.

2

n=100

n=100

1.00

1.00

0.75

0.75

0.50

0.50

n=200

n=200

Recovered

1.00

Recoverec

品

1.00

0.75

0.75

0.50

0.50

n=500

n=500

1.00

1.00

rop.

0.75

rop.

0.75

P

0.50

P

0.50

n=1000

n=1000

1.00

1.00

0.75

0.75

0.50

0.50

p=0.5，p=0.5,p=0.5，p=1,

p=1,

p=1,

p=2,

p=2,

p=2,

p=0.5，p=0.5，p=0.5，

p=1,

p=1,

p=1,

p=2,

p=2,

p=2,

q=2q=0.5

q=1

q=2

q=0.5

q=1

q=2

q=2

q=0.5

q=1

q=2

q=0.5

q=1

q=2

Node2VecParameters

Node2VecParameters

of a large research institution, available as part of the Stanford Network Analysis Project [31]. There are 25571 directed edges between the nodes in this network, with known ground truth communities consisting of 42 departments present in this institution. We also consider a widely used dataset of directed edges between 1490 U.S political blogs, collected before the 2004 elections [2]. Here the directed edges correspond to hyperlinks, with ground truth communities corresponding to whether the blogs has been identified as liberal or conservative.

For each of these datasets we compare the community recovery of Node2Vec and traditional spectral clustering, using the normalized graph Laplacian. As is common in the literature, we remove the direction from these edges and take the largest connected component, forming symmetric adjacency matrices with 986 and 1222 nodes respectively. We then use the previously described procedure to perform community detection using Node2Vec. We consider a range of embedding dimensions ( d = 16 , 32 , 64 , 128 , 256 ) and unigram sampling parameter ( α = -1 , 0 . 0 , 0 . 25 , 0 . 5 , 0 . 75 , 1 . 0 ), while keeping all other parameters fixed at the defaults considered before. With the true number of communities known, we then compare the estimated communities from 10 simulations for each of these parameter settings, along with performing 10 simulations of spectral clustering for each of these settings.

In Figure S7 we compare the performance of Node2Vec and spectral clustering for the Email network and in Figure S8 we use the Political Blogs network. We measure community recovery in terms of the normalized mutual information (NMI) between the estimated and true communities. Other metrics such as the adjusted rand index (ARI) showing similar results. In each case the communities estimated by Node2Vec are substantially closer to the true communities than those estimated by spectral clustering. As highlighted by Karrer and Newman [23] for the political blog data, models which do not account for degree heterogeneity can struggle to recover the underlying community structure. As shown in Figure S8, spectral clustering is unable to recover the communities due to this heterogeneity, while clustering using the Node2Vec embedding shows strong performance at community recovery.

We also further expand on the role of the embedding parameters in the performance of Node2Vec on these real networks. In Figure S9 we examine community recovery for the Email data as we vary the embedding dimension d and the unigram sampling parameter α . As we vary each of these parameters we see good community recovery in all settings. For this dataset all choices of embedding dimension and unigram parameter give good NMI scores.

B Additional Notation

We give a brief recap of some of the notation introduced in the main paper, along with some more notation which is used purely within the Supplemntary Material. Throughout, we will suppose that

1.00-

0.75-

0.50

0.25

0.00-

Node2Vec

Spectral Clustering

1.00

0.75

0.50-

0.25

0.00

Node2Vec

Spectral Clustering

the graph G = ( V , E ) is drawn according to the following generative model: each vertex u ∈ V have latent variables λ u = ( c ( u ) , θ u ) where c ( u ) ∈ [ κ ] is a community assignment, and θ u is a degree-heterogenity correction factor. We then suppose that the edges a uv ∈ { 0 , 1 } in the graph G n on n vertices arise independently with probability

for u < v , with a uv = a vu by symmetry for u > v 2 . The factor ρ n accounts for sparsity in the network. The above model corresponds to a degree corrected stochastic block model [23]; we

1.00

0.75

NMI

0.50

0.25

0.00

d=16

d=32

d=64

d=128

d=256

Node2VecDimension

(a) Varying the embedding dimension used.

1.00

0.75

NMI

0.50

0.25

0.00

0

0.25

0.5

0.75

UnigramParameterα

(b) Varying the Unigram Parameter α .

highlight that the case where θ u is constant across all u ∈ V corresponds to the original stochastic block model [20]. For convenience, we will write

We then introduce the notation

Note that under the assumptions that the community assignments are drawn i.i.d from a Categorical( π ) random variable, and the degree correction factors are drawn i.i.d from a distribution ϑ independently of the community assignments, we have

For convenience, we will write ˜ P c ( i ) = ∑ κ j =1 π j P c ( i ) ,j .

Recall that node2vec attempts to minimize the objective

̸

where U, V ∈ R n × d , with u i , v j ∈ R d denoting the i -th and j -th rows of U and V respectively, and σ ( x ) := (1 + e -x ) -1 denoting the sigmoid function. Here P and N correspond to the positive and negative sampling schemes induced by the random walk and unigram mechanisms respectively.

C Proof of Theorems 2 and 3

C.1 Proof overview

To give an overview of the proof approach, we work by forming successive approximations to the function L n ( U, V ) where we have uniform convergence of the approximation error as n →∞ over either level sets of the function considered, or the overall domain of optimization of the embedding matrices U and V . We break these approximations up into multiple steps:

Theorems S1, S2, S3 and Proposition S4 - We begin by working with an approximation ̂ L n ( U, V ) of L n ( U, V ) , where the sampling weights P ( ( i, j ) ∈ P ( G n ) | G n ) and P ( ( i, j ) ∈ N ( G n ) | G n ) are replaced by functions of the latent variables ( λ i , λ j ) of the vertices i and j , along with a ij in the case of f P ( λ i , λ j ) .

The resulting approximation ̂ L n ( U, V ) has a dependence on the adjacency matrix of the network. We argue that this loss function converges uniformly to its average over the adjacency matrix when the vertex latent variables remain fixed; this is the contents of Theorem S5.

̸

So far, the loss function only looks between interactions of u i and v j for i = j . For theoretical purposes, it is more convenient to work with a loss function where the term with i = j is included. This is handled within Lemma S6.

Now that we have an averaged version of the loss function to work with, we are able to examine the minima of this loss function, and find that there is a unique minima (in the sense that for any pair of optima matrices U ∗ and V ∗ , the matrix U ∗ ( V ∗ ) T is unique). Moreover, in certain circumstances we can give closed forms for these minima. This is the contents of Section C.6.

This is then all combined together in order to give Theorems S13 and S14, which correspond to Theorems 1 and 2 of the main text.

We recap that we consider three scenarios - referred to as Scenario (i), (ii) and (iii) throughout - when proving the following result:

(i) We use DeepWalk ( p = q = 1 in node2vec), and the graph is drawn according to a SBM with ρ n ≫ log( n ) /n ;

(ii) We use node2vec, and the graph is drawn according to a SBM with ρ n = n -α for some α < α ′ , where α ′ depends on node2vec's hyperparameters;

(iii) We use DeepWalk and a unigram parameter of α = 1 , and the graph is drawn according to a DCSBM with ρ n ≫ log( n ) /n where the degree heterogeneity parameters θ u ∈ [ C -1 , C ] for some C > ∞ .

Generally speaking, the approach is the exact same for all three scenarios. As we have a closed formula in the case where we examine DeepWalk, we will consistently provide the details for the DeepWalk case first, and then discuss afterwards how the results and proofs change (if at all) when considering node2vec in generality. Throughout, we also contextualize the proof by examining what it says for a SBM ( n, κ, ˜ p, ˜ q, ρ n ) model. This corresponds to a balanced network with π = ( κ -1 , . . . , κ -1 ) .

C.2 Replacing the sampling weights

Before giving an approximation to L n ( U, V ) , we need to first come up with approximate forms of P ( ( i, j ) ∈ P ( G n ) | G n ) and P ( ( i, j ) ∈ N ( G n ) | G n ) . The next three results give examples of this. In this section we prove three main results. The first two give us guarantees for the sampling probabilities of vertex pairs ( u, v ) for node2vec for any choice of the hyperparameters ( p, q ) . In particular they will allow us to argue that when the underlying graph arises from a SBM, the sampling probabilities asymptotically depend only on the underlying communities. The last specializes this to the case of DeepWalk (where p = q = 1 ), which has enough structure to allow us to get some additional information, such as closed formula for these sampling probabilities, which can be used in the case where the graph arises through a DCSBM.

Theorem S1. There exists α sufficiently small, depending on the walk length k , such that if ρ n = n -α then there exists a symmetric measurable (with respect to the sigma field generated by W ) function f P ( λ, λ ′ ) which is bounded below away from zero, and bounded above by Cρ -1 n for some constant C < ∞ , such that

̸

Theorem S2. There exists α sufficiently small, depending on the walk length k , such that if ρ n = n -α then there exists a symmetric measurable (with respect to the sigma field generated by W ) function f P ( λ, λ ′ ) which is bounded below away from zero, and bounded above by some constant C < ∞ , such that

̸

The proof of these two results are given in Appendix E.1.1 and E.1.2 respectively. We note that while in principle we could give a closed formula for f P and f N in this scenario, they are sufficiently intractable to inspection that doing so would not provide any benefit.

In the case of DeepWalk where p = q = 1 , the calculations involved are tractable enough such that we can improve the sparsity constraints, give closed forms for the measurable functions discussed above, and also provide rates of convergence.

Theorem S3. Denote

Then we have that

̸

̸

Proof. This is a consequence of [11, Proposition 26]. We highlight the referenced result supposes that for the negative sampling scheme, vertices for which a ij = 0 are rejected, whereas this does not happen here. Other than for the factor of (1 -a ij ) in the quoted result, the proof is otherwise unchanged, which gives the statement above for P ( ( i, j ) ∈ N ( G n ) | G n ) .

With this, we then get the following result:

Proposition S4. Denote

̸

and define the set

for any constant ˜ A > 1 , where 0 n × d denotes the zero matrix in R n × d . Then for any set X ⊆ R n × d × R n × d containing the pair of zero matrices O n × d , we have under Scenario i) and iii) that

In Scenario (ii), the O p ( · ) bound is replaced by an o p (1) bound.

Proof. The proof is essentially equivalent to Lemma 32 of [11] up to changes in notation, and so we do not repeat the details.

Note that in practice we can choose A to be any constant greater than 1 but fixed with n - e.g A = 10 , and have the result hold. We will do so going forward.

C.3 Averaging over the adjacency matrix of the graph

Following the proof outline, the next step is to argue that L n ( U, V ) is close to its expectation when we average over the adjacency matrix of the graph G n . We begin with showing what occurs in the

DeepWalk case (Scenarios (i) and (iii)), and at the end of the section we discuss how the proof changes for the more general node2vec case. Note that we have

̸

̸

Note that E [ E n ( U, V ) | λ ] = 0 , and so it therefore suffices to control E n ( U, V ) -E [ E n ( U, V ) | λ ] uniformly over embedding matrices U, V ∈ R n × d . This is the contents of the next theorem.

Theorem S5. Begin by defining the set

Then we have the bound

In particular, we also have that

Proof. Begin by noting that for any set C ⊆ R n × d × R n × d for which 0 n × d × 0 n × d ∈ C , we have that

We therefore need to control these two terms. We begin with the second; note that as

̸

it follows by Lemma S30 that this term is O p (( n 2 ρ n ) -1 / 2 ) . For the first term, we make use of a chaining bound. Note that if we write T ij = -log σ ( ⟨ u i , v j ⟩ ) and S ij = -log σ ( ⟨ ˜ u i , ˜ v j ⟩ ) for i, j ∈ [ n ] , then we have that

̸

Because the function x ↦→-log σ ( x ) is 1-Lipschitz, it follows that

and consequently we have that

and so

as a result of Lemma S30. Now, as U, V ∈ B F ( A F ) ∩ B 2 , ∞ ( ˜ A 2 , ∞ ) , by Lemma S19 if we define the metrics

then we have that

As a result of Corollary S22, it therefore follows that

The desired conclusion follows by combining the bounds (S24) and (S33).

For the more abstract node2vec case under Scenario (ii), we highlight that we can take

̸

Now, as f P ( λ u , λ v ) is a function of the community assignments only within the SBM case, we can replace this by a matrix of constants f P ,c,c ′ for c, c ′ ∈ [ κ ] , and therefore the error term can be decomposed into a sum

̸

where we recall that max c 1 ,c 2 ( ρ n f P ,c 1 ,c 2 ) < ∞ as guaranteed by Theorem S1. Each of these terms (of which there are finitely many) can be controlled using the exact same argument as in Theorem S5, and so the conclusion of the Theorem also holds with the same overall rate of convergence in Scenario (ii).

C.4 Adding in a diagonal term

Currently the sum in E [ ̂ L n ( U, V ) | λ ] is defined only terms i, j with i = j - it is more convenient to work with the version where the diagonal term is added in:

̸

We show that this does not significantly change the size of the loss function.

Lemma S6. With the same notation as in Theorem S5, we have that

In particular, in the case of DeepWalk we have that

Proof. Begin by noting that

Note that we can bound

and similarly -log(1 -σ ( ⟨ u i , v i ⟩ )) ≤ |⟨ u i , v i ⟩| ≤ ∥ u i ∥ 2 ∥ v i ∥ 2 . Moreover, we have the bounds

under our assumptions. As a result, because U, V ∈ B 2 , ∞ ( ˜ A 2 , ∞ ) , we end up with the final bound

C.5 Chaining up the loss function approximations

By chaining up the prior results, we end up with the following result:

Proposition S7. There exists a non-empty set Ψ n for each n such that, for any set X ⊆ R n × d × R n × d containing 0 n × d × 0 n × d , we have for DeepWalk that

and

For node2vec, the same result holds when we replace the (log n/nρ n ) 1 / 2 term with an o p (1) term and add the constraint that d ≪ nρ n . The same result also holds when we constrain U = V , but otherwise keep everything else unchanged.

C.6 Minimizers of R n ( U, V )

Recall that we have earlier defined

We now want to reason about the minima of these functions. To do so, note that the optimization domain is non-convex - firstly due to the rank constraints on the matrix UV T , and secondly due to the fact that the loss function is invariant to any mapping ( U, V ) → ( UM,VM -1 ) for any invertible d × d matrix M . To handle the second part, we consider the global minima of this function when parameterized only in term of the matrix UV T . We will then see that the minima matrix is already low rank.

We first begin by giving some basic facts about the function R n ( U, V ) when parameterized as a function of UV T .

Lemma S8. Define the modified function

over all matrices M ∈ R n × n . Then we have the following:

The function R n ( M ) is strictly convex in M .

The global minimizer of R n ( M ) is given by

and satisfies ∇ M R n ( M ) = 0 .

When restricted to a cone of semi-positive definite matrices M ∈ M ⋟ 0 n , there exists a unique minimizer to R n ( M ) over this set, which we call M ⋟ 0 . Moreover, M ⋟ 0 has the property that ⟨∇ M R n ( M ⋟ 0 ) , M ⋟ 0 -M ⟩ ≤ 0 for all M ∈ M ⋟ 0 n .

Proof. For part a), this follows by the fact that the functions -log( σ ( x )) and -log(1 -σ ( x )) are positive and strictly convex functions of x ∈ R , the fact that f P ( λ i , λ j ) ρ n W ( λ i , λ j ) and f N ( λ i , λ j ) are positive quantities which are bounded above (see e.g Lemma S6), and the fact that the sum of strictly convex functions is strictly convex. For part b), this follows by noting that each of the M ∗ ij are pointwise minima of the functions

defined over x ∈ R . Indeed, note that

so setting this equal to zero, rearranging and making use of the equality σ -1 ( a/ ( a + b )) = log( a/b ) gives the stated result. Part c) is a consequence of strong convexity, the optimization domain being convex and self dual, and the KKT conditions.

To understand the form of the the global minimizer of R n ( M ) in the DeepWalk case, by substituting in the values for f P ( λ i , λ j ) and f N ( λ i , λ j ) we end up with

In particular, from the above formula we get the following lemma as a consequence:

Lemma S9. Suppose that Scenarios (i) or (iii) holds, so that either a) θ i is constant for all i , or b) α = 1 . Then if we write Π C ∈ R n × κ for the matrix where (Π C ) il = 1[ c ( i ) = l ] , and define the matrix

then we have that M ∗ = Π C ˜ M ∗ α Π T C . In particular, as soon as the matrix Π C is of full rank (which occurs with asymptotic probability 1), then the rank of M ∗ equals the rank of ˜ M ∗ α . Moreover, as soon as d is greater than or equal to the rank of ˜ M ∗ α , ( U, V ) is a minimizer of R n ( U, V ) if and only if UV T = M ∗ .

Under Scenario (ii), the same result applies noting that f P and f N are functions only of the underling communities, and so if we abuse notation and write e.g f P ( l, m ) to indicate the value of f P ( λ i , λ j ) when c ( i ) = l and c ( j ) = m , one can take

and have the above result hold.

In this case, we have that

Substituting these values into the matrix ˜ M ∗ α gives

We highlight this is a matrix of the form αδ lm + β (1 -δ lm ) , and so it is straightforward to describe the spectral behavior of the matrix (see Lemma S31).

C.6.1 Minimizers in the constrained regime U = V

In the case where we have constrained U = V , it is not possible in general to write down the closed form of the minimizer of R n ( M ) over M ⋟ 0 n . However, it is still possible to draw enough conclusions about the form of the minimizer in order to give guarantees for community detection. We begin with the proposition below. We state the next two results for DeepWalk only, but note that the first generalizes to the node2vec case immediately.

Proposition S10. Suppose that θ i is constant across all i . Supposing that ˜ M ∈ R κ × κ is of the form ˜ M = ˜ U ˜ U T for matrices ˜ U ∈ R κ × d , define the function

where we define ˆ p n ( l ) := n -1 |{ i : c ( i ) = l }| for l ∈ [ κ ] . Then ˜ R n ( ˜ M ) is strongly convex, and moreover has a unique minimizer as soon as d ≥ κ .

Moreover, any minimizer of R n ( M ) over matrices M of the form M = UU T where U ∈ R n × d must take the form M = Π C M ∗ Π T C where (Π C ) il = 1[ c ( i ) = l ] where M ∗ is a minimizer of ˜ R n ( ˜ M ) . In particular, once d ≥ κ , there is a unique minimizer to R n ( M ) .

Proof. The properties of ˜ R n ( ˜ M ) are immediate by similar arguments to Lemma S8 and standard facts in convex analysis. We begin by noting that if we substitute in the values

for f P ( λ i , λ j ) and f N ( λ i , λ j ) , then we can write that (recalling that M ij = ⟨ u i , u j ⟩ )

̸

Wediscuss in Appendix F what happens when we apply DeepWalk in the DCSBM regime when α = 1 . To give an example of what M ∗ looks like, we write it down in the case of a SBM( n, κ, ˜ p, ˜ q, ρ n ) model, which is frequently used to illustrate the behavior of various community detection algorithms. Such a model assumes that the community assignments π l = 1 /κ for all l ∈ [ κ ] , and that

̸

where for l ∈ [ κ ] we define ˆ p n ( l ) := n -1 |{ i : c ( i ) = l }| , along with the sets C l = { i : c ( i ) = l } . Now, note that as the functions -log( σ ( x )) and -log(1 -σ ( x )) are strictly convex, by Jensen's inequality we have that e.g

(where we also used bilinearity of the inner product) where equality holds above if and only if the u i are constant are across all indices i . In particular, any minimizer of R n ( M ) must have the u i constant across i ∈ C l for each l ∈ [ κ ] , which defines the function ˜ R n ( ˜ M ) . This gives the claimed statement.

In certain cases, we are able to give a closed form to the minimizer. We illustrate this for the case of the SBM ( n, κ, ˜ p, ˜ q, ρ n ) model.

Proposition S11. Let ˜ M ∗ be the unique minimizer of ˜ R n ( ˜ M ) as introduced in Proposition S10. In the case of a SBM ( n, κ, ˜ p, ˜ q, ρ n ) model, we have that κ -2 ∥ ˜ M ∗ -M ∗ ∥ 1 = O p (( κ log κ/n ) 1 / 4 ) , where M ∗ is of the form

for some α ∗ = α ∗ (˜ p, ˜ q ) ≥ 0 . Moreover, α ∗ > 0 iff ˜ p > ˜ q .

Proof. We begin by arguing that the objective function ˜ R n ( ˜ M ) converges uniformly to the objective

over a set containing the minimizers of both functions. Note that this function is also strictly convex, and has a unique minimizer as soon as d ≥ κ . To do so, we highlight that as we have that

̸

by standard concentration results for Binomial random variables (e.g Proposition 47 of [11]), it follows that

Consequently, ˜ R n ( ˜ M ) converges to ¯ R n ( ˜ M ) uniformly over any level set of ¯ R n ( ˜ M ) , which necessarily contains the minima of ¯ R n ( ˜ M ) . If one does so over the set (for example)

(for example), then as ¯ R n (0) is constant across n , we have uniform convergence of (S67) over the set A at a rate of O p ( (log κ/np ) 1 / 2 ) . This argument can be reversed, which therefore ensures uniform convergence (over the same set) which contains the minimizers (with the minimizer of ˜ R n ( M ) being contained within this set with asymptotic probability 1 ) at a rate of O p ( ( κ log κ/n ) 1 / 2 ) .

With this, we note that an application of Lemma S33 gives that for any matrices ˜ M 1 and ˜ M 2 we have that

where to save on notation, we define

In particular, if ˜ M 2 = ¯ M ∗ is an optimum of ¯ R n ( ˜ M ) , then by the KKT conditions (similarly as in Lemma S8) we have that

In particular, if we then let ˜ M ∗ be any minimizer of ˜ R n ( ˜ M ) , then we have that

on an event of asymptotic probability 1 . Consequently, it follows by Lemma S34 that

We now need to find the minimizing positive semi-definite matrix which optimizes ¯ R n ( ˜ M ) . To do so, we will argue that one can find α for which

̸

for some positive constant C , as then the KKT conditions for the constrained optimization problem will hold. Indeed, for any positive definite matrix M , as by definition of ̂ M we have that ⟨∇ ¯ R n ( ̂ M ) , ̂ M ⟩ = 0 as all of the eigenvectors of ̂ M are orthogonal to the unit vector 1 κ (Lemma S31). It consequently follows that as ∇ ¯ R n ( ̂ M ) is itself positive definite, we get that ⟨-∇ ¯ R n ( ̂ M ) , ̂ M -M ⟩ = ⟨∇ ¯ R n ( ̂ M ) , M ⟩ ≥ 0 . We now need to verify the existence of a constant α for which this condition holds. We note that as ̂ M ij is constant across i = j , and also constant across i = j , to verify the condition that ∇ ¯ R n ( ̂ M ) is proportional to 1 κ 1 T κ , it suffices to check whether the on and off diagonal terms of ∇ ¯ R n ( ̂ M ) are equal to each other. This gives the equation

By applying Lemma S32, this has a singular positive solution in α if and only if k (˜ p -˜ q ) ≥ k (˜ p -˜ q ) / 2 , which holds iff ˜ p ≥ ˜ q . In the case where ˜ p < ˜ q , it follows that the solution has α = 0 .

C.7 Strong convexity properties of the minima matrix

Proposition S12. Define the modified function

over all matrices M ∈ R n × n . Then we have for any matrices M 1 , M 2 ∈ R n × n with ∥ M 1 ∥ ∞ , ∥ M 2 ∥ ∞ ≤ ˜ A ∞ that

where ˜ C = min l,m { 2 kP l,m , ˜ P α l ˜ P m } for Scenarios (i) and (iii), and ˜ C = min {∥ ρ n f P ( λ, λ ′ ) ∥ -∞ , ∥ f N ( λ, λ ′ ) ∥ -∞ } > 0 for Scenario (ii). Moreover,

If R n ( M ) is constrained over a set X = { M = UV T : U, V ∈ R n × d , ∥ M ∥ ∞ ≤ ˜ A ∞ } , and there exists M ∗ in X such that ∇R n ( M ∗ ) = 0 , then we have that

ii) If R n ( M ) is constrained over a set X ≥ 0 = { M = UU T : U ∈ R n × d , ∥ M ∥ ∞ ≤ ˜ A ∞ }, and there exists M ∗ in X ≥ 0 such that ⟨∇R n ( M ∗ ) , M -M ∗ ⟩ ≥ 0 for all M ∈ X ≥ 0 , then we get the same inequality as in part i) above.

Proof. The first inequality follows by an application of Lemma S33, with the second and third parts following by applying the conditions stated and rearranging.

C.8 Convergence of the gram matrices of the embeddings

By combining together Proposition S12 and Proposition S7 we end up with the following result:

Theorem S13. Suppose that the conditions of Lemma S9 hold. (In particular, recall that d ≥ κ .) Then there exist constants ˜ A ∞ and ˜ A 2 , ∞ (depending on the parameters of the model and sampling scheme) and a matrix M ∗ ∈ R κ × κ (also depending on the parameters of the model and the sampling scheme) such that for any minimizer ( U ∗ , V ∗ ) of L ( U, V ) over the set

we have that

(S81)

for some constant C depending on the model, the node2vec hyperparameters, ˜ A ∞ and ˜ A 2 , ∞ . In the case where we constrain U = V , the same result holds provided the conditions of Proposition S10 hold.

Proof. We note that by Lemma S9, there exists a minimizer ˜ M ∗ for R n ( M ) of the form ˜ M ∗ = Π M ∗ Π T for a matrix M ∗ ∈ R κ × κ . We can then take ˜ A ∞ and ˜ A 2 , ∞ as 2 ∥ M ∗ ∥ ∞ and 2 ∥ M ∗ ∥ 2 , ∞ . We highlight that we can do this even when d > κ , as we can embed M ∗ into the block diagonal matrix diag( M ∗ , O d -κ,d -κ ) , which preserves both the norms above. Lemma S8 and Proposition S12 then guarantee that

for some constant ˜ C depending only on the quantities mentioned in the theorem statement. As X is a subset of B 2 , ∞ ( ˜ A 2 , ∞ ) , and ( U ∗ , V ∗ ) is a minimizer of L ( U, V ) , we end up getting that

from which we can apply Proposition S7 to then give the claimed result.

We give some brief intuition as to the size of the constants involved here, to understand any potential hidden dependencies involved in them. Of greatest concern are the constants ˜ A ∞ and ˜ A 2 , ∞ (as the remaining constants are explicit throughout the proof, and depend only on the hyperparameters of the sampling schema and the model in a polynomial fashion). Note that in the case where k is large and we have a SBM ( n, κ, ˜ p, ˜ q, ρ n ) model and we apply the DeepWalk scheme, from the discussion after Lemma S9, the minimizing matrix M ∗ takes the form

Supposing for simplicity that ˜ p > ˜ q , it follows that we can take can take ˜ A ∞ to be of the order O (log(˜ p/ ˜ q )) when κ is large. In the rate from Proposition S12, this gives a rate of O (˜ p/ ˜ q ) from the e ˜ A ∞ factor; note that the dependence on the parameters of the models here are not unreasonable. As for ˜ A 2 , ∞ , we first highlight the fact that

By Lemma S31 we can therefore take ˜ A 2 , ∞ to be a scalar multiple of | log(˜ p/ ˜ q ) | 1 / 2 , avoiding any implicit dependence on κ or the embedding dimension d .

C.9 Convergence of the embedding vectors

We can then get results guaranteeing the convergence of the individual embedding vectors (rather than their gram matrix) up to rotations, as stated by the following theorem.

Theorem S14. Suppose that the conclusion of Theorem S13 holds, and further suppose that d equals the rank of the matrix M ∗ . Then there exists a matrix ˜ U ∗ ∈ R κ × d such that

̸

Proof. We handle the cases where U = V and U = V separately. For the case where U = V , we note that without loss of generality we can suppose that UU T = V V T , in which case we can apply Lemma S23 and Theorem S13 to give the stated result. To do so, we note that by Lemma S25 we have that n -1 σ d (Π M ∗ Π T ) ≥ cσ d ( M ∗ ) for some constant c with asymptotic probability 1 , as a result of the fact that n k (Π) ≥ 1 / 2 nπ k with asymptotic probability 1 uniformly across all communities k ∈ [ κ ] . As moreover we have that n -1 ∥ UV T -Π M ∗ Π T ∥ op ≤ n -1 ∥ UV T -Π M ∗ Π T ∥ F = o p (1) , the condition that ∥ UV T -Π M ∗ Π T ∥ op ≤ 1 / 2 σ d (Π M ∗ Π T ) holds with asymptotic probability 1, we have verified the conditions in Lemma S23, giving the desired result. In the case where we constrain U = V , the same argument holds, except we no longer need to verify the condition that ∥ UU ∗ -M ∗ ∥ op is sufficiently small, and so we have concluded in this case also.

̸

In the case of a SBM ( n, κ, ˜ p, ˜ q, ρ n ) model it is actually able to give closed form expressions for the embedding vectors which are converged to by factorizing the minima matrix M ∗ in the way described by the above proof. These details are given in Lemma S31.

D Proof of Theorem 4 and Corollary 5

D.1 Guarantees for community detection

We begin with a discussion of how we can get guarantees for community detection via approximate k-means clustering method, using the convergence criteria for embeddings we have derived already. To do so, suppose we have a matrix U ∈ R n × d corresponding of n columns of d -dimensional vectors. Defining the set

we seek to find a factorization U ≈ Π X for matrices Π ∈ M n,K and X ∈ R K × d . To do so, we minimize the objective

In practice, this minimization problem is NP-hard [5], but we can find (1 + ϵ ) -approximate solutions in polynomial time [25]. As a result, we consider any minimizers ˆ Π and ˆ X such that

We want to examine the behavior of k-means clustering on the matrix U , when it is close to a matrix U ∗ which has an exact factorization U ∗ = Π ∗ X ∗ for some matrices Π ∗ ∈ M n,K and X ∗ ∈ R K × d . We introduce the notation

for the columns of U which are assigned as closest to the k -th column of X as according to the matrix Π .

We make use of the following theorem from Lei and Rinaldo [30], which we restate for ease of use.

̸

Proposition S15 (Lemma 5.3 of Lei and Rinaldo [30]) . Let ( ˆ Π , ˆ X ) be any (1 + ϵ ) -approximate minimizer to the k-means problem given a matrix U ∈ R n × d . Suppose that U ∗ = Π ∗ X ∗ for some matrices Π ∗ ∈ M n,κ and X ∗ ∈ R κ × d . Fix any δ k ≤ min l = k ∥ X ∗ l · -X ∗ k · ∥ 2 , and suppose that the condition

holds. Then there exist subsets S k ⊆ G k (Π ∗ ) and a permutation matrix σ ∈ R κ × κ such that the following holds:

For G = ⋃ k ( G k (Π ∗ ) \ S k ) , we have that (Π ∗ ) G · = σ Π G · . In words, outside of the sets S k we recover the assignments given by Π ∗ up to a re-labelling of the clusters.

In particular, we can then apply this to our consistency results with the embeddings learned by node2vec. Recall that we are interested in the following metrics measuring recovery of communities by any given procedure:

̸

̸

These measure the overall misclassification rate and worst-case class misclassification rate respectively.

Corollary S16. Suppose that we have embedding vectors ω i ∈ R d for i ∈ [ n ] such that

for some rate function r n → 0 as n →∞ and vectors η l ∈ R d for l ∈ [ κ ] . Moreover suppose that δ := min l = k ∥ η l -η k ∥ 2 > 0 . Then if ˆ c ( i ) are the community assignments produced by applying a (1 + ϵ ) -approximate k-means clustering to the matrix whose columns are the ω i , we have that L ( c, ˆ c ) = O p ( δ -2 r n ) and ˜ L ( c, ˆ c ) = O p ( δ -2 r n ) . If the RHS of (S96) is instead o p (1) , then we replace O p ( r n ) by o p (1) in the statements for L ( c, ˆ c ) and ˜ L ( c, ˆ c ) .

̸

Proof. We apply Proposition S15 with Π ∗ corresponding to the matrix of community assignments according to c ( · ) , and X ∗ the matrix whose columns are the Qη l for l ∈ [ κ ] where Q ∈ O ( d ) attains the minimizer in (S96). Letting U be the matrix whose columns are the ω i and taking δ k = δ , the condition (S93) to verify becomes

As r n → 0 and |C l | /n > c > 0 for some constant c uniformly across vertices l ∈ [ κ ] with asymptotic probability 1 (as a result of the community generation mechanism, the communities are balanced), the above event will be satisfied with asymptotic probability 1. The desired conclusion follows by making use of the inequalities

which hold by the first consequence in Proposition S15, and then applying the bound

We note that in order to apply this theorem, we require the further separation criterion of δ > 0 . As a result of Lemma S31, we can guarantee this for the SBM ( n, κ, ˜ p, ˜ q, ρ n ) model when either a) DeepWalk is trained in the unconstrained setting, or b) we are in the constrained setting with ˜ p > ˜ q . As we know that the embedding vectors converge to the zero vector on average when we are in the constrained setting with ˜ p ≤ ˜ q , as a result we know that community detection is possible in the constrained setting iff ˜ p > ˜ q , which gives Corollary 5 of the main paper.

D.2 Guarantees for node classification and link prediction

We now discuss what guarantees we can make when using the embedding vectors for classification. In this section, we suppose that we have a guarantee

for some constant C ( τ ) and rate function r n → 0 as n → ∞ . This is the same as saying that the LHS is O p ( r n ) - it will happen to be more convenient to use this formulation. We also suppose that there exists a positive constant δ > 0 for which

̸

We begin with a lemma which discusses the underlying geometry when we take a small sample of the embedding vectors.

Lemma S17. Suppose we sample K embeddings from the set ( u i ) i ∈ [ n ] , which we denote as u i 1 , . . . , u i K . Define the sets

Then there exists n 0 ( K,δ, τ ′ ) such that if n ≥ n 0 , with probability 1 -τ ′ we have that u i j ∈ S c ( i j ) for all j ∈ [ K ] .

Proof. Without loss of generality, we will suppose that Q = I . For each l ∈ [ κ ] , define the sets S l = { i ∈ C l : ∥ u i -η l ∥ 2 ≤ δ/ 4 } . Then by the condition (S100), by Markov's inequality we know that with probability 1 -τ we have that

We now suppose that we sample K embeddings uniformly at random; for convenience, we suppose that they are done so with replacement. Then the probability that all of the embeddings are outside the set ⋃ l ( C l \ S l ) is given by (1 -1 n ∑ l |C l \ S l | ) K ≥ 1 -K n ∑ l |C l \ S l | . In particular, this means with probability no less than 1 -τ -4 Kδ -1 C ( τ ) r n , if we sample K embeddings with indices i 1 , . . . , i K at random from the set of n embeddings, they lie within the sets S C ( i 1 ) , . . . , S C ( i K ) respectively. The desired result then follows by noting that we take τ = τ ′ / 2 , and choose n such that 4 δ -2 C ( τ/ 2) r n < τ ′ / 2 .

To understand how this lemma can give insights into the downstream use of embeddings, suppose that we have access to an oracle which provides the community assignments of a vertex when requested, but otherwise the community assignments are unseen.

We note that in practice, only a small number of labels are needed to be provided to embedding vectors in order to achieve good classification results (see e.g the experiments in Hamilton et al. [17], Veliˇ ckovi´ c et al. [47]). As a result, we can imagine keeping K fixed in the regime where n is large. Moreover, the constant δ simply reflects the underlying geometry of the learned embeddings, and τ ′ is a tolerance we can choose such that the stated result is very likely to hold (by e.g choosing τ ′ = 10 -2 or 10 -3 ). As a consequence, the above lemma tells us with high probability, we can

learn a classifier which is able to distinguish between the sets S l given use of the sampled embeddings u i 1 , . . . , u i K and the labels c ( i 1 ) , . . . , c ( i K ) , provided the classifier is flexible enough to separate κ disjoint convex sets; and

ii) as a consequence of (S103), this classifier will correctly classify a large proportion of vertices within the correct sets S l .

The same argument applies if instead we have classes assigned to embedding vectors which form a coarser partitioning of the underlying community assignments. The importance of the above result is that in order to understand the behavior of embedding methods for classification, it suffices to understand which geometries particular classifiers are able to separate - for example, when the number of classes equals 2 , this reduces down to the classic concept of linear separability, in which case a logistic classifier would suffice.

We end with a discussion as to the task of link prediction, which asks to predict whether two vertices are connected or not given a partial observation of the network. To do so, we suppose that from the observed network, we delete half of the edges in the network, and then train node2vec on the resulting network. Note that the node2vec mechanism only makes explicit use of known edges within the network. This corresponds to training the node2vec model on the data with sparsity factor ρ n → ρ n / 2 ; in particular, this leaves the underlying asymptotic representations unchanged and slows the rate of convergence by a factor of 2. With this, a link prediction classifier is formed by the following process:

Take a set of edges J ⊆ { ( i, j ) : a ij = 1 } for which the node2vec algorithm was not trained on, and a set of non-edges ˜ J ⊆ { ( i, j ) : a ij = 0 } . As in practice networks are sparse, these sets are not sampled randomly from the network, but are assumed to be sampled in a balanced fashion so that the sets J and ˜ J are roughly balanced in size. One way of doing so is to pick a number of edges in advance, say E , and then sample E elements from the set of edges and non-edges in order to form J and ˜ J respectively.

Form edge embeddings e ij = f ( u i , u j ) given some symmetric function f ( x, y ) and node embeddings u i . Two popular choices of functions are the average function f ( x, y ) = ( x + y ) / 2 and the Hadamard product f ( x, y ) = ( x i y i ) i ∈ [ d ] .

Using the features e ij and the labels provided by the sets J and ˜ J , build a classifier using your favorite ML algorithm.

By our convergence guarantees, we know that the asymptotic distribution of the edge embeddings e ij will approach some vectors η c ( i ) ,c ( j ) ∈ R d , giving at most κ 2 distinct vectors overall. Note that these embedding vectors in of themselves contain little information about whether the edges are connected; that said, even given perfect information of the communities and the connectivity matrix P , one can only form probabilistic guesses as to whether two vertices are connected. That said, by clustering together the link embeddings we can identify together edges as having vertices belonging to a particular pair of communities. With knowledge of the sampling mechanism, it is then possible to backout estimates for p and q by counting the overlap of the sets J and ˜ J in the neighbourhoods of the clustered node embeddings.

Wenote that in practice, ML classification algorithms such as logistic regression are used instead. This instead depends on the typical geometry of the sets J and ˜ J . Suppose we have a SBM ( n, 2 , ˜ p, ˜ q, ρ n ) model. In this case, the set J will approximately consist of ˜ p/ 2(˜ p + ˜ q ) × E vectors from η 11 , ˜ p/ 2(˜ p + ˜ q ) × E vectors from η 22 , ˜ q/ 2(˜ p + ˜ q ) × E vectors from η 12 and ˜ q/ 2(˜ p + ˜ q ) × E vectors from η 21 . In contrast, the set ˜ J will approximately have E/ 4 of each of η 11 , η 12 , η 21 and η 22 . As a result, in the case where ˜ p ≫ ˜ q , a linear classifier (for example) will be biased towards classifying more frequently vectors with c ( i ) = c ( j ) , which is at least directionally correct.

So far, we have not talked about the particular mechanism used to form link embeddings from the node embeddings. The Hadamard product is popular, but particularly difficult to analyze given our results, as it does not remain invariant to an orthogonal rotation of the embedding vectors. In contrast, the average link function retains this information. In the SBM ( n, 2 , ˜ p, ˜ q, ρ n ) , it ends up giving embeddings which will asymptotically depend on only whether c ( i ) = c ( j ) or not (i.e, whether the vertices belong to the same community or not).

E Intermediate results

E.1 Sampling probabilities for node2vec

In this section, we derive asymptotic results for the sampling probabilities of edges within node2vec. We begin by recapping the second-order random walk defined for node2vec. To do so, we define a random process ( X n ) n ≥ 1 via the second-order Markov property

where d u,s denotes the length of the shortest path between u and s . Given the extra information that ( u, s ) is an edge, d u,v = 0 occurs iff u = v , d u,v = 1 occurs iff ( u, v ) is an edge, and d u,v = 2 occurs iff ( u, v ) is not an edge (as given that ( v, s ) is an edge, the shortest path must be v → s → u ). With this, we select positive samples by selecting k concurrent edges within the walk (via taking a walk of length k +1 ).

To initialize the random walk, we note that for the second order walk we need to specify a distribution on the first two vertices; for DeepWalk where this collapses down to a first order walk, we only need to specify a distribution on ther first vertex. To do so generally, we consider an initial distribution of selecting the first vertex via π ( u ) = deg( u ) / ∑ v deg( v ) = deg( u ) / 2 E n with E n being the number of edges in the graph (single counting ( u, v ) ∈ E and ( v, u ) ∈ E ), and select the second vertex uniformly at random from those connected to the first. (Note that this is the transition kernel used for DeepWalk, and so we handle both cases via this argument.) One can show this is equivalent to selecting an edge uniformly at random.

For the negative sampling mechanism, we consider the vertices which arose as part of the positive sampling process - which we denote V ( P ) - and then sample l vertices independently according to the unigram distribution

̸

where u ∈ V ( P ) . We note that the case where α → 0 corresponds to the uniform distribution on vertices not equal to u .

E.1.1 Proof of Theorem S1

In this section and the next, it will be convenient to use the notation ∼ p to indicate that two positive random variables X n and Y n are asymptotic in the sense that | X n /Y n -1 | = o p (1) when n →∞ . If we say such a bound happens uniformly over some free variables - say X n,k ∼ p Y n,k uniformly over k - then this means max k | X n,k /Y n,k -1 | = o p (1) . We also make extensive use of the result that if X ( i ) n ∼ p r n Y ( i ) n for i ∈ { 0 , 1 } and Y ( i ) n ∈ [ C -1 , C ] for C > 1 , then X (0) n + X (1) n ∼ p r n ( Y (0) n + Y (1) n ) . Indeed, if we write X ( i ) n = Y ( i ) n r n (1 + ϵ ( i ) n where ϵ (1) n = o p (1) , then

from which the claimed result follows as the terms weighting the ϵ (1) n can be bounded below away from zero, and are bounded above by 1 . We also note that X (0) n -X (1) n = O p ( r n ) , meaning that the order of magnitude of terms cannot increase (only decrease) by subtracting them.

As we are interested in the sampling probability of edges within node2vec, it will be convenient to instead study the first order Markov process Y n = ( X n , X n -1 ) , as then we instead study the sampling probability of individual states in a regular Markov chain. We note that normally we use the notation ( u, v ) to refer an unordered pair belonging to an edge in a graph, but for the Markov process ( Y n ) n ≥ 1 the order matters, we will write Y n = e v → u whenever X n = u and X n -1 = v . In such a scenario, the random walk is therefore defined on the state space

with the law of Y given by

̸

̸

One can calculate the normalizing factor for the probability distribution as being

from which we observe that when p = q = 1 we recover the simple random walk defined by DeepWalk, as then the probability an edge is selected with source node u is uniform over edges ( u, v ) where v is a neighbour of u .

With this in mind, we define the transition matrix

̸

governing the transition probabilities on the above chain. We note that by [11, Proposition 72] and Theorem S26 respectively that

(S112)

uniformly over all s, u, v . As a result, we define

where δ uv := 1[ u = v ] and the numerator is the same as in P v → s,s → u (only written in a more convenient to use fashion), and the denominator makes use of the asymptotic statements (S111) and (S112). As a result, we have that P v → s,s → u ∼ p ˜ P v → s,s → u uniformly over v, s, u . In particular, we have that ˜ P v → s,s → u = Θ p ( a su ( nρ n ) -1 ) uniformly over all triples of indices ( v, s, u ) .

Let A j ( u → v ) = { Y j = e u → v } . We then note that the sampling probability of ( u, v ) being sampled within the first k +1 steps of the second order random walk is given by

To ease on the notation going forward, we write P n ( · ) := P ( · | G n ) . By the inclusion-exclusion principle, we can write this probability as equalling

We note that the number of terms in this sum is bounded above by (2 k )! (some terms will be zero, as we cannot select e u → v two times in a row), and so for asymptotic purposes we can focus on the individual terms.

̸

We now address the individual probabilities making up this sum. Intuitively, we want to show the following: that the terms for which ( l, m ) = (1 , 0) or (0 , 1) are asymptotically negligible, and that asymptotically these terms are functions only of ( λ u , λ v ) . We fix a particular instance of the i 1 , . . . , i l and j 1 , . . . , j m , and denote β 1 < β 2 < · · · < β l + m for the ordering of these indices. As we use indices i k to denote the direction u → v and j k for the direction v → u , we write

where the third argument (which we refer to as the orientation herein) indicates which of the first two arguments are used as the source node for the edge. For each β k for k ≤ l + m , we write o k to denote this orientation. As a result, it suffices for us to analyze

over all sequences 1 ≤ β 1 < β 2 < · · · < β l + m ≤ k and orientations ( o k ) l + m k =1 . For this, we then note that by the Markov property of the random walk, we are able to write this probability as

Focusing now on the terms in the product, if β k +1 -β k = 1 , then this term equals zero if o k = o k =1 , or otherwise equals e.g P u → v,v → u which is O p (( nρ n ) -1 ) as discussed above. If the walk is longer, then by the same argument as in [11, Proposition 73], by conditioning on the second step in the walk one can show this probability is asymptotically of the same order of a walk of length β k +1 -β k -1 initialized from the uniform distribution on the edges of G n . As a result, we therefore only need to analyze events of the form

which will allow us to then show that the events of the form ( l, m ) = (1 , 0) or (0 , 1) are the only ones we need to consider in the asymptotic expansion. Going forward, we assume that o = 0 , as the sum (S115) is symmetric in the orientation o and the arguments are unchanged.

To do so, we begin by writing writing π ′ = ( a uv / |E| ) u,v for the initial distribution provided to Y 1 . To analyze p n ( u, v, β ) := P n ( A β ( u, v, 0)) , note that when β = 1 we trivially have that this probability equals a uv / |E| and we know that |E| ∼ p n 2 ρ n E W (1) . In the case where β ≥ 2 , we consider the set of sequences α = ( α 0 , . . . , α β -2 ) ∈ V β -1 , where we then have that

for β ≥ 3 .

To study these sums, we begin by noting that they are asymptotic to their versions where we replace P → ˜ P . Indeed, we note that if we have positive sequences ( a i ) and ( b i ) , then

and so the fact that we know P ∼ p ˜ P uniformly, means that we can apply this to obtain asymptotic formulae for their sums also. With this, if we write N ( λ s , λ t ) for the denominator of ˜ P t → s,s → u , p n ( u, v, β ) can be asymptotically be decomposed into a linear combination of terms (bounded in number by a function of k independent of n ) of the form

where:

we write ˜ α for the concatenation ( α, u, v ) , meaning ˜ α is of length β +1 , with ˜ α k = α k for k ≤ β -1 , ˜ α β = u and ˜ α β +1 = v ;

c ( p, q ) = ( q -1 ) β -| J |-| K | (1 -q -1 ) | J | ( p -1 -q -1 ) | K | is a polynomial in p -1 and q -1 ;

J and K are possibly empty subsets of { 1 , . . . , β } which are disjoint.

The more tedious part to handle is when the set K is non-empty; as each delta function acts to contract the sum along one variable, doing so allows us to rewrite (S124) as

after a) performing some relabeling of the indices and modification to the set J , to give a new set ˜ J which is a subset of { 1 , . . . , β - | K |} and b) introducing some multiplicities n i which sum to β -1 . By Theorem S26 we uniformly have that this quantity is asymptotic, uniformly over all the free variables in the expression, to

where we write λ ′ = ( ˜ λ 0 , . . . , ˜ λ β -2 -| K | , λ u , λ v ) and ˜ λ is an independent copy of λ , and N ′ ( λ u , λ v ) := ( nρ n ) -1 N ( λ u , λ v ) . As nρ n →∞ under the prescribed conditions, we only need to consider leading terms of the order ρ -1 n n 2 , which shows that the sampling probability is asymptotic (uniformly over all vertices) to ρ -1 n n 2 for some function g P ( λ u , λ v ) . To argue that this function is bounded above away from zero, we note that the terms where | J | + | K | > 0 will be asymptotically negligible, and the remainder of the terms give a positive weighted sum.

E.1.2 Proof of Theorem S2

To understand the selection probability for the vertex pair ( u, v ) to be selected via negative sampling, define the events

so then

We note that

As a result, we need to begin by understanding the asymptotic probabilities of P ( A i ( v ) | G n ) and the unigram sampling probability. We begin with understanding the first probability. If i ∈ { 0 , 1 } , then we have that P ( A i ( v ) | G n ) = deg( v ) / 2 E n ∼ p W ( λ v , · ) /n E W (1) uniformly in v [11, Proposition 72]. For i ≥ 2 , we have that

using the same notation as in Appendix E.1.1. Consequently, via the same arguments as in Appendix E.1.1, it will be asymptotic to a positive linear combination of statistics of the form

where we write ˜ α = ( α, v ) for α ∈ V β . Using the same relabeling and arguments as given in Appendix E.1.1 will be asymptotic to

uniformly in all the free variables involved, where λ ′ = ( ˜ λ 0 , . . . , ˜ λ β -1 -| K | , λ v ) and ˜ λ is an independent copy of λ . (We note that while Theorem S26 is expressed in terms of concentration of quantities around functions which depend on both λ u and λ v , the exact same reasoning will apply for statistics which only end up depending on λ v .) In particular by taking the highest order terms of this expansion, we have that there exists some measurable function g i ( · ) which is bounded below and above, for each i , such that P ( A i ( u ) | G n ) ∼ p n -1 g i ( λ u ) uniformly in u .

As for the unigram sampling term, we note that by [11, Proposition 77] we have that

uniformly in the vertices v, u . With this, we note that the same arguments via self-intersection allow us to argue that

which gives the claimed result.

E.2 Chaining and bounds on Talagrand functionals

In this section, let L > 0 denote a universal constant (which may differ across occurrences) and K ( α ) a universal constant which depends on a variable α (but for fixed α also differs across occurrences). For a metric space ( T, d ) , we define the diameter of T as

We also define the entropy and covering numbers respectively by

We then define the Talagrand γ α functional [43] of the metric space ( T, d ) by

where the infimum is taking over all admissable sequences ; these are increasing sequences ( A n ) n ≥ 0 of T such that |A 0 | = 1 and |A n | ≤ 2 2 n for all n , with A n ( t ) being the unique element of A n which contains t . We will shortly see that this quantity helps to control the supremum of empirical processes on the metric space ( T, d ) . We first give some generic properties for the above functional.

Lemma S18. a) Suppose that d is a metric on T , and M > 0 is a constant. Then γ α ( T, Md ) = Mγ α ( T, d ) . If U ⊆ T , then γ α ( U, d ) ≤ γ α ( T, d ) .

Suppose that ( T 1 , d 1 ) and ( T 2 , d 2 ) are metric spaces, so d = d 1 + d 2 is a metric on the product space T = T 1 × T 2 . Then γ α ( T, d ) ≤ K ( α )( γ α ( T 1 , d 1 ) + γ α ( T 2 , d 2 )) .

We have the upper bounds

Suppose that ∥ · ∥ is a norm on R m , d is the metric induced by ∥ · ∥ , and B A = { x : ∥ x ∥ ≤ A } . Then one has the bound N ( B A , d, ϵ ) ≤ max { (3 A/ϵ ) m , 1 } , and consequently γ α ( B A , d ) ≤ K ( α ) Am 1 /α .

Proof. The first statement in a) is immediate, and the second part is Theorem 2.7.5 a) of Talagrand [43].

For part b), suppose that A i n are admissable sequences for ( T i , d i ) such that

If we then form the sequence of sets B n := { A 1 × A 2 : A i ∈ A i n -1 } for n ≥ 1 and B 0 = T 1 × T 2 , we have that B n is a partition of T for each n , |B 0 | = 1 and |B n | = |A 1 n -1 | · |A 2 n -1 | ≤ 2 2 n for each n , meaning that B n is an admissable sequence for the metric space ( T, d ) . Moreover, note that we have

for all sets A 1 ⊆ T 1 , A 2 ⊆ T 2 and t 1 ∈ T 1 , t 2 ∈ T 2 . As a result, if write B n ( t 1 , t 2 ) = A 1 n -1 ( t 1 ) × A 2 n -1 ( t 2 ) for the unique set in B n for which the point ( t 1 , t 2 ) lies within it, then we have that

In particular, taking supremum over all t ∈ T then gives the result, as the resuling LHS is lower bounded by γ α ( T, d ) , and the resulting RHS is upper bounded by 2( γ α ( T 1 , d 1 ) + γ α ( T 2 , d 2 )) .

For part c), the first inequality is Corollary 2.3.2 in Talagrand [43]. As for the second inequality, note that if ϵ ≤ e n ( T ) , then N ( T, d, ϵ ) > 2 2 n and consequently N ( T, d, ϵ ) ≥ 2 2 n +1 (recall that both quantities are integers). Writing N n = 2 2 n , this implies that

As log(1 + N n ) ≤ 2 n log(2) for all n ≥ 0 , summation over all n ≥ 0 implies that

As we have that

combining this and the prior inequality gives the stated result.

For part d), we can calculate that

For the remaining integral, note that if we make the substitution y = exp( -t α ) , then the integral equals

which we recognize as the mean of an Exp(1) random variabe in the case where α = 1 , and the variance of an unnormalized N(0 , 2) density in the case where α = 2 , and so in both cases the integral is finite. The desired conclusion follows.

Before stating a corollary of this result involving bounds on the γ -functional of some of the sets introduced in Theorem S5, we discuss some of the properties of these sets.

Lemma S19. Define the sets

Moreover, define the metrics

defined on the space R n × d × R n × d of pairs of n × d matrices. Then we have that for U 1 , U 2 , V 1 , V 2 ∈ B F ( A F ) ∩ B 2 , ∞ ( ˜ A 2 , ∞ ) that

Moreover, if U ∈ B 2 , ∞ ( A ) , then U ∈ B F ( √ nA ) also, and consequently if U ∈ B 2 , ∞ ( ˜ A 2 , ∞ ) then we have that U ∈ B 2 , ∞ ( ˜ A 2 , ∞ ) ∩ B F ( √ n ˜ A 2 , ∞ ) .

Proof. Begin by noting that, if U 1 , V 1 , U 2 , V 2 ∈ R n × d are matrices, then we have that

and similarly

As a result, we therefore have that in the case where U 1 , V 1 , U 2 , V 2 all have ∥ · ∥ F ≤ A F , then

and similarly if each of U 1 , V 1 , U 2 , V 2 have ∥ · ∥ 2 , ∞ ≤ ˜ A 2 , ∞ then

giving the first result of the lemma. The second part follows by noting that

and taking square roots.

Corollary S20. With the same notation as in Lemma S19, and writing T = B F ( A F ) ∩B 2 , ∞ ( ˜ A 2 , ∞ ) , we have that for any constant C > 0 that

Proof. This is a combination of Lemma S18 and Lemma S19

We now state a result which illustrates the usefulness of the above quantity when trying to control the supremum of empirical processes on a metric space ( T, d ) .

Theorem S21. Suppose ( X t ) t ∈ T is a mean-zero stochastic process, where d 1 and d 2 are two metrics on T . Suppose for all s, t ∈ T we have the inequality

Then we have that

Proof. This can be found within the proof of Theorem 2.2.23 in Talagrand [43].

Corollary S22. With the notation of Theorem S5, Lemma S19 and Corollary S20, if we have the bound

then as a consequence we can deduce that

Proof. This is a consequence of Corollary S20 and Theorem S21.

E.3 Matrix Algebra

Proposition S23. Suppose that we have matrices U, X ∈ R n × d with n ≥ d , and suppose that X is a full rank matrix so σ d ( XX T ) > 0 . Then we have that

Now instead suppose we have matrices U, V ∈ R n × d and a matrix M ∈ R n × d of rank d . Let M = U M Σ V T M be a SVD of M . Moreover suppose that U T U = V T V , and ∥ UV T -M ∥ op ≤ σ d ( M ) / 2 . Then we have that

Proof. The first part of the theorem statement is Lemma 5.4 of Tu et al. [44]. For the second part, we note that by Proposition S24, we can let U = U M Σ 1 / 2 Q and V = V M Σ 1 / 2 Q for some orthonormal matrix Q , where ˜ U ˜ Σ ˜ V T is the SVD of UV T . As a result, we can therefore apply without loss of generality Lemma 5.14 of Tu et al. [44], which then gives the desired statement.

Proposition S24. Suppose that U, V ∈ R n × d are matrices such that UV T = M for some rank d matrix M ∈ R n × n . Moreover suppose that U T U = V T V . Let M = U M Σ V T M be the SVD of M . Then there exists an orthonormal matrix Q ∈ O ( d ) such that V = V M Σ 1 / 2 Q . In particular, the symmetry group of the mapping ( U, V ) → UV T under the constraint U T U = V T V is exactly the orthogonal group O ( d ) .

Proof. Begin by noting that the condition U T U = V T V forces there to exist an orthonormal matrix R ∈ O ( n ) such that RU = V (e.g by Theorem 7.3.11 of Horn and Johnson [21]). As a consequence, we therefore have that M = R -1 V V T . This is a polar decomposition of M , and therefore as the semi-positive definite factor is unique, we have that V V T = ( V M Σ 1 / 2 )( V M Σ 1 / 2 ) T , where M = U M Σ V T M is the SVD of M , and we highlight that the polar decomposition of M is usually represented by M = ( U M V -1 M ) · ( V M Σ V T M ) . As V V T = ( V M Σ 1 / 2 )( V M Σ 1 / 2 ) T , again by e.g Theorem 7.3.11 of Horn and Johnson [21] we have that there exists an orthonormal matrix Q ∈ O ( d ) such that V = V M Σ 1 / 2 Q , giving the desired result.

Lemma S25. Suppose X ∈ R n × n is a symmetric matrix such that X = Π A Π T where A ∈ R d × d is of full rank, and Π ∈ R n × d is the assignment matrix for a partition of [ n ] ; that is, there exists a partition of [ n ] into d sets B (1) , . . . , B ( d ) such that Π il = 1[ i ∈ B ( l )] . Suppose further that Π is of full rank. Then we have that σ d ( X ) ≥ σ d ( A ) × min l | B ( l ) | .

where (Π∆ -1 ) is an orthonormal matrix. As a result, we can simply concentrate on the spectrum of the matrix ∆ A ∆ . As the smallest singular value of a matrix product is less than the product of the smallest singular values, the stated result follows.

E.4 Concentration inequalities

Theorem S26. Suppose that H is a graph on a vertex set { r 1 , . . . , r l , v 1 , . . . , v m } where the vertices r i are referred to as root vertices, and the remaining vertices as free vertices. We refer to such a graph as a rooted graph . Suppose that all the edges in H have at least one free vertex as an endpoint. Write x = ( x 1 , . . . , x m ) for the collection of m variables x i , and let Y be a statistic of the form

where the random variables t x i ,x j are independent and { 0 , 1 } valued with c p ≤ P ( t x i ,x j = 1) ≤ 1 -c p for all x i , x j ; the coefficients c g ≤ g x ≤ ∥ g ∥ ∞ < ∞ for some c g > 0 ; and i ∼ H j iff ( i, j ) is an edge within the graph H . Suppose that ρ n = n -α for some α < 1 /m ′ ( H ) where m ′ ( H ) = max 2 ≤ j ≤ k ( j -1) / ( v ( j ) -2) , v ( j ) = min | A |≥ j v ( A ) and v ( A ) for a set of edges A indicates the number of vertices in A . Then there exist constants c, δ, ∆ which depend only on c g , c p , ∥ g ∥ ∞ , H and α such that

for all ∆ ≤ λ ≤ n δ .

Proof. Without loss of generality suppose that ∥ g ∥ ∞ = 1 . The proof is essentially the same as Vu [48, Corollary 6.4], where we extend the result derived for the asymptotics of subgraph counts to that of a weighted count of rooted subgraph counts. To do so, we introduce some notation introduced within [48]. If H has k edges, and A is a set of pairs { x i , x j } , we write ∂ A T for the polynomial ∏ x ∈ A ∂ x T when interpreting T as a formal sum in the variables a x i ,x j (which we recall are { 0 , 1 } valued. We then define for 1 ≤ j ≤ k the quantities

Let v ( A ) denote the number of vertices specified within the set A , and let v ( j ) -min | A |≥ j v ( A ) . With this, we note that E [ Y ] = Θ( n m ρ k n ) and E [ ∂ A Y ) = Θ( n m -v ( A ) ρ k -| A | n ) . Consequently, we have that

where the implied constants depend only on k , c g and c p . The same arguments as given in Claim 6.2 and Corollary 6.4 in [48] can then be applied verbatim to give the claimed result.

Lemma S27. Let T be a statistic of the form

̸

̸

where c g ≤ g ( · ) ≤ ∥ g ∥ ∞ < ∞ . Then we have that

Consequently, if we define

where c g ≤ g ( · ) ≤ ∥ g ∥ ∞ < ∞ as above, then we have that

where the implied constant depends only on m and c g .

̸

̸

Proof. The first part is an immediate consequence of Hoeffding's inequality for U-statistics [39], which states that for U = (( n -m )! /n !) · T that

by substituting in t ↦→ t E [ U ] and making use of the bound E [ U ] ≥ c g .

For the second part, we work conditionally on λ l , λ k and note we can decompose T l,m for each l, m into a sum of statistics of the form T ′ , one of order Θ p ( n m ) and ( m k ) of order Θ p ( n m -k ) (corresponding to when some of the indices x i are equal) for 1 ≤ k ≤ m . By applying the first concentration inequality to these m ! · n 2 random variables, conditional on the ( λ l , λ k ) , we note the RHS is independent of these quantities, and so the probability bounds hold unconditionally. Consequently, we know that asymptotically T l,k is asymptotic to T ′ l,k , from which we can then apply the resulting concentration bound for this term.

Theorem S28. Suppose we have a statistic of the form

where ˜ α = ( α, u, v ) is a concatenation of α , u and v in order, g : R β +1 → R is a positive function which satisfies c g ≤ g ≤ ∥ g ∥ ∞ < ∞ for some constant c g , and J is a possibly empty set of indices. Define λ ′ = ( ˜ λ 0 , . . . , ˜ λ β -1 , λ u , λ v ) where ˜ λ is an independent copy of λ . Further define the statistic

Then for any ρ n = n -α for α sufficiently small, we have that

Proof. For this, we apply the above results. We begin by working conditionally on all of the λ , whose collection we denote λ , and note that by Theorem S26 by taking λ = (log n ) k for some k > 1 and a union bound, we have that

uniformly over all O ( m 2 m ! · n 2 ) random variables with probability 1 -exp( O ((log n ) k )) . As we have that

where the function is bounded below by c g · c β + | J | p and is bounded above by ∥ g ∥ ∞ , we can make use of Lemma S27 to show that

from which the claimed result follows.

Remark 1. One natural question to ask about the necessity of the range of values of ρ n specified above. Generally speaking, one can show for Erdos-Renyi graphs G ( n, p ) that the number of subgraphs Y H of H in G n satisfy a zero-one law, where

for some constant c ( H ) which relates to the geometry of the graph G [6]. In the latter regime, one can then show that Y H ∼ E [ Y H ] asymptotically again, and in the former this shows that the term is asymptotically negligible. As the purpose of this result is to derive an asymptotic expansion for the sum of various statistics of the form of T to the highest order, provided ρ n is of an order which avoids any of the "phase transition" stages of the form above we could eventually generalize our results further. As this involves even more additional book-keeping, we do not do so here.

Lemma S29. Let I be a finite index set of size | I | = m . Suppose that there exist constants τ > 0 , a bounded non-negative sequence ( p i ) i ∈ I such that p i ≤ τ -1 for all i , and a real sequence ( t i ) i ∈ I . Define the random variable

Then for all u > 0 , we have that

Proof. This follows by an application of Bernstein's inequality, by noting that X is a sum of independent mean zero random variables X i = m -1 ( τ -1 a i -p i ) t i which satisfy

Lemma S30. Define the random variable

̸

̸

for some constants ( T ij ) . Write ∥ T ∥ 2 2 = ∑ i = j T 2 ij and ∥ T ∥ ∞ = max i = j | T ij | . Then we have that

̸

̸

In particular, when T ij = 1 for all i = j , we have that Y = O p (( n 2 ρ n ) -1 / 2 ) .

̸

Proof. Note that under the assumptions on the model (where we have that a ij = a ji and W ( λ i , λ j ) = W ( λ j , λ i ) for all i = j ), we can write

Note that

where we have used the inequality ( a + b ) 2 ≤ 2( a 2 + b 2 ) which holds for all a, b ∈ R . Consequently, as a result of Lemma S29, we have conditional on λ that

̸

As the right hand side has no dependence on λ , taking expectations gives the first part of the lemma statement. For the second part, note that if T ij = 1 for all i = j , then we have that ∥ T ∥ 2 2 ≤ n 2 and ∥ T ∥ ∞ = 1 , and consequently

In particular, this implies that Y = O p (( n 2 ρ n ) -1 / 2 ) .

E.5 Miscellaneous results

Lemma S31. Suppose that A ∈ R m × m is a matrix whose diagonal entries are α , and off-diagonal entries are β , so A ij = αδ ij + β (1 -δ ij ) , where δ ij is the Kronecker delta. Then A has an eigenvalue α +( m -1) β of multiplicity one with eigenvector 1 m , and an eigenvalue α -β of multiplicity m -1 , whose eigenvectors form an orthonormal basis of the subspace { v : ⟨ v, 1 m ⟩ = 0 } . For the subspace { v : ⟨ v, 1 m ⟩ = 0 } , we can take the eigenvectors to be

where e m,i are the unit column vectors in R m , The singular values of A are | α -β | and | α +( κ -1) β | . Consequently, we can write A = UV T for matrices U, V ∈ R m × m with UU T = V V T , where the rows of U satisfy

̸

Consequently, we then have that ∥ U i · ∥ 2 ≤ ( 2 | α + β ( m -1) | /m + | α -β | / 2 ) 1 / 2 for all i , and min i = j ∥ U i · -U j · ∥ 2 = ( | α -β | ) 1 / 2 .

Further suppose that β = -α/ ( m -1) . Then provided α > 0 , A is positive semi-definite, is of rank m -1 , with a singular non-zero eigenvalue αm/ ( m -1) of multiplicity m -1 . Consequently one can write A = UU T where U ∈ R m × ( m -1) and whose columns equal the √ αm/ ( m -1) v i . In particular, the rows of U equal

Consequently, one has that ∥ U i · ∥ 2 = √ αm/ ( m -1) for all i , and moreover we have the separability condition min 1 ≤ i<j ≤ m ∥ U i · -U j · ∥ 2 = ( αm/ ( m -1)) 1 / 2 .

Proof. It is straightforward to verify that A has an eigenvalue of α + ( n -1) β with the claimed eigenvector. For the second part, we note that the characteristic polynomial of A is

and so A has m -1 eigenvalues equal to α -β ; as A is symmetric, we know that we can always take eigenvectors to be orthogonal to each other, and consequently the eigenspace associated with such an eigenvalue must be a subspace of { v : ⟨ v, 1 m ⟩ = 0 } . As both of these subspaces are of dimension m -1 , it consequently follows that they are equal. We then highlight that if A is a symmetric matrix with eigendecomposition A = Q Λ Q T for an orthogonal matrix Q , then the SVD is given by Q | Λ | sgn(Λ) Q T , and we can write A = UV T with U = Q | Λ | 1 / 2 and V = Q sgn(Λ) | Λ | 1 / 2 such that UU T = V V T . This allows us to derive the remaining statements about the matrix A which hold in generality. The remaining parts discussing what occurs when β = -α/ ( m -1) follow by routine calculation.

Lemma S32. Let σ ( x ) = (1 + exp( -x )) -1 be the sigmoid function. Then there exists a unique y ∈ R which solves the equation

for α, γ, s > 0 and β ∈ R if and only if β < α and β + γ > 0 . Moreover, y > 0 if and only if β + γ/ 2 > α/ 2 .

Proof. Note that ασ ( x ) is a function whose range is (0 , α ) on x ∈ ( -∞ , ∞ ) , and is strictly monotone increasing on the domain. Similarly, β + γσ ( -y/s ) is strictly monotone decreasing with range ( β, β + γ ) , and so simple geometric considerations of the graphs of the two functions gives the existence result. For the second part, note that the ranges of the functions on the LHS and the RHS on the range y > 0 are [ α/ 2 , α ) and ( β, β + γ/ 2] respectively, and so the same considerations as above give the second claim.

Lemma S33. Let σ ( x ) = ( e x ) / (1 + e x ) be the sigmoid function. Then for any x, y ∈ R , we have that

where

Proof. Note that by the integral version of Taylor's theorem, for a twice differentiable function f one has for all x, y ∈ R that

Applying this to f ( x ) = -log σ ( x ) gives

where σ ′ ( x ) = e x / (1 + e x ) 2 . Applying this to f ( x ) = log(1 -σ ( x )) gives

As the integral terms are the same, we concentrate on lower bounding this quantity. To do so, we make use of the lower bound σ ′ ( x ) ≥ e -| x | / 4 (Lemma 68 of Davison and Austern [11]) which holds for all x ∈ R . We then note that if | x | , | y | ≤ A , then we have that

Alternatively, if we only make use of the fact that | x | ≤ A (without loss of generality - the argument is essentially equivalent if we only assume that | y | ≤ A ), then we have that

and consequently we get that

Lemma S34. Suppose that we have a function

Then if f ( X ) ≤ r , we have that m -2 ∑ m i,j =1 | X ij | ≤ r + r 1 / 2 .

Proof. To proceed, note that if we have that

for a non-negative random variable X , then by Jensen's inequality we get that

and consequently E [ X ] ≤ r + r 1 / 2 by decomposing E [ X ] into the parts where X ≥ 2 and X < 2 . Applying this result to the empirical measure on the | X ij | across indices i, j ∈ [ m ] gives the desired result.

̸

F Minimizers for degree corrected SBMs when α = 1

̸

In this section, we give an informal discussion of how to study the minimizers of R n ( M ) for degree corrected SBMs when the unigram parameter α = 1 . We begin by highlighting that R n ( M ) does not concentrate around its expectation when averaging over only the degree heterogenity parameters θ i , which rules out using a similar proof approach as to what was carried out earlier in Appendix 1.

Recall that we were able to derive that the global minima of R n ( M ) was the matrix

When α = 1 or the θ i are constant, this allows us to write M ∗ = Π M Π T where Π is the matrix of community assignments for the network and M is some matrix, which allows us to simplify the problem. If we supposed that the θ actually had some dependence on the c ( i ) and were discrete - in that θ i | c ( i ) = l ∼ Q l for some discrete distributions Q l for l ∈ [ κ ] , then we could in fact employ the same type of argument as done throughout the paper. The major change is that then the embedding vectors would each concentrate around a vector decided by both a) their community assignment, and b) the particular degree correction parameter they were assigned. This would then potentially effect our ability to perform community detection depending on the underlying geometry of these vectors. One possible idea would be to explore R n ( M ) partially averaged over the θ i - we divide the θ i into B bins where B = n β for some β ∈ (0 , 1) , and average over only over the refinement of the θ i as belonging to the different bins. This would be similar to the argument employed in Davison and Austern [11].

An alternative perspective to give some type of guarantee on the concentration of the embedding vectors is to study the rank of the matrix M ∗ . If we are able to prove that is of finite rank r even as n grows large, then we are able to give a convergence result for the embeddings as soon as the embedding dimension d is greater than or equal to r . To study this, it suffices to look at the matrix

and argue that this is low rank (due to the logarithm, we can write M ∗ as the difference between this matrix and a matrix of rank κ , which is therefore also low rank). The entry-wise logarithm is a complicating factor here, as otherwise it is straightforward to argue that the entry-wise exponential of this matrix is of rank 2. One can reduce studying the rank of the matrix M ∗ E to studying the rank of the kernel

of an operator L 2 ( P ) → L 2 ( P ) , where P is the product measure induced by θ and the community assignment mechanism c . As K M is of finite rank r if and only if it can be written as

for some functions ϕ i , ψ i , it follows that the matrix ( M ∗ E ) ij will be of finite rank r also. Indeed, this representation forces that M ∗ E = ΦΨ T for some matrices Φ , Ψ ∈ R n × r , meaning that M ∗ E is of rank ≤ r ; Corollary 5.5 of Koltchinskii and Giné [24] then guarantees convergence of the eigenvalues of the matrix M ∗ E to the operator K M so that M ∗ E is actually of full rank.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims in the abstract and introduction match the theoretical contributions of the paper. These are supported by experimental verification. Similarly, where theoretical results are not obtained we investigate these scenarios using simulation.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We highlight and discuss the assumptions required for the theoretical results presented within the paper and in detail in the appendix. We demonstrate empirically the performance of our procedure when these assumptions are relaxed, if theoretical results were not obtained.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Due to space constraints all proofs appear in the supplemental material. We provide intuition for these proofs in the paper where space allows. Complete proofs are included in the supplemental material, along with all required Lemmas and exact assumptions.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We detail the experimental setup used in this work in the supplemental procedure, along with providing all code required to run and replicate these experiments also.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have included an anonymized version of the code repository used to create all experimental results in this paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide sufficient detail in the main text to understand the experimental results presented. In the appendix, we completely detail all experimental details, along with providing the exact code used as supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: For all experimental results we either show error bars corresponding to one standard error or all simulation results (in the case of box plots).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computation required for individual experiments was relatively small (in terms of both memory and time) and is detailed in the appendix. These were run on a computing cluster.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have ensured the research conforms with the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper provides theoretical guarantees for community detection in a specific class of statistical network models. Any potential societal impacts, positive or negative, will be ancillary from the theoretical focus of this paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: As this work theoretical guarantees for community detection in a specific class of statistical network models, such safeguards are not applicable.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We credit the original owners of code and data used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide an anonymized zip file which details the code used to generate all results.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Crowdsourcing or human subjects were not used in this research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Crowdsourcing or human subjects were not used in this research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents theoretical results for the quality of embeddings learned by random walk based methods like DeepWalk and its generalization node2vec. More precisely, the main result shows that embeddings learned node2vec, and DeepWalk as a special case of node2vec, preserve the community structure of graphs that exhibit a community structure. This is formalized for graphs generated by a stochastic block model (SBM), a common tool for the analysis community detection algorithms. The paper is mostly theoretical with experimental results based on synthetic graphs generated by SBM.\n\nSTRENGTHS:\n- The paper provides theoretical insights into a widely used embedding algorithm.\n- The theoretical results are non-trivial, and the proof techniques are advanced.\n- Overall, the paper is well-written.\n\nWEAKNESSES:\n- The considered problem is too specific. Of course, DeepWalk and node2vec are widely used but community detection is just one of the application of node embeddings. In the appendix, the authors discuss node classification and link prediction but my understanding is that this is again based on community structures and not really convincing. There are other factors that influence node labels such as structural roles, node degree, etc.\n- The paper is about node2vec but the effect of the p and q parameters is not really analyzed. The results would have been much more interesting if they analyzed the interplay of p and q in node2vec and p and q in SBM. In particular, given a graph G generated by SBM(p, q) what values p, q in node2vec(p,q) would be necessary to preserve the community structure of G.\n- The informal Theorem 1 is actually confusing. It does not say anything about the structure of the graph or the number of communities c(u). If the graph consists of connected components or the number of communities is n, then the result is trivial.\n\nQUESTIONS:\nComment on the above issues, in particular why the influence of p and q in node2vec(p,q) is not thoroughly analyzed. I see the short discussion after Theorem 2 but cannot gain much from it. Also, Corollary 5 does not make sense as scenario i) is DeepWalk with p=q=1.  Should the corollary read $\\tilde{p} > \\tilde{q}$ ?\n\nLIMITATIONS:\nThe paper does not present a new approach so the limitation section does not really apply here. The assumptions for the theoretical results are correctly shown, if this can be considered a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTheoretical results showing the consistency of node2vec embedding for community detection in SBMs and DCSBMs using k-means clustering.  Versions of this problem have been approached in other work, for example, node2vec rephrased as a matrix factorisation problem [39], and rank $d$ approximations for DeepWalk [51]. This paper addresses the issues in previous works relating to approximations made within previous work, such as an Frobenius norm approximation to the cross-entropy loss, at the cost of weaker consistency results, $\\ell^2$-norm as opposed to two-to-infinity norm.\n\nSTRENGTHS:\nThe strength and novelty of this work lies in the technical results in the Appendix. The paper would be better served with this material given more attention, perhaps in a journal rather than conference paper. However, I was unable to give the material in the Appendix a full read through, so I am working under the assumption that these results are technically sound. More clarity would be needed to show how the results differ from previous work (see below).\n\nThere are some important insights about the use of node2vec for node clustering in different types of SBM and DCSBM. For example, setting $U = V$ in constrained DeepWalk where $p \\le q$ is unable to cope with disassortitive (DC)SBMs. This feels like a significant issue and I would be interested to learn more why random walk based embedding algorithms fail on bipartite-like networks. It is also interesting that node2vec on DCSBMs produce clusters that only depend on the SBM community and that the degree parameters $\\theta$ are unimportant. Being able to perform k-means clustering directly on the node2vec embedding is an advantage over, say, spectral methods where hyperplane or spherical projection is required to remove the effect of $\\theta$ from the embeddings.\n\nWEAKNESSES:\nWithout delving too deeply into the mathematics in the Appendix, it is not obvious to me how this paper extends the results in [10] and [11] which only get briefly mentioned on Lines 121-123. These two papers analyse the asymptotic distribution of the node2vec embedding for latent position graph models, a wide range of models which include the SBM and DCSBM. It is not clear how the theoretical results in this paper extend those results, except for looking at the consistency for k-means clustering, which is non-specific to the node2vec embedding stage. I don't want to say that the contribution in this work is 'poor', but it is hard to know what is new here in the current state.\n\nUnderstanding random walk embedding algorithms is important, but they have been overtaken by graph neural networks designed for this exact problem, rather than ideas shoehorned in from natural language processing.\n\nThe experiment shown in Figure 3 is somewhat misleading. I read this plot as showing that k-means clustering is unsuitable for spectral embeddings of DCSBM, like the political blog network, rather than spectral embedding failing.\n\nQUESTIONS:\nThe main issue is how do the results here compare to the work of [10] and [11]? Section 1.2 on Related Works describes how this work extends [39] and [51], but the fact that these two strongly relevant papers are only given a brief sentence feels a significant oversight.\n\nWhat conditions are needed/prevent two-to-infinity norm consistency compared to the $\\ell_2$-norm in this setting?\n\nWhat about other clustering algorithms other that k-means? The results show asymptotically k-means will give the correct clustering, other papers [41] show that there are issues for fixed $n$ which are avoided by other clustering algorithms such as fitting Gaussian mixture models. Practically, any matrix $M$ transformation $U \\rightarrow UM$ and $V \\rightarrow VM^{-\\top}$ is a valid embedding which will affect the k-means clusters.\n\nSimilar results for core-periphery SBMs (On a 'two-truths' phenomenon in spectral graph clustering, Priebe et al), another important type of SBM for practical embeddings?\n\nLIMITATIONS:\nError bars needed in Figures 1 and 2 showing the multiple experiments described in Lines 296-297. Needs comparison to other embedding algorithms, for example, spectral embedding. Misleading experiment given in Figure 3 (see above).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

comments: SUMMARY:
The performance of node2vec for community detection in the degree-corrected stochastic block model is studied. It is proven that for sufficiently sparse DCSBMs, node2vec followed by $k$-means results in a clustering with only a vanishing fraction of misclassified vertices. Experiments are performed to demonstrate the results.

STRENGTHS:
The paper is very well-written and pleasant to read. 

The results are extremely useful in understanding the performance of node2vec in downstream tasks.

WEAKNESSES:
The presented results are interesting, but not much intuition is provided.

Minor comments:
* The sentence from lines 21 to 24 is unnecessarily long and difficult.
* on line 110: shouldn't $d\ge O(\kappa)$ be $d=\Omega(\kappa)$? Or do you mean something else?
* Brackets mismatch in the $\log\sigma$ at the end of (10)
* The $\mathcal{E}$ (presumably the edge-set) of (7) is not introduced
* The notation $A_{2,\infty}$ in Theorem 2 collides with the adjacency matrix notation.
* Figures are too small

QUESTIONS:
Is there any hope of extending the results to partial recovery in the sparse regime? That is,  $\rho_n=\Theta(n^{-1})$ so that the average degree remains bounded as $n\rightarrow\infty$.

Can the results be extended to the case where the number of communities grows with $n$? If so, how fast can it grow with $n$?

The NMI$\approx0$ that is shown in Figure 3 is surprising. Can we improve this by taking a different Laplacian?

LIMITATIONS:
The authors have adequately addressed the limitations by clearly stating the assumptions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-08