Efficient and Local Parallel Random Walks

Random walks are a fundamental primitive used in many machine learning algorithms with several applications in clustering and semi-supervised learning . Despite their relevance , the first efficient parallel algorithm to compute random walks has been introduced very recently ( Łącki et al . ) Unfortunately their method has a fundamental shortcoming : their algorithm is non-local in that it heavily relies on computing random walks out of all nodes in the input graph , even though in many practical applications one is interested in computing random walks only from a small subset of nodes in the graph . In this paper , we present a new algorithm that overcomes this limitation by building random walk efficiently and locally at the same time . We show that our technique is both memory and round efficient , and in particular yields an efficient parallel local clustering algorithm . Finally , we complement our theoretical analysis with experimental results showing that our algorithm is significantly more scalable than previous approaches . 1 Introduction . Random walks are key components of many machine learning algorithms with applications in computing graph partitioning [ ST04 , GKL+21 ] , spectral embeddings [ CPS15 , CKK+18 ] , or network inference [ HMMT18 ] , as well as learning image segmentation [ MS00 ] , ranking nodes in a graph [ AC07 ] and many other applications . With the increasing availability and importance of large scale datasets it is important to design efficient algorithms to compute random walks in large networks . Several algorithms for computing random walks in parallel and streaming models have been proposed in the literature . In the streaming setting , Sarma , Gollapudi and Panigrahy [ DSGP11 ] introduced multipass streaming algorithms for simulating random walks , and recently Jin [ Jin19 ] gave algorithms for generating a single random walk from a prespecified vertex in one pass . The first efficient parallel algorithms for this problem have been introduced in the PRAM model [ KNP92 , HZ96 ] . In a more recent line of work , Bahmani , Chakrabarti , and Xin [ BCX11 ] designed a parallel algorithm that constructs a single random walk of length ` from every node in O ( log ` ) rounds in the massively parallel computation model ( MPC ) , with the important caveat that these walks are not independent ( an important property in many applications ) . This was followed by the work of Assadi , Sun and Weinstein [ ASW19 ] , which gave an MPC algorithm for generating random walks in an undirected regular graph . Finally , Łącki et al . [ ŁMOS20 ] presented a new algorithm to compute random walks of length ` from every node in an arbitrary undirected graph . The algorithm of [ ŁMOS20 ] still uses only O ( log ` ) parallel rounds , and walks computed are now independent . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . From a high level perspective , the main idea behind all the MPC algorithms presented in [ BCX11 , ASW19 , ŁMOS20 ] is to compute random walks of length ` by stitching together random walks of length /̀2 in a single parallel round . The walks of length /̀2 are computed by stitching together random walks of length /̀4 and so on . It is possible to prove that such strategy leads to algorithms that run in O ( log ` ) parallel rounds as shown in previous work ( this is also optimal under the 1-vs-2 cycle conjecture , as shown in [ ŁMOS20 ] ) . Note that this technique in order to succeed computes in round i several random walks of length 2i for all the nodes in the network in parallel . This technique is very effective if we are interested in computing random walks from all the nodes in the graph , or , more precisely , when the number of walks computed out of a node is proportional to its stationary distribution . However , this approach leads to significant inefficiencies when we are interested in computing random walks only out of a subset of nodes or for a single node in the graph . This is even more important when we consider that in many applications as in clustering [ GLMY11 , GS12 , WGD13 ] we are interested in running random walks only from a small subset of seed nodes . This leads to the natural question : Is it possible to compute efficiently and in parallel random walks only from a subset of nodes in a graph ? In this paper we answer this question in the affirmative , and we show an application of such a result in local clustering . Before describing our results in detail , we discuss the precise model of parallelism that we use in this work . The MPC model . We design algorithms in the massively parallel computation ( MPC ) model , which is a theoretical abstraction of real-world system , such as MapReduce [ DG08 ] , Hadoop [ Whi12 ] , Spark [ ZCF+10 ] and Dryad [ IBY+07 ] . The MPC model [ KSV10 , GSZ11 , BKS13 ] is the de-facto standard for analyzing algorithms for large-scale parallel computing . Computation in MPC is divided into synchronous rounds over multiple machines . Each machine has memory S and at the beginning data is partitioned arbitrarily across machines . During each round , machines process data locally and then exchange data with the restriction that no machine receives more than S bits of data . The efficiency of an algorithm in this model is measured by the number of rounds it takes for the algorithm to terminate , by the size of the memory of every machine and by the total memory used in the computation . In this paper we focus on designing algorithm in the most restrictive and realistic regime where S ∈ O ( nδ ) for a small constant δ ∈ ( 0 , 1 ) – these algorithms are called fully scalable . Our contributions . Our first contribution is an efficient algorithm for computing multiple random walks from a single node in a graph efficiently . Theorem 1 . There exists a fully scalable MPC algorithm that , given a graph G = ( V , E ) with n vertices and m edges , a root vertex r , and parameters B∗ , ` and λ , can simulate B∗ independent random walks on G from r of length ` with an arbitrarily low error , in O ( log ` logλB ∗ ) rounds and Õ ( mλ ` 4 +B∗λ ` ) total space . Our algorithm also applies to the more general problem of generating independent random walks from a subset of nodes in the graph : Theorem 2 . There exists a fully scalable MPC algorithm that , given a graph G = ( V , E ) with n vertices and m edges and a collection of non-negative integer budgets ( bu ) u∈V for vertices in G such that ∑ u∈V bu = B ∗ , parameters ` and λ , can simulate , for every u ∈ V , bu independent random walks on G of length ` from u with an arbitrarily low error , in O ( log ` logλB ∗ ) rounds and Õ ( mλ ` 4 +B∗λ ` ) total space . The generated walks are independent across starting vertices u ∈ V . The following remark clarifies the effect of parameter λ on the number of machines . Remark 1 . The parameter λ has nothing to do with the input of the algorithm , but is a trade-off parameter between space and round complexity . It is useful to think of it as λ = n for some ( not necessarily a constant ) , in which case we get a round complexity of O ( log ` logB∗/ ( log n ) ) ≤ O ( log ` / ) and a total memory of Õ ( mn ` 4 +B∗n ` ) . We can set to , for example , 1/ log log n , to get nearly optimal total space and Õ ( log ` ) round complexity . If we compare our results with previous works , our algorithm computes truly independent random walks as [ ŁMOS20 ] does . This is in contrast with the algorithm of [ BCX11 ] , which introduces dependent constructs not independent walks . Our algorithm has significantly better total memory than [ ŁMOS20 ] , which would result in memory Ω ( m · B∗ ) for generating B∗ walks out of a root node r. This comes at the cost of a slightly higher number of rounds ( logλB ∗ , a factor that in many applications can be considered constant ) . The main idea is to preform multiple cycles of stitching algorithms , changing the initial distribution of the random walks adaptively . More precisely , in an initial cycle we construct only a few random walks , distributed according to the stationary distribution – this is known to be doable from previous work . Then , in each cycle we increase the number of walks that we build for node r by a factor of λ and we construct the walks only by activating other nodes in the graph that contribute actively in the construction of the random walks for r. In this way we obtain an algorithm that is significantly more work efficient in terms of total memory , compared with previous work . Our second contribution is to present an application of our algorithm to estimating Personalized PageRank and to local graph clustering . To the best of our knowledge , our algorithm is the first local clustering algorithm that uses a number of parallel rounds that only have a logarithmic dependency on the length of the random walk used by the local clustering algorithm . Theorem 3 . For any λ > 1 , let α ∈ ( 0 , 1 ] be a constant and let C be a set satisfying that the conductance of C , Φ ( C ) , is at most α/10 and Vol ( C ) ≤ 23Vol ( G ) . Then there is an MPC algorithm for local clustering that usesO ( log ` · logλB∗ ) = O ( log log n · logλ ( Vol ( C ) ) ) rounds of communication and total memory Õ ( mλ ` 4 + B∗λ ` ) = Õ ( mλ+ λVol ( C ) 2 ) , where B∗ : = 10 6 log3 n η2α2 , ` : = 10 logn α and η = 110Vol ( C ) , and outputs a cluster with conductance O ( √ α log ( Vol ( C ) ) ) . Finally we present an experimental analysis of our results where we show that our algorithm to compute random walk is significantly more efficient than previous work [ ŁMOS20 ] , and that our technique scale to very large graphs . Additional related works . Efficient parallel random walks algorithm have also been presented in distributed computing [ DSNPT13 ] and using multicores [ SRFM16 ] . Although the algorithms in [ DSNPT13 ] require a number of rounds linear in the diameter of the graph . The results in [ SRFM16 ] are closer in spirit to our work here but from an algorithmic perspective the challenges in developing algorithms in multicore and MPC settings are quite different . In our setting , most of the difficulty is in the fact that there is no shared memory and coordination between machines . As a result , bounding communication between machines and number of rounds is the main focus of this line of research . From an experimental perspective an advantage of the MPC environment is that it can scale to larger instances , in contrast an advantage of the multicore approach is that it is usually faster in practice for medium size instances . In our case study , where one is interested in computing several local clusters from multiple nodes ( for example to detect sybil attack ( look at [ ACE+14 ] and following work for applications ) the MPC approach is often more suitable . This is due to the fact that the computation of multiple Personalized PageRank vectors at the same time often requires a large amount of space . Several parallel algorithm have also been presented for estimating PageRank or PersonalizedPageRank [ DSGP11 , DSMPU15 , BCX11 ] , but those algorithms either have higher round complexity or introduce dependencies between the PersonalizedPageRank vectors computed for different nodes . Finally there has been also some work in parallelize local clustering algorithm [ CS15 ] , although all previously known methods have complexity linear in the number of steps executed by the random walk/process used in the algorithm ( in fact , our method could potentially be used to speed-up this work as well ) . Notation . We work on undirected unweighted graphs , which we usually denote by G = ( V , E ) , where V and E are the set of vertices and the set of edges , respectively . We also have n : = |V | and m : = |E| , unless otherwise stated . We define matrix D as the diagonal matrix of degrees , i.e. , Di , i = d ( vi ) . Also , we let A be the adjacency matrix , where Ai , j = 1 if and only if there is an edge joining vi and vj , and Ai , j = 0 , otherwise .

Summary: This paper studies the problem of parallelizing single-source and subset random walk computations. The primary technical contribution is an algorithm with the number of parallel rounds that depends on the length of the random walk in a logarithmic way. In addition, two applications are discussed in the article: personalized PageRank estimation and local clustering.