Streaming Belief Propagation for Community Detection

Abstract: The community detection problem requires to cluster the nodes of a network into a small number of well-connected ‘communities’. There has been substantial recent progress in characterizing the fundamental statistical limits of community detection under simple stochastic block models.  However, in real-world applications, the network structure is typically dynamic, with nodes that join over time. In this setting, we would like a detection algorithm to perform only a limited number of updates at each node arrival. While standard voting approaches satisfy this constraint, it is unclear whether they exploit the network information optimally. We introduce a simple model for networks growing over time which we refer to as streaming stochastic block model (StSBM). Within this model, we prove that voting algorithms have fundamental limitations.  We also develop a streaming belief-propagation (STREAMBP)  approach, for which we prove optimality in certain regimes. We validate our theoretical findings on synthetic and real data


Reviews and author responses:

 Thanks for the very detailed responses. This answers my questions about the technical contribution. For this I am keeping my current positive evaluation.

 We thank the reviewer for careful reading and helpful comments.

“Strength: The problem considered in the paper is well motivated. To me the proposed results are novel. For the StSBM, the proposed algorithm StreamBP uses only local information and achieves an accuracy asymptotically as good as that of the offline algorithm. This is shown both theoretically and empirically. Although the authors only focus on the symmetric version of the model with k = 2, it seems the proof is extendable. In the experiment section it is shown that for k > 2, the proposed algorithm is still as good as the offline algorithm.

The paper is well written and organized for most of the time. The model is clearly defined in the preliminary section.”

We thank the reviewer for the comments, and would like to ensure that the proof for Theorem 2 is indeed extendable to $k \geq 3$ with more effort, and we will include it in our future version. 

“Weakness: I am not sure technically how much the proof in this paper is different from that in [MX16]. Despite of the streaming vs. offline change, it seems that the proof of Theorem 2 is similar to that in [MX16].”

The proof of Theorem 2 in appendix is completely different from that in [MX16]. We prove that StreamBP performs at least as well as offline BP under the current modeling assumption. This is a comparison between online and offline algorithm that does nto use any of the methods of [MX16]. In Corollary 2, we use Theorem 2, together with the results (but not the techniques) in [MX16]. Since [MX16] proves information-theoretic optimality  of offline BP under the stated conditions, our result in Theorem implies information theoretic optimality of StreamBP under the same conditions. 

“Instead of predicting the true cluster label of each node, what can be said about StreamBP if one is interested in identifying just the cluster membership, i.e., recovery up to permutation of clusters? The latter is a more common objective in SBM literature (e.g., [Abb17, ABH15]). In such case one does not need access to side information.”

In cases where side information is not available, local streaming algorithms (and in particular, StreamBP) cannot achieve non-trivial clustering even up to permutations. This follows from Corollary 1, and this is true for the metric with permutation incorporated (i.e., equation (2)). In cases where side information is available, permutation is actually not required for achieving optimal recovery rate.  However, for the purpose of consistency, we still include maximizing over permutations step in the statement of Theorem 2 and Corollary 2. We would like to clarify that the metric we use in equation (2) is for recovery up to a permutation. 

“There are some notations that are not clearly defined by the time of their first appearance:

Neighborhood B_R^t (Line 129)
What does \Xi do (Line 131)? Furthermore, it will be helpful if the authors can provide some example figures explaining StSBM in Section 2. A simple way to do so would be showing the evolution of a network for several t's.”

$B_R^t$ is defined earlier in section 2.1. $\xi$ is simply some randomness we include here allowing us to consider randomized algorithms. For instance, if $\xi \sim Bernoulli(1/2)$, then our algorithm may decide to simply ignore each node with probability 0.5.

We have a figure for StreamBP (Figure 1), which shows the evolution of a network for one round. We agree that adding a figure illustrating the evolving process of StSBM is a good idea, and will definitely consider it in our future versions. 


 We thank the reviewer for careful reading and helpful comments..

“NEGATIVE COMMENTS

I may have missed it, but it seems a and b are known for the message passing algorithm. This seems like a strong assumption, and I would have liked to see it discussed.”

This is an excellent question. In Appendix A we showed that the performance of our algorithms is robust to the choice of a and b, and therefore replacing the true a,b by consistent estimates produces a negligible additional error.

If (potentially noisy) ground-truth labels are available for a subset of vertices, a and b can be estimated as we did in Section 5.3 (line 335-336). This estimator is obviously consistent.

If no such labels are not available, we can use the algorithm in [3], Theorem 3. This provides an efficient and consistent estimator of the connection probabilities. 

In summary, the case of unknown parameters a,b can be reduced to the case of known a,b. The real challenge is addressing the latter (and indeed many papers in this area assume a,b known [1,2]).

“Similarly, we would intuitively expect that the number of communities could grow when we add nodes. A comment on the fact that this is not the setting considered and why could improve the paper.”

The problem of considering a growing number of communities is interesting. We would like to point out that this can be addressed by a modification of our framework. We can stipulate that k is the number of communities at the final time (when all nodes have joined). While in the paper nodes join uniformly at random (and hence the proportion of nodes of each community is the same at any time), we could decide that the appearance time of a node depends on its community label. At the extreme, we could have that all nodes of community 1 join first, then all nodes of community 2, and so on. More generally, this modification could model a variety of scenarios. While studying this generalization is an interesting research direction, we believe it could be addressed by extending our approach.

“Minor comments:

line 139, adding a line saying "all nodes connected to each nodes in the neighborhood R of v are now connected through v while the rest of the graph stays unchanged" would help understanding the following lines faster, since the notation is heavy.”

Thanks for the suggestion, we will include this sentence in our future version.

“line 142, "Let us emphasize that the ‘neighborhoods’ Gt are not symmetric" could be explained further, for instance stating that neighborhood are updated only up to depth R.”

We agree with this point, and we will add that explanation into our future version. If space permits, we will also add an example to help with the illustration. 





References

[1] Mossel, E., & Xu, J. (2016, January). Local algorithms for block models with side information. In   Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science (pp. 71-80).

[2] Mossel, E., Neeman, J., & Sly, A. (2014, May). Belief propagation, robust reconstruction and optimal recovery of block models. In Conference on Learning Theory (pp. 356-370). PMLR.

[3] Mossel, E., Neeman, J., & Sly, A. (2015). Reconstruction and estimation in the planted partition model. Probability Theory and Related Fields, 162(3), 431-461.


 We thank the reviewer for careful reading and helpful comments.

“There has been a lot of work on stochastic block models, and paper studies a problem of reconstructing an SBM in which nodes appear in a sequence. The main result showing that side information gives the same performance as the offline algorithm is quite interesting. The paper is technically quite involved. The experiments match the theoretical results. The notion of streaming and the specific variation of SBM considered here is a bit confusing--- the algorithms are not streaming in the standard models, and the motivation for the graph model is not very clear.”

Our model compares to the streaming model as follows:

(1) The graph is revealed sequentially. At each step a neighborhood of depth $R$ of a new node is revealed. For a graph of average degree $d$, this is $O(d^R)$ bits revealed at each step. When both $d$ and $R$ are bounded, $O(d^R) = O(1)$.

(2) The algorithm keeps in memory the labels of all vertices (which requires $O(n)$ memory), but not all the graph (which would be $O(dn)$ memory).

(3) At each step, we constrain the algorithm to perform only $O(d^R)$ operations ($=O(1)$ in the present setting).

The main difference with respect to more standard streaming models is that we allow for a relatively large memory $O(n)$ to keep the labels of all vertices, but we constrain the amount of computation at each step. The requirement to keep in memory the labels is motivated by the objective to be able to output the whole vector of labels at any time. Also notice that $O(n)$ memory can be in practice significantly smaller than the memory required to keep the whole graph, $O(dn)$ (even if the average degree is bounded, say $d=30$).

We agree that the use of the term “streaming” is somewhat different than in the “streaming algorithms” literature. “Dynamic” could be an alternative choice, but it seems too generic to us. Our model is motivated  by applications, see line 18-24. 

“The presentation could be improved quite a bit to make it more readable. There is quite a bit of notation, which needs some work, and a table can be helpful.”

Adding a table is a nice suggestion, we will definitely consider that if space permits. 

“Additional comments:

lines 88-91: it would be useful to give some more motivation for the noisy labels \tilde{\tau}. Is the specific model here relevant in some application? It is not clear if "side information" is the right name for this”

Thank you for the comment.  In many applications, noisy labels can be computed by a classification algorithm which uses data available at each node. For instance, in a social network we have the profile of each user (posts, demographics, activity). This is sometimes referred to as “contextual information”. They can be used to train a classification or clustering algorithm. The output of this algorithm can be modeled as a noisy observation of the true labels. 

Notice that more detailed models of this contextual information have been developed, see e.g. [3]
and references therein. However, in the present paper we abstract away these details. 
Finally,  the problem with side information is standard in the mathematical literature and there is a tight relationship between the two settings [1,2].

“eqn (2): shouldn't it be \tau\circ\pi, instead of the other way around? The permutation is on the nodes, not the labels”

The permutation is on the labels, as in line 123, where [k] = {1,2,...,k} is the set of all labels. For instance, suppose we have four nodes {a,b,c,d}, and their true labels are {a:1, b:1, c:2, d:2}, and the output of our algorithm gives {a:2, b:2, c:1, d:1}, then after permutation on the labels we still get a perfect recovery.

“Lines 118-122: if we can assume that the arrival order is known, then the accuracy metric for the algorithm in (2) is not clear. Wouldn't the permutation \pi be fixed in that case?”

The permutation is not fixed, as it is actually permutation over labels. Using the examples given above for illustration, if the output of our algorithm gives {a:2, b:2, c:1, d:1}, then the permutation maximizing this would be {$\pi$(2) = 1, $\pi$(1) = 2}. If the output of our algorithm gives {a:1, b:1, c:2, d:2}, then the permutation maximizing this would be {$\pi$(1) = 1, $\pi$(2) = 2}.

“The streaming aspect of the StSBM model is confusing. Looks like the edges and labels are defined independent of the node ordering.”

If we consider the edges and labels on the whole graph, then they are independent of the node ordering. If we consider the output as a sequence of graphs as defined in line 83, then edges and labels are not independent of the node ordering: node ordering determines when a node and an edge appears in this graph sequence. 

“It would be useful to explain the implication of Theorem 1 a bit more in the remarks. For instance, how large does r_{\epsilon} have to be?”

Currently our theorem says $r_{\epsilon}$ is fixed and is independent of $n$, thus in the limit $n \rightarrow \infty$, we make the argument that any local streaming algorithm is gathering information from a bounded radius neighborhood with high probability. 

According to our proof we can choose $r_{\epsilon}$ to be $2^{2R + 3}R(1 + (1 - e^{-d})^{-1}\sum\limits_{i = 0}^R d^i) d^{2R}e$, where $R$ is the radius of local streaming algorithm, and $d$ is the average node degree, which depends only on the connection probabilities. We agree it will be useful to add such remarks, and we thank the reviewer for pointing this out. 

“section 4 talks about a contrast with the assymetric model, which hasn't been defined so far.”

Sorry for not making this clear enough. In line 106 we defined the symmetric model, and the asymmetric model is just the original definition introduced above in lines 99-102 (without adding the symmetry constraints mentioned in lines 106-107). 

“The use of the term "streaming" is a bit confusing, considering the standard notion of streaming algorithms. For instance, Algorithm 1 is not implemented in the standard streaming model.”

See the arguments above.

“section 5.3: are the parameters k=7 and k=6 good fit for the Cora and Citeseer networks? They also have skewed degree distributions, so is an SBM an approprate model to consider for them?”

We obtained the values k=7 and k=6 directly from the ground-truth labels from the two datasets. This is therefore an “oracle” choice that we make to understand the behavior of the algorithm under optimal choice of k. Furthermore, we believe that in real data sets, usually the total number of communities will be given as prior knowledge. 

Regarding the modeling assumptions:

(1) We tested our algorithm on real graphs that do not follow precisely the SBM on purpose. The empirical finding that the algorithm performs well in several such examples shows that it is robust to model misspecification.

(2) While understanding this robustness is an interesting open question, we can offer some intuition. It is true that real graphs present larger degree heterogeneities than in an SBM. However, these affect a relatively small number of vertices (in the case of Cora, ~3% of nodes have degrees above 10 and ~1% have degrees above 20). The algorithm seems to be more sensitive to the majority of nodes (in particular when clipping the beliefs). 


References

[1] Mossel, E., & Xu, J. (2016, January). Local algorithms for block models with side information. In   Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science (pp. 71-80).

[2] Kanade, V., Mossel, E., & Schramm, T. (2016). Global and local information in clustering labeled block models. IEEE Transactions on Information Theory, 62(10), 5906-5917

[3] Deshpande, Y., Sen, S., Montanari, A. and Mossel, E., 2018, January. Contextual Stochastic Block Models. In NeurIPS.


This paper proposes to do community detection in a variant of the stochastic block model for which nodes arrive in a streaming fashion, and there may exists noisy estimates of the real labels. They prove that if the noise render the side information uninformative, a local streaming algorithm is equivalent to a local algorithm, which cannot achieve non-trivial accuracy. If the side information is informative, they provide a belief propagation algorithm in the streaming setting which achieves comparable accuracy to offline BP asymptotically. Offline BP has been conjectured to be optimal, so their algorithm would also be optimal if the conjecture is proven (it is in special cases). They also provide another algorithm which is in practice more consistent than StreamBP.  #################################################################### 

                  DISCLAIMER                                                
####################################################################

I'm not an expert on community detection nor on the stochastic block model, and only have basic knowledge of it. I therefore cannot vouch for the novelty of their algorithm/proof techniques or the originality of the work. I did check their proofs and did not notice errors, but similarly I'm not sure I would have noticed if there was any.

####################################################################

                  OVERALL                                                
####################################################################

I liked this paper.
1. The writing is excellent. 
2. While the algorithm seems intuitive, the proofs appear (to me, a non-expert) technically strong.
3. The results are interesting and powerful if that conjecture holds.
4. The experimental section is convincing, with experiments on synthetic and real-life networks.
5. Too often, when a paper is technically strong, the experiments seem like an afterthought to show it's not terrible in practice. Here, it seems the chaotic behavior of StreamBP informed the design of StreamBP*, which in turn got its own analysis, which I appreciate.


####################################################################

                  POSITIVE COMMENTS                                                
####################################################################
+ The authors provides many insights and intuition throughout the paper (e.g. "An equivalent description is that[...] defines a Markov chain
over graphs", "one can obtain non-trivial estimation accuracy by simply using the degree of each vertex" etc.). This makes I very easy to follow the proof. It was excellent writing.
+ The results appear strong. There is a negative result when there is no side information, and an (up to a conjecture) optimal algorithm when there exists enough information.
+ Their algorithm is simple.
+ I was surprised online algorithms compared with offline SOTA in this setting.

####################################################################

                  NEGATIVE COMMENTS                                                
####################################################################
- I may have missed it, but it seems $a$ and $b$ are known for the message passing algorithm. This seems like a strong assumption, and I would have liked to see it discussed.
- Similarly, we would intuitively expect that the number of communities could grow when we add nodes. A comment on the fact that this is not the setting considered and why could improve the paper.

Minor comments:
- line 139, adding a line saying "all nodes connected to each nodes in the neighborhood R of $v$ are now connected through $v$ while the rest of the graph stays unchanged" would help understanding the following lines faster, since the notation is heavy.
- line 142, "Let us emphasize that the ‘neighborhoods’ Gt are not symmetric" could be explained further, for instance stating that neighborhood are updated only up to depth $R$. NA

The paper considers the problem of recovering the communities in a streaming version of
stochastic block models, using information within a bounded radius. The main technical
results are the limits of recovery with no side information, and a streaming version of
belief propagation which has the same reconstruction accuracy as the offline BP
algorithm using side information. The algorithms are evaluated on synthetic and real
networks, and show good performance.  There has been a lot of work on stochastic block models, and
paper studies a problem of reconstructing an SBM in which nodes appear in a sequence.
The main result showing that side information gives the same performance as the 
offline algorithm is quite intereesting.
The paper is technically quite involved. The experiments match the theoretical results.
The notion of streaming and the specific variation of SBM considered here
is a bit confusing--- the algorithms are not streaming in the standard models,
and the motivation for the graph model is not very clear.

The presentation could be improved quite a bit to make it more readable. There is quite
a bit of notation, which needs some work, and a table can be helpful.

Additional comments:

lines 88-91: it would be useful to give some more motivation for the noisy labels
\tilde{\tau}. Is the specific model here relevant in some application? It is not
clear if "side information" is the right name for this

eqn (2): shouldn't it be \tau\circ\pi, instead of the other way around? The permutation
is on the nodes, not the labels

Lines 118-122: if we can assume that the arrival order is known, then the accuracy metric
for the algorithm in (2) is not clear. Wouldn't the permutation \pi be fixed in that case?

The streaming aspect of the StSBM model is confusing. Looks like the edges and labels
are defined independent of the node ordering

It would be useful to explain the implication of Theorem 1 a bit more in the remarks.
For instance, how large does r_{\epsilon} have to be?

section 4 talks about a contrast with the assymetric model, which hasn't been defined
so far.

The use of the term "streaming" is a bit confusing, considering the standard notion of
streaming algorithms. For instance, Algorithm 1 is not implemented in the standard
streaming model.

section 5.3: are the parameters k=7 and k=6 good fit for the Cora and Citeseer networks?
They also have skewed degree distributions, so is an SBM an approprate model to
consider for them?

 There is no discussion on this

This paper studies a dynamic version of the stochastic block model, named as streaming stochastic block model (StSBM). The authors consider a sparse regime and analyze the asymptotic behavior of the model. In particular, the authors show that without some side information about the node labels, recovery of the true labels through a R-local streaming algorithm is no better than random guessing (Corollary 1). The authors also propose a streaming R-local belief propagation algorithm (StreamBP), and show their streaming algorithm perform as good as the offline version algorithm (Theorem 2). Empirical evaluations using synthetic and real-world datasets are provided in the paper.  Strength:
The problem considered in the paper is well motivated. To me the proposed results are novel. For the StSBM, the proposed algorithm StreamBP uses only local information and achieves an accuracy asymptotically as good as that of the offline algorithm. This is shown both theoretically and empirically. Although the authors only focus on the symmetric version of the model with k = 2, it seems the proof is extendable. In the experiment section it is shown that for k > 2, the proposed algorithm is still as good as the offline algorithm. 

The paper is well written and organized for most of the time. The model is clearly defined in the preliminary section.

Weakness:
I am not sure technically how much the proof in this paper is different from that in [MX16]. Despite of the streaming vs. offline change, it seems that the proof of Theorem 2 is similar to that in [MX16].

Instead of predicting the true cluster label of each node, what can be said about StreamBP if one is interested in identifying just the cluster membership, i.e., recovery up to permutation of clusters? The latter is a more common objective in SBM literature (e.g., [Abb17, ABH15]). In such case one does not need access to side information.

There are some notations that are not clearly defined by the time of their first appearance:
- Neighborhood B_R^t (Line 129)
- What does \Xi do (Line 131)?
Furthermore, it will be helpful if the authors can provide some example figures explaining StSBM in Section 2. A simple way to do so would be showing the evolution of a network for several t's.  This is a theoretic work.

Meta-review: Each reviewer has a positive opinion of this submission, which deals with a relevant problem. I recommend acceptance.

Decision: accept