SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf {C} ^ {\infty} = \left[ \begin{array}{c c c} \bullet & \bullet & 2 \updownarrow \\ 2 \updownarrow & \bullet & 2 \updownarrow \\ \updownarrow & \updownarrow & 2 \updownarrow \\ \end{array} + 2 \updownarrow & 2 \updownarrow \\ \updownarrow & \updownarrow & \updownarrow \end{array} \right] \tag {21}
$$
$$
\mathbf {C} ^ {\mathbf {B}} = \left[ \begin{array}{c c c} \bullet & 2 \downarrow & \uparrow \\ 2 \downarrow & 2 \downarrow + 2 \downarrow & 2 \downarrow \\ \uparrow & 2 \downarrow & \downarrow \end{array} \right] \tag {22}
$$
Note that, once unrooted, isomorphic graphs are equivalent, e.g.:  $\mathbf{y} = \mathbf{y}$
As promised, the eigenvalues of  $\mathbf{C}^{\mathbf{B}}(\mathbf{C}^{\infty})^{-1}$  are precisely the entries of  $\mathbf{B}$
$$
\operatorname {e i g v a l} \left(\mathbf {C} ^ {\mathbf {B}} \left(\mathbf {C} ^ {\circ \circ}\right) ^ {- 1}\right) = \left\{B _ {k k ^ {\prime}} \right\} _ {k \leq k ^ {\prime}} \tag {23}
$$
Moreover, the structure of their corresponding eigenvectors allows us to select specific entries  $B_{k k'}$ .
$$
\operatorname {e i g v e c} \left(\mathbf {C} ^ {\mathbf {B}} \left(\mathbf {C} ^ {\circ \circ}\right) ^ {- 1}\right) = \left\{\left[ \begin{array}{c} 1 \\ d _ {k} + d _ {k ^ {\prime}} \\ d _ {k} d _ {k ^ {\prime}} \end{array} \right] \right\} _ {k \leq k ^ {\prime}} \tag {24}
$$
Thus, for each pair of blocks  $k$  and  $k'$  with normalized degrees  $d_k$  and  $d_{k'}$ , we can estimate their connection probability  $B_{kk'}$ , estimate the eigenvalue of its corresponding eigenvector:
$$
B _ {k k ^ {\prime}} = \frac {\mathbf {v} ^ {\top} (\mathbf {C} ^ {\mathbf {B}} (\mathbf {C} ^ {\circ \circ}) ^ {- 1}) \mathbf {v}}{\mathbf {v} ^ {\top} \mathbf {v}} \qquad \mathbf {v} = \left[ \begin{array}{c} 1 \\ d _ {k} + d _ {k ^ {\prime}} \\ d _ {k} d _ {k ^ {\prime}} \end{array} \right]
$$
Remark 2 (Sufficiency of subgraph densities). Suppose we fix a finite number  $K$  of blocks, and consider all SBMs with  $K$  blocks. The  $2K - 1$  subgraph densities defined by all stars with up to  $2K - 1$  edges then completely determines the degrees and sizes of blocks. Adding the subgraph densities of all bistars with maximum degree at most  $K$  then completely determines the connection probabilities between blocks (as a function of their degrees), and therefore the entire distribution of the random graph. In other words, the vector of these densities of stars and bistars is a sufficient statistic for the class of degree-separated stochastic block models with  $K$  blocks.
4.3 Beyond Degree Correlations: Adding "Two-Hop" Subgraphs
The method described above can be thought of as a "basic" minimum working example of a much more general family of graph pencil methods. The subgraph densities in  $\mathbf{C}^{\circ}$  and  $\mathbf{C}^{\mathrm{d}}$  are sensitive only to the degree distribution, and the subgraphs in  $\mathbf{C}^{\infty}$  and  $\mathbf{C}^{\mathrm{B}}$  contain information about the degree-degree correlations. While this is sufficient to recover the parameters of an SBM when the normalized degrees of the blocks are well-separated, the method can be made more robust by adding additional subgraphs.
In particular, define the "two-hop" matrix of connection probabilities  $\Lambda = \mathbf{B} \bullet_{\pi} \mathbf{B}$ , where  $\bullet_{\pi}$  denotes the matrix product weighted by  $\pi$ , i.e.:
$$
\Lambda_ {i j} = \sum_ {k} B _ {i k} \pi_ {k} B _ {k j} \tag {25}
$$
These birooted subgraphs interact the others via gluing and unrooting (see table 3).
Table 3: Incorporating the "two-hop" connection probability  $\Lambda$  into the gluing algebra.
glyph	symbol	meaning
△	B •πB = Λ	two-hop connection probability
△	Λ o B	entrywise multiplication
△	π • (Λ o B) • π = μ(△)	homomorphism density of triangles
We can use this to add more columns to  $\mathbf{C}^{\infty}$  and  $\mathbf{C}^{\mathrm{B}}$ :
$$
\mathbf {C} ^ {\circ \circ} = \left[ \begin{array}{c c c c c c c} \bullet & 2 \uparrow & \uparrow & \uparrow & \uparrow & 2 \uparrow & 2 \uparrow \\ 2 \uparrow & 2 \downarrow & + 2 \uparrow & 2 \downarrow & \uparrow & 2 \downarrow & + 2 \uparrow \\ \uparrow & 2 \downarrow & \uparrow & \downarrow & \uparrow & 2 \downarrow & \uparrow \end{array} \right] \tag {26}
$$
$$
\mathbf {C} ^ {\mathbf {B}} = \left[ \begin{array}{c c c c c c c} \bullet & 2 \downarrow & \downarrow & \uparrow & \bigtriangleup & 2 \uparrow & \uparrow \\ 2 \downarrow & 2 \downarrow + 2 & 2 \downarrow & \downarrow & 2 \uparrow & 2 \downarrow + 2 & 2 \downarrow \\ \uparrow & 2 \downarrow & \downarrow & \uparrow & \uparrow & 2 \downarrow & \downarrow \\ \end{array} \right] \tag {27}
$$
Remarkably, we can use these larger matrices in exactly the same way as before (using the Moore-Penrose pseudoinverse). As shown in in figure 1, including these additional subgraphs can make the recovery of the SBM more robust.
Figure 1: Adding "two-hop" subgraphs helps recover (dis)assortativity.
We use 2-by-2 SBMs with varying degrees of assortativity to compare our basic method using bistars from section 4.2 (green squares) and the method that adds the two-hop subgraphs from section 4.3 (purple circles). The vertical axis measures the expected squared error of the probability of an edge between two random nodes, and shading denotes  $\pm 1$  standard deviation from the average value. The dashed orange line denotes the expected squared error if the latent blocks of the nodes were known, and both methods appear to converge at this optimal rate. When the SBM is particularly assortative (left) or disassortative (right), the inclusion of two-hop subgraphs results in a notable improvement.
5 From an Observed Graph to a Stochastic Block Model
In order to apply the graph pencil method to an observed graph, we need to obtain unbiased estimators of the homomorphism densities used to construct the  $\mathbf{C}$  matrices. In particular, for a subgraph  $g$  and a (large) graph  $G$ , we want to compute the fraction of injective maps  $\varphi$  from  $V(g)$  to  $V(G)$ , such that if  $(u,v) \in E(g)$ , then  $(\varphi(u),\varphi(v)) \in E(G)$ .
5.1 Quickly Counting (Injective) Homomorphisms
We start with the adjacency matrix  $\mathbf{A}$ , the identity matrix  $\mathbf{I}$ , and the all-ones matrix  $\mathbf{1}$ . Then, we perform our only difficult matrix multiplication:  $\sum_{k} A_{ik} A_{kj} = D_{ij} + \Lambda_{ij}$ , where  $\mathbf{D}$  is the diagonal
matrix of node degrees, and  $\Lambda$  is the (traceless) "two-hop adjacency" matrix. From these, we can recursively count all the relevant subgraphs using only entrywise multiplication.
We index our subgraphs with a tuple of non-negative integers  $(\ell, c, r)$ , corresponding to: the number of edges incident to only the "left" node, the number of two-hop paths between the "left" and "right" nodes, and the number of edges incident to only the "right" node  $j$ . First, we add the two-hop paths:
$$
\mathbf {M} ^ {(0, 0, 0)} = \mathbf {1} - \mathbf {I} \tag {28}
$$
$$
\mathbf {M} ^ {(0, c + 1, 0)} = \mathbf {M} ^ {(0, c, 0)} \circ (\boldsymbol {\Lambda} - c \mathbf {1}) \tag {29}
$$
Define the matrix of "left" degrees to be  $\mathbf{L} = \mathbf{D} \bullet (\mathbf{1} - \mathbf{I}) - \mathbf{A}$  (i.e., the degree of node  $i$  if node  $j \neq i$  were deleted, and zero when  $i = j$ ), and the matrix of "right" degrees to be its transpose  $\mathbf{R} = \mathbf{L}^{\top}$ . Next, we add single edges to node  $i$ , then to node  $j$ :
$$
\mathbf {M} ^ {(\ell + 1, c, 0)} = \mathbf {M} ^ {(\ell , c, 0)} \circ (\mathbf {L} - (\ell + c) \mathbf {1}) \tag {30}
$$
$$
\mathbf {M} ^ {(\ell , c, r + 1)} = \mathbf {M} ^ {(\ell , c, r)} \circ (\mathbf {R} - (r + c) \mathbf {1}) - \ell \mathbf {M} ^ {(\ell - 1, c + 1, r - 1)} \tag {31}
$$
Finally, if an edge connecting the left and right nodes is to be included, we put a mark on the middle integer:
$$
\mathbf {M} ^ {(\ell , c ^ {\prime}, r)} = \mathbf {M} ^ {(\ell , c, r)} \circ \mathbf {A} \tag {32}
$$
To obtain the counts of a subgraph  $g$  in the entire graph, simply sum the entries of the corresponding matrix  $\mathbf{M}^{(\ell, c, r)}$ . This is analogous to the "unrooting" operation from before.
$$
\llbracket \mathbf {M} ^ {(\ell , c, r)} \rrbracket = \sum_ {i} \sum_ {j} \mathbf {M} ^ {(\ell , c, r)} \tag {33}
$$
For example,  $\mathbb{[A]}$  is twice the number of edges, as the injective homomorphisms count both orientations. The injective homomorphism densities are obtained by dividing this count by the number of injective maps
$$
\mu \left(g ^ {(\ell , c, r)}\right) = \frac {\llbracket \mathbf {M} ^ {(\ell , c , r)} \rrbracket}{N (N - 1) \cdots (N - | V (g) | + 1)} \tag {34}
$$
The map from a stochastic block model to the density of a given subgraph can be computed analytically (e.g., [12]), or approximated by simulation (sample a graph from the model, count the number of occurrences of the given subgraph, and normalize appropriately). In contrast, the map from the subgraph densities to the stochastic block model is rather nontrivial. This paper provides a way to compute this map.
Since asymptotic rates are known for subgraph densities (see, e.g., [5] and [3]), and the mapping we construct here is differentiable in the interior of its domain, we should expect similar asymptotic rates for recovery of the model parameters. We do not currently know how the rate behaves at the boundaries. At the boundary, the method may in principle output invalid probabilities, as there is at present no explicit constraint enforcing the connection probabilities to be between 0 and 1. Hence, as many other methods for inferring SBMs, our method is well-behaved in the interior.
Finally, we note that the method presented here can be extended in a variety of ways, for example to directed edges, weighted edges, directed weighted edges, etc.
Code implementing our method is available at https://github.com/TheGravLab/TheGraphPencilMethod.
LMG and PO were funded by the Gatsby Charitable Foundation.
[21] Lovász, L. (2012). Large networks and graph limits, volume 60. American Mathematical Soc.
[22] Lovász, L. and Szegedy, B. (2010). The graph theoretic moment problem. arXiv:1010.5159.
[23] Markus, A. S. (1988). Introduction to the Spectral Theory of Polynomial Operator Pencils. American Mathematical Society.
[24] Pearson, K. (1894). Contributions to the mathematical theory of evolution. Philosophical Transactions of the Royal Society of London, 185:71-110.
[25] Peixoto, T. P. (2014). Hierarchical block structures and high-resolution model selection in large networks. Physical Review X, 4(1):011047.
[26] Prony, G. (1795). Essai experimental et analytique sur les lois de la dilatabilité des fluides élastiques, et sur celles de la force expansive de l'eau et de la vapeur de l'alkool, à différents températures. Journal de l'École Polytechnique (Paris), 1:24-76.
[27] Saeed, M., Javed, K., and Babri, H. A. (2013). Machine learning using Bernoulli mixture models: Clustering, rule extraction and dimensionality reduction. Neurocomputing, 119:366-374.
[28] Sauer, T. (2018). Prony's method: an old trick for new problems.
[29] Wolfe, P. J. and Olhede, S. C. (2013). Nonparametric graphon estimation. arXiv:1309.5936.
[30] Zhang, Y. and Xia, D. (2022). Edgeworth expansions for network moments. Annals of Statistics, 50(2):726-753.