SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We expand on this result in two ways. First, give a new bound for exact embedding in terms of arboricity, rather than max degree. This increases the applicability to real-world networks, which often are sparse (i.e., low arboricity) and have right-skewed degree distributions (i.e., high max degree). Second, we show that any rank- $k$  LPCA factorization can be converted to our model's symmetric nonnegative factorization with  $O(k)$  communities. This extends the guarantees on the LPCA model's power for exact representation of graphs, both the prior guarantee in terms of max degree and our new one in terms of arboricity, to our community-based model as well. In Appendix A.1, we also introduce a natural family of graphs - Community Overlap Threshold (COT) graphs - for which our model's community-based factorization not only exactly represents the graph, but also must capture some latent structure to do so with sufficiently low embedding dimensionality.
Arboricity bound for exact representation We will use the following well-known fact: the rank of the entrywise product of two matrices is at most the product of their individual ranks, that is,
$$
\operatorname {r a n k} (\boldsymbol {X} \circ \boldsymbol {Y}) \leq \operatorname {r a n k} (\boldsymbol {X}) \cdot \operatorname {r a n k} (\boldsymbol {Y}).
$$
Theorem 5.2 (Exact LPCA Factorization for Bounded-Arboricity Graphs). Let  $\mathbf{A} \in \{0,1\}^{n \times n}$  be the adjacency matrix of an undirected graph  $G$  with arboricity  $\alpha$ . Then there exist embeddings  $\mathbf{X}, \mathbf{Y} \in \mathbb{R}^{n \times (4\alpha^2 + 1)}$  such that  $\mathbf{A} = H(\mathbf{X}\mathbf{Y}^\top)$ .
Proof. Let the undirected graph  $A$  have arboricity  $\alpha$ , i.e., the edges can be partitioned into  $\alpha$  forests. We produce a directed graph  $B$  from  $A$  by orienting the edges in these forests so that each node's edges point towards its children. Now  $A = B + B^{\top}$ , and every node in  $B$  has in-degree at most  $\alpha$ .
Let  $V \in \mathbb{R}^{n \times 2\alpha}$  be the Vandermonde matrix with  $V_{t,j} = t^{j-1}$ . For any  $c \in \mathbb{R}^{2\alpha}$ ,  $[Vc](t) = \sum_{j=1}^{2\alpha} c(j) \cdot t^{j-1}$ , that is,  $Vc \in \mathbb{R}^n$  is a degree-(2α) polynomial with coefficients  $c$  evaluated at the integers  $t \in [n] = \{1, \ldots, n\}$ . Let  $b_i$  be the  $i^{\text{th}}$  column of  $B$ . We seek to construct a polynomial such that for  $t$  with  $b_i(t) = 1$ ,  $[Vc_i](t) = 0$ , and  $[Vc_i](t) < 0$  elsewhere; that is, when inputting an index  $t \in [n]$  such that the  $t^{\text{th}}$  node is an in-neighbor of the  $i^{\text{th}}$  node, we want the polynomial to output 0, and for all other indices in  $[n]$ , we want it to have a negative output. Letting  $N(i)$  denote the in-neighbors of the  $i^{\text{th}}$  node, a simple instantiation of such a polynomial in  $t$  is  $-1 \cdot \prod_{j \in N(i)} (t - j)^2$ . Note that since all nodes have in-degree at most  $\alpha$ , this polynomial's degree is at most  $2\alpha$ , and hence there exists a coefficient vector  $c_i \in \mathbb{R}^{2\alpha}$  encoding this polynomial.
Let  $C \in \mathbb{R}^{n \times 2\alpha}$  be the matrix resulting from stacking such coefficient vectors for each of the  $n$  nodes. Consider  $\pmb{P} = \pmb{V}\pmb{C} \in \mathbb{R}^{n \times n}$ :  $\pmb{P}_{i,j}$  is 0 if  $\pmb{B}_{i,j} = 1$  and negative otherwise. Then  $(\pmb{P} \circ \pmb{P}^{\top})_{i,j}$  is 0 when either  $\pmb{B}_{i,j} = 1$  or  $(\pmb{B}^{\top})_{i,j} = 1$  and positive otherwise; equivalently, since  $\pmb{A} = \pmb{B} + \pmb{B}^{\top}$ ,  $(\pmb{P} \circ \pmb{P}^{\top})_{i,j} = 0$  iff  $\pmb{A}_{i,j} = 1$ . Take any positive  $\epsilon$  less than the smallest positive entry of  $\pmb{P} \circ \pmb{P}^{\top}$ . Letting  $\pmb{J}$  be an all-ones matrix, define  $M = \epsilon J - (P \circ P^{\top})$ . Note that  $M_{i,j} > 0$  if  $\pmb{A} = 1$  and  $M_{i,j} < 0$  if  $\pmb{A} = 0$ , that is,  $M = H(\pmb{A})$  as desired. Since  $\mathrm{rank}(J) = 1$  and  $\mathrm{rank}(P) \leq 2\alpha$ , by the bound on the rank of entrywise products of matrices, the rank of  $\pmb{M}$  is at most  $(2\alpha)^2 + 1$ .
Exact representation with community factorization LPCA factors  $X, Y \in \mathbb{R}^{n \times k}$  can be processed into nonnegative factors  $B \in \mathbb{R}_{+}^{n \times k_{B}}$  and  $C \in \mathbb{R}_{+}^{n \times k_{C}}$  such that  $k_{B} + k_{C} = 6k$  and
$$
B B ^ {\top} - C C ^ {\top} = \frac {1}{2} \left(\boldsymbol {X} \boldsymbol {Y} ^ {\top} + \boldsymbol {Y} \boldsymbol {X} ^ {\top}\right). \tag {5}
$$
As a rough outline of the argument that follows, we will need  $6k$  columns in the new factors  $\pmb{B},\pmb{C}$ , up from the  $k$  columns in  $\pmb{X},\pmb{Y}$ , because accounting for the possible asymmetry in  $\pmb{X}\pmb{Y}^{\top}$  will double the required columns, and accounting for the nonnegativity of  $\pmb{B},\pmb{C}$  will triple the required columns. Observe that the left-hand side can only represent symmetric matrices, but  $\pmb{X}\pmb{Y}^{\top}$  is not necessarily symmetric even if  $H(\pmb{X}\pmb{Y}^{\top}) = \pmb{A}$  for a symmetric  $\pmb{A}$ . For this reason, we use a symmetrization: let  $\pmb{L} = \frac{1}{2}\left(\pmb{X}\pmb{Y}^{\top} + \pmb{Y}\pmb{X}^{\top}\right)$ . Note that  $H(\pmb{L}) = H(\pmb{X}\pmb{Y}^{\top})$ , so if  $\pmb{X}\pmb{Y}^{\top}$  constitutes an exact representation of  $\pmb{A}$  in that  $H(\pmb{X}\pmb{Y}^{\top}) = \pmb{A}$ , so too do both expressions for  $\pmb{L}$  in Equation 5. Pseudocode for the procedure of constructing  $\pmb{B},\pmb{C}$  given  $\pmb{X},\pmb{Y}$  is given in Algorithm 1. The concept of this algorithm is to first separate the logit matrix  $\pmb{L}$  into a sum and difference of rank-1 components via eigendecomposition. Each of these components can be written as  $+\mathbf{v}\mathbf{v}^{\top}$  or  $-\mathbf{v}\mathbf{v}^{\top}$  with  $\mathbf{v} \in \mathbb{R}^{n}$ , where the sign depends on the sign of the eigenvalue. Each component is then separated into a sum and difference of three outer products of nonnegative vectors, via the following Lemma 5.3.
Lemma 5.3. Let  $\phi :\mathbb{R}\to \mathbb{R}$  denote the ReLU function, i.e.,  $\phi (z) = \max \{z,0\}$ . For any vector  $\mathbf{v}$ ,
$$
\mathbf {v} \mathbf {v} ^ {\top} = 2 \phi (\mathbf {v}) \phi (\mathbf {v}) ^ {\top} + 2 \phi (- \mathbf {v}) \phi (- \mathbf {v}) ^ {\top} - | \mathbf {v} | | \mathbf {v} | ^ {\top}.
$$
Proof. Take any  $\mathbf{v} \in \mathbb{R}^k$ . Then
$$
\begin{array}{l} \mathbf {v} \mathbf {v} ^ {\top} = (\phi (\mathbf {v}) - \phi (- \mathbf {v})) \cdot (\phi (\mathbf {v}) ^ {\top} - \phi (- \mathbf {v}) ^ {\top}) \\ = + \phi (\mathbf {v}) \phi (\mathbf {v}) ^ {\top} + \phi (- \mathbf {v}) \phi (- \mathbf {v}) ^ {\top} - \phi (\mathbf {v}) \phi (- \mathbf {v}) ^ {\top} - \phi (- \mathbf {v}) \phi (\mathbf {v}) ^ {\top} \\ = + 2 \phi (\mathbf {v}) \phi (\mathbf {v}) ^ {\top} + 2 \phi (- \mathbf {v}) \phi (- \mathbf {v}) ^ {\top} - (\phi (\mathbf {v}) + \phi (- \mathbf {v})) \cdot (\phi (\mathbf {v}) + \phi (- \mathbf {v})) ^ {\top} \\ = + 2 \phi (\mathbf {v}) \phi (\mathbf {v}) ^ {\top} + 2 \phi (- \mathbf {v}) \phi (- \mathbf {v}) ^ {\top} - | \mathbf {v} | | \mathbf {v} | ^ {\top}, \\ \end{array}
$$
where the first step follows from  $\mathbf{v} = \phi (\mathbf{v}) - \phi (-\mathbf{v})$  , and the last step from  $|\mathbf{v}| = \phi (\mathbf{v}) + \phi (-\mathbf{v})$
Algorithm 1 follows from Lemma 5.3 and constitutes a constructive proof of the following theorem:
Theorem 5.4 (Exact Community Factorization from Exact LPCA Factorization). Given a symmetric matrix  $\mathbf{A} \in \{0,1\}$  and  $\mathbf{X}, \mathbf{Y} \in \mathbb{R}^{n \times k}$  such that  $\mathbf{A} = H(\mathbf{X}\mathbf{Y}^{\top})$ , there exist nonnegative matrices  $\mathbf{B} \in \mathbb{R}_{+}^{n \times k_{B}}$  and  $\mathbf{C} \in \mathbb{R}_{+}^{n \times k_{C}}$  such that  $k_{B} + k_{C} = 6k$  and  $\mathbf{A} = H(\mathbf{BB}^{\top} - \mathbf{CC}^{\top})$ .
Algorithm 1 Converting LPCFA Factors to Community Factors
Input logit factors  $X, Y \in \mathbb{R}^{n \times k}$
Output  $B \in \mathbb{R}_{+}^{n \times k_{B}}$ ,  $C \in \mathbb{R}_{+}^{n \times k_{C}}$  such that  $k_{B} + k_{C} = 6k$
$$
\text {a n d} \boldsymbol {B B} ^ {\top} - \boldsymbol {C C} ^ {\top} = \frac {1}{2} \left(\boldsymbol {X Y} ^ {\top} + \boldsymbol {Y X} ^ {\top}\right)
$$
1: Set  $\pmb{Q} \in \mathbb{R}^{n \times 2k}$  and  $\lambda \in \mathbb{R}^{2k}$  by truncated eigendecomposition such that
$$
\boldsymbol {Q} \times \operatorname {d i a g} (\boldsymbol {\lambda}) \times \boldsymbol {Q} ^ {\top} = \frac {1}{2} \left(\boldsymbol {X} \boldsymbol {Y} ^ {\top} + \boldsymbol {Y} \bar {\boldsymbol {X}} ^ {\top}\right)
$$
2:  $B^{*}\gets Q^{+}\times \mathrm{diag}(\sqrt{+\lambda^{+}})$  , where  $\lambda^+,Q^+$  are the positive eigenvalues/vectors
3:  $C^* \gets Q^- \times \mathrm{diag}(\sqrt{-\lambda^-})$ , where  $\lambda^{-}, Q^{-}$  are the negative eigenvalues/vectors
4:  $\pmb{B} \gets (\sqrt{2}\phi(\pmb{B}^{*}); \sqrt{2}\phi(-\pmb{B}^{*}); |\pmb{C}^{*}|)$ $\triangleright$ $\phi$  and  $|\cdot|$  are entrywise ReLU and absolute value
5:  $\pmb{C} \gets (\sqrt{2}\phi(\pmb{C}^{*}); \sqrt{2}\phi(-\pmb{C}^{*}); |\pmb{B}^{*}|)$
6: return  $B, C$
As stated in the introduction to this section, Theorem 5.4 extends any upper bound on the exact factorization dimensionality from the LPCA model to our community-based model. That is, up to a constant factor, the bound in terms of max degree from Theorem 5.1 and the bound in terms of arboricity from Theorem 5.2 also apply to our model; for brevity, we state just the latter here.
Corollary 5.5 (Exact Community Factorization for Bounded-Arboricity Graphs). Let  $\mathbf{A} \in \{0,1\}^{n \times n}$  be the adjacency matrix of an undirected graph  $G$  with arboricity  $\alpha$ . Then there exist nonnegative embeddings  $\mathbf{B} \in \mathbb{R}_+^{n \times k_B}$  and  $\mathbf{C} \in \mathbb{R}_+^{n \times k_C}$  such that  $k_B + k_C = 6(4\alpha^2 + 1)$  and  $\mathbf{A} = H(\mathbf{BB}^\top - \mathbf{CC}^\top)$ .
Note that Corollary 5.5 is purely a statement about the capacity of our model; Theorem 5.2 stems from a constructive proof based on polynomial interpolation, and therefore so too does this corollary. We do not expect this factorization to be informative about the graph's latent structure. In the following Section 6, we will fit the model with an entirely different algorithm for downstream applications.
We now present a training algorithm to fit our model, then evaluate our method on a benchmark of five real-world networks.
6.1 Training Algorithm
Given an input graph  $\mathbf{A} \in \{0,1\}^{n \times n}$ , we find low-rank nonnegative matrices  $\mathbf{B}$  and  $\mathbf{C}$  such that the model produces  $\tilde{\mathbf{A}} = \sigma(\mathbf{BB}^{\top} - \mathbf{CC}^{\top}) \in (0,1)^{n \times n}$  as in Equation 1 which approximately matches  $\mathbf{A}$ . In particular, we train the model to minimize the sum of binary cross-entropies of the link predictions over all pairs of nodes:
$$
R = - \sum_ {i j} \left(\boldsymbol {A} _ {i j} \log \left(\tilde {\boldsymbol {A}} _ {i j}\right) + (1 - \boldsymbol {A} _ {i j}) \log \left(1 - \tilde {\boldsymbol {A}} _ {i j}\right)\right). \tag {6}
$$
We fit the parameters by gradient descent over this loss, as well as  $L_{2}$  regularization of the factors  $B$  and  $C$ , subject to the nonnegativity of  $B$  and  $C$ . This algorithm is fairly straightforward; pseudocode is given in Algorithm 2. This is quite similar to the training algorithm of Chanpuriya et al. (2020), but in contrast to that work, which only targets an exact fit, we explore the expression of graph structure in the factors and their utility in downstream tasks. Regularization of the factors is implemented to this end to avoid overfitting. Though in the main paper we outline and evaluate a non-stochastic version of the training algorithm, it can generalize straightforwardly to a more scalable stochastic version, e.g., by sampling links and non-links for the loss function and using projected SGD. In Appendix A.2, we discuss an industry application of our model to tabular dataset completion, for which we employ such stochastic training. Here, we use the simpler non-stochastic training to isolate the impact of model capacity, which is the focus of this work, as opposed to optimization.
Algorithm 2 Fitting the Constrained Model
Input adjacency matrix  $A\in \{0,1\}^{n\times n}$  , regularization weight  $\lambda \geq 0$  # of iterations I, # of homo/heterophilous communities  $k_{B} / k_{C}$    
Output fitted factors  $B\in \mathbb{R}_+^{n\times k_B},C\in \mathbb{R}_+^{n\times k_C}$  such that  $\sigma (BB^{\top} - CC^{\top})\approx A$    
1: Initialize  $B,C$  by setting entries to independent samples of Unif(0,  $1 / \sqrt{k_B}$  ),Unif(0,  $1 / \sqrt{k_C}$    
2: for  $i\gets 1$  to  $I$  do   
3:  $\tilde{\pmb{A}}\gets \sigma (\pmb {BB}^{\top} - \pmb {CC}^{\top})$    
4:  $R\gets -\sum_{ij}\left(\pmb {A}_{ij}\log (\tilde{\pmb{A}}_{ij}) + (1 - \pmb {A}_{ij})\log (1 - \tilde{\pmb{A}}_{ij})\right)$    
5:  $R\gets R + \lambda \left(\| B\| _F^2 +\| C\| _F^2\right)$    
6: Calculate  $\partial_{B,C}R$  , the gradient of  $R$  w.r.t.  $B,C$  , via differentiation through Steps 2 to 4   
7: Update  $B,C$  to minimize  $R$  using  $\partial_{B,C}R$  , subject to  $B,C\geq 0$    
8: end for   
9: return  $B,C$
Implementation details Our implementation uses PyTorch (Paszke et al., 2019) for automatic differentiation and minimizes loss using the SciPy (Jones et al., 2001) implementation of the L-BFGS (Liu & Nocedal, 1989; Zhu et al., 1997) algorithm with default hyperparameters and up to a max of 200 iterations of optimization. We set regularization weight  $\lambda = 10$  as in Yang & Leskovec (2013). We include code in the form of a Jupyter notebook (Pérez & Granger, 2007) demo.
6.2 Datasets
We use five fairly common mid-size datasets ranging from around 1K to 10K nodes. The selection of these five datasets is partly based on the presence of ground-truth multi-labels, which allows for evaluating the overlapping clustering methods. Statistics for these datasets, including the degeneracy
Table 1: Statistics of datasets used in our experiments. As in Sun et al. (2019), for YOUTUBE and AMAZON, we take only nodes that participate in at least one of the largest 5 ground-truth communities.
Name	Reference	Nodes	Edges	Labels	Max Degree	Degeneracy
BLOG	Tang & Liu (2009)	10,312	333,983	39	3,992	114
YOUTUBE	Yang & Leskovec (2015)	5,346	24,121	5	628	19
POS	Qiu et al. (2018)	4,777	92,406	40	3,644	49
PPI	Breitkreutz et al. (2007)	3,852	76,546	50	593	29
AMAZON	Yang & Leskovec (2015)	794	2,109	5	29	6
of each network, are given in Table 1. Note that degeneracy is an upper bound on arboricity. We note that the mid-sized networks used in our empirical work actually underemphasize the significance of our theoretical arboricity bound: see, e.g., the real-world networks in Pashanasangi & Seshadhri (2021), which have up to tens of millions of nodes but still have degeneracies at most in the hundreds.
BLOG is a social network of relationships between online bloggers; the node labels represent interests of the bloggers. Similarly, YOUTUBE is a social network of YouTube users, and the labels represent groups that the users joined.
POS is a word co-occurrence network: nodes represent words, and there are edges between words which are frequently adjacent in a section of the Wikipedia corpus. Each node label represents the part-of-speech of the word. PPI is a subgraph of the protein-protein interaction network for Homo Sapiens. Labels represent biological states. Finally, Amazon is a co-purchasing network: nodes represent products, and there are edges between products which are frequently purchased together. Labels represent categories of products.
While social networks like the former two in this list are generally dominated by homophily (McPherson et al., 2001), the latter three should exhibit significant heterophily. For co-purchasing networks like Amazon, depending on the product, two of the same kind of product are generally not copurchased, e.g., Pepsi and Coke, as discussed in Peysakhovich & Bottou (2021). Though less intuitively accessible, there is also prior discussion of disassortativity in word adjacencies (Foster et al., 2010; Zweig, 2016), as well as in PPI networks (Newman, 2002; Hase et al., 2010).
6.3 Results
Expressiveness First, we investigate the expressiveness of our generative model, that is, the fidelity with which it can reproduce an input network. In Section 1, we used a simple synthetic network to show that our model is more expressive than others due to its ability to represent heterophilous structures in addition to homophilous ones. We now evaluate the expressiveness of our model on real-world networks. As with the synthetic graph, we fix the number of communities or singular vectors, fit the model, then evaluate the reconstruction error. In Figure 4, we compare the results of our model with those of SVD, BIGCLAM (which is discussed in detail in Section 4), and SYMNMF (Kuang et al., 2015). SYMNMF simply factors the adjacency matrix as  $\mathbf{A} \approx \mathbf{H}\mathbf{H}^{\top}$ , where  $\mathbf{H} \in \mathbb{R}_{+}^{n \times k}$ ; note that, like SVD, SYMNMF does not necessarily output a matrix whose entries are probabilities (i.e., bounded in [0, 1]), and hence it is not a natural graph generative model like ours and BIGCLAM.
Figure 4: Reconstruction error on real-world networks, relative to our model's error.
For each method, we fix the number of communities or singular vectors at the ground-truth number. For this experiment only, we are not concerned with learning the latent structure of the graph; the only goal is accurate representation of the network with limited parameters. So, for a fair comparison with SVD, we do not regularize the training of the other methods. Our method consistently has the lowest reconstruction error, both in terms of Frobenius error and entrywise cross-entropy (Equation 6). We particularly highlight the improvement over BIGCLAM; the salient difference between these models, both of which are factorization-based, include a nonlinear link function, and assign node communities using nonnegative factors, is the presence of the heterophilous  $-CC^{\top}$  term in our model. Thus, the improvement directly reflects the value of incorporating heterophily into the model. Interestingly, we find the most significant improvement exactly on the three datasets which have been noted to exhibit significant heterophily: POS, PPI, and AMAZON.
Similarity to ground-truth communities To assess the interpretability of clusters generated by our method, we evaluate the similarity of these clusters to ground-truth communities (i.e., class labels), and we compare other methods for overlapping clustering. We additionally compare to another recent but non-generative approach, the vGRAPH method of Sun et al. (2019), which is based on link clustering; the authors found their method to generally achieve state-of-the-art results in this task. For all methods, we set the number of communities to be detected as the number of ground-truth communities. We report F1-Score as computed in Yang & Leskovec (2013). See Figure 5 (left): the performance of our method is competitive with SYMNMF, BIGCLAM, and vGraph.
Figure 5: Left: Similarity of recovered communities to ground-truth labels of real-world datasets. (Note: vGRAPH is omitted on Blog due to memory limitations.) Right: Accuracy of link prediction.
Interpretable link prediction We assess the predictive power of our generative model on the link prediction task. As discussed in Section 3, the link probabilities output by our model are interpretable in terms of a clustering of nodes that it generates; we compare results with our method to those with other models which permit similar interpretation, namely BIGCLAM and SYMNMF. We randomly select  $10\%$  of node pairs to hold out, fit the models on the remaining  $90\%$ , then use the trained models to predict links between node pairs in the held out  $10\%$ . As a baseline, we also show results for randomly predicting link or no link with equal probability. See Figure 5 (right). The performance of our method is competitive with or exceeds those of the other methods in terms of F1 Score.
We introduce a community-based graph generative model based on symmetric nonnegative matrix factorization which can represent both homophily and heterophily. We add to a prior guarantee of exact representation for bounded degree graphs with a broader guarantee for bounded arboricity graphs, and we show that both of these guarantees apply to our more interpretable graph model. We illustrate our model's capabilities with experiments on a synthetic motivating example. Experiments on real-world networks show its effectiveness on several key tasks. Broadly, our results suggest that incorporating heterophily into methods for networks can improve both theoretical grounding and empirical performance, while maintaining interpretability. Future directions include deeper understanding of the expressiveness of low-rank logit models and convergence of training algorithms.
This project was partially supported by an Adobe Research grant, along with NSF Grants 2046235 and 1763618.