SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sudhanshu Chanpuriya $^{1}$ , Ryan A. Rossi $^{2}$ , Anup Rao $^{2}$ , Tung Mai $^{2}$ , Nedim Lipka $^{2}$ , Zhao Song $^{2}$ , and Cameron Musco $^{3}$
<sup>1</sup>University of Illinois Urbana-Champaign, schariya@illinois.edu  
<sup>2</sup>Adobe Research, {ryrossi, anuprao, tumai, lipka, zsong}@adobe.com  
<sup>3</sup>University of Massachusetts Amherst, cmusco@cs.umass.edu
Graph models based on factorization of the adjacency matrix often fail to capture network structures related to links between dissimilar nodes (heterophily). We introduce a novel graph factorization model that leverages two nonnegative vectors per node to interpretably account for links between both similar and dissimilar nodes. We prove that our model can exactly represent any graph with low arboricity, a property that many real-world networks satisfy; our proof also applies to related models but has much greater scope than the closest prior bound, which is based on low max degree. Our factorization also has compelling properties besides expressiveness: due to its symmetric structure and nonnegativity, fitting the model inherently finds node communities, and the model's link predictions can be interpreted in terms of these communities. In experiments on real-world networks, we demonstrate our factorization's effectiveness on a variety of tasks, including community detection and link prediction.
Graphs data naturally arises in a variety of fields including sociology (Mason & Verwoerd, 2007), biology (Scott, 1988), and computer networking (Bonato, 2004). A key task in machine learning for graph data is forming models of graphs that can predict edges between nodes, form useful representations of nodes, and reveal interpretable structure in the graph, such as detecting clusters of nodes. Many graph models fall under the framework of edge-independent graph generative models, which output the probabilities of edges existing between any pair of nodes. The parameters of such models can be trained iteratively on the network, or some fraction of the network which is known, in the link prediction task, i.e., by minimizing a predictive loss. To choose among these models, one must consider two criteria: 1) whether the model can express structures of interest in the graph, 2) whether the model expresses these structures in an interpretable way.
Expressiveness of low-dimensional embeddings As real-world graphs are high-dimensional objects, graph models generally compress information about the graph. For example, dot product models associate each node with a real-valued "embedding" vector; the predicted probability of a link between two nodes increases with the similarity of their embeddings. These models can alternatively be seen as factorizing the graph's adjacency matrix to approximate it with a low-rank matrix. Recent work of Seshadhri et al. (2020) has shown that dot product models are limited in their ability to model common structures in real-world graphs, such as triangles incident only on low-degree nodes. In response, Chanpuriya et al. (2020) showed that with the logistic principal components analysis (LPCA) model, which has two embeddings per node (i.e., using the dot product of the 'left' embedding of one node and the 'right' embedding of another), not only can such structures be represented, but further, any graph can be exactly represented with embedding vectors whose lengths are linear in the
max degree of the graph. There are two keys to this result. First is the presence of a nonlinear linking function in LPCA; since adjacency matrices are generally not low-rank, exact low-rank factorization is impossible without a linking function. Second is that having two embeddings rather than one allows for expression of non-positive semidefinite (PSD) matrices. As discussed in Peysakhovich & Bottou (2021), that the single-embedding models can only represent PSD matrices precludes representation of 'heterophilous' structures in graphs; heterophilous structures are those wherein dissimilar nodes are linked, in contrast to more intuitive 'homophilous' linking between similar nodes.
Interpretability and node clustering Beyond being able to capture a given network accurately, it is often desirable for a graph model to form interpretable representations of nodes and to produce edge probabilities in an interpretable fashion. Dot product models can achieve this by restricting the node embeddings to be nonnegative. Nonnegative factorization has long been used to decompose data into parts (Donoho & Stodden, 2003). In the context of graphs, this entails decomposing the set of nodes of the network into clusters or communities. In particular, each entry of the nonnegative embedding vector of a node represents the intensity with which the node participates in a community. This allows the edge probabilities output by dot product models to be interpretable in terms of coparticipation in communities. Depending on the model, these vectors may have restrictions such as a sum-to-one requirement, meaning the node is assigned a categorical distribution over communities. The least restrictive and most expressive case is that of soft assignments to overlapping communities, where the entries can vary totally independently. In such models, which include the BIGCLAM model of Yang & Leskovec (2013), the output of the dot product may be mapped through a nonlinear link function (as in LPCA) to produce a probability for each edge, i.e., to ensure the values lie in  $[0,1]$ .
Summary of contributions The key contributions of this work are as follows:
- We prove that LPCA admits exact low-rank factorizations of graphs with bounded arboricity, which is the minimum number of forests into which a graph's edges can be partitioned. By the Nash-Williams theorem, arboricity is a measure of a graph's density in that, letting  $S$  denote an induced subgraph and  $n_S$  and  $m_S$  denote the number of nodes and edges in  $S$ , arboricity is the maximum over all  $S$  of  $\lceil \frac{m_S}{n_S - 1} \rceil$ . Our result is more applicable to real-world graphs than the prior result of Chanpuriya et al. (2020) for graphs with bounded max degree, since sparsity is a common feature of real networks, whereas low max degree is not.
- We introduce a graph model that extends LPCA and is both highly expressive and interpretable. Our model incorporates two embeddings per node and a nonlinear linking function, and hence is able to express both heterophily and overlapping communities. At the same time, our model is based on symmetric nonnegative matrix factorization, so it outputs link probabilities that are interpretable in terms of the communities it detects. While prior graph factorizations incorporate some aspects of nonnegativity, heterophily, and/or nonlinearity, our proposed model lies at the intersection of all three.
- We show how any graph with a low-rank factorization in the LPCA model also admits a low-rank factorization in our community-based model. This means that the guarantees on low-rank representation for bounded max degree and arboricity also apply to our model.
- In experiments, we show that our method is competitive with and often outperforms other comparable models on real-world graphs in terms of representing the network, doing interpretable link prediction, and detecting communities that align with ground-truth.
2 Motivating Example
To demonstrate how heterophily can manifest in networks, as well as how models which assume homophily can fail to represent such networks, we provide a simple synthetic example involving a graph of matches between users of a dating app. Suppose that users of this app are generally seeking partners of a different gender (to simplify this example, we assume that each user of the app is either a man or a woman), and suppose that each user comes from one of ten cities. Members from the same city are likely to match with each other; this typifies homophily, wherein links occur between similar nodes. Furthermore, users having the same gender are unlikely to match with each other; this typifies heterophily. Figure 1 shows an instantiation of such an adjacency matrix with 1000 nodes, which are randomly assigned to man or woman and to one of the ten cities. We recreate this network with our proposed embedding model and with BIGCLAM, which explicitly assumes
homophily. We also compare with the SVD of the adjacency matrix, which outputs the best (lowest Frobenius error) low-rank approximation that is possible without a nonlinear linking function. Since SVD lacks nonnegativity constraints on the factors, we do not expect interpretability. In Figure 2, we show how BIGCLAM captures only the ten communities based on city, i.e., only the homophilous structure, and fails to capture the heterophilous distinction between men and women. We also plot the error of the reconstructions as the embedding length increases. There are  $10 \cdot 2 = 20$  different kinds of nodes, meaning the expected adjacency matrix is rank-20, and our model maintains the lowest error up to this embedding length; by contrast, BIGCLAM is unable to decrease error after capturing city information with length-10 embeddings. In Figure 3, we visualize the features generated by the three methods, i.e., the factors returned by each factorization. Our model's factors capture the relevant latent structure in an interpretable way. By contrast, SVD's factors are harder to interpret, and BIGCLAM does not represent the heterophilous structure.
Figure 1: The motivating synthetic graph. The expected adjacency matrix (left) and the sampled matrix (right); the latter, which is passed to the training algorithms, is produced by treating the entries of the former as parameters of Bernoulli distributions and sampling. The network is approximately a union of ten bipartite graphs, each of which correspond to men and women in one of the ten cities.
Figure 2: Left: Reconstructions of the motivating synthetic graph of Figure 1 with SVD, BIGCLAM, and our model, using 12 communities or singular vectors. Note the lack of the small diagonal structure in BIGCLAM's reconstruction; this corresponds to its inability to capture the heterophilous interaction between men and women. Right: Frobenius error when reconstructing the motivating synthetic graph of Figure 1 with SVD, BIGCLAM, and our model, as the embedding length is varied. The error is normalized by the sum of the true adjacency matrix (i.e., twice the number of edges).
Figure 3: Factors resulting from decomposition of the motivating synthetic graph of Figure 1 with the three models, using 12 communities or singular vectors. The top/bottom rows represent the positive/negative eigenvalues corresponding to homophilous/heterophilous communities (note that BIGCLAM does not include the latter). The homophilous factors from BIGCLAM and our model reflect the 10 cities, and the heterophilous factor from our model reflects men and women. The factors from SVD are harder to interpret. Note that the order of the communities in the factors is arbitrary.
3 Community-Based Graph Factorization
Consider the set of undirected, unweighted graphs on  $n$  nodes, i.e., the set of graphs with symmetric adjacency matrices in  $\{0,1\}^{n\times n}$ . We propose an edge-independent generative model for such graphs. Given nonnegative parameter matrices  $\pmb{B}\in \mathbb{R}_{+}^{n\times k_{B}}$  and  $\pmb {C}\in \mathbb{R}_+^{n\times k_C}$ , we set the probability of an edge existing between nodes  $i$  and  $j$  to be the  $(i,j)$ -th entry of matrix  $\tilde{\pmb{A}}$ :
$$
\tilde {\boldsymbol {A}} := \sigma (\boldsymbol {B} \boldsymbol {B} ^ {\top} - \boldsymbol {C} \boldsymbol {C} ^ {\top}), \tag {1}
$$
where  $\sigma$  is the logistic function. Here  $k_{B}$ ,  $k_{C}$  are the number of homophilous/heterophilous clusters. Intuitively, if  $\pmb{b}_{i} \in \mathbb{R}_{+}^{k_{B}}$  is the  $i$ -th row of matrix  $\pmb{B}$ , then  $\pmb{b}_{i}$  is the affinity of node  $i$  to each of the  $k_{B}$  homophilous communities. Similarly,  $\pmb{c}_{i} \in \mathbb{R}_{+}^{k_{C}}$  is the affinity of node  $i$  to the  $k_{C}$  heterophilous communities. As an equivalent statement, for each pair of nodes  $i$  and  $j$ ,  $\tilde{\pmb{A}}_{i,j} \coloneqq \sigma (\pmb{b}_{i}\pmb{b}_{j} - \pmb{c}_{i}\pmb{c}_{j}^{\top})$ . We will soon discuss the precise interpretation of this model, but the idea is roughly similar to the attract-repel framework of Peysakhovich & Bottou (2021). When nodes  $i$  and  $j$  have similar 'attractive'  $\pmb{b}$  embeddings, i.e., when  $\pmb{b}_{i}\pmb{b}_{j}^{\top}$  is high, the likelihood of an edge between them increases, hence why the  $\pmb{B}$  factor is homophilous. By contrast, the  $\pmb{C}$  factor is 'repulsive'/heterophilous since, when  $\pmb{c}_{i}\pmb{c}_{j}^{\top}$  is high, the likelihood of an edge between  $i$  and  $j$  decreases.
Alternate expression We note that the model above can also be expressed in a form that normalizes cluster assignments and is more compact, in that it combines the homophilous and heterophilous cluster assignments. Instead of  $\pmb{B}$  and  $\pmb{C}$ , this form uses a matrix  $V \in [0,1]^{n \times k}$  and a diagonal matrix  $W \in \mathbb{R}^{k \times k}$ , where  $k = k_{B} + k_{C}$  is the total number of clusters. In particular, let  $m_{B}$  and  $m_{C}$  be the vectors containing the maximums of each column of  $\pmb{B}$  and  $\pmb{C}$ . By setting
$$
\boldsymbol {V} = \left(\boldsymbol {B} \cdot \operatorname {d i a g} \left(\boldsymbol {m} _ {B} ^ {- 1}\right); \quad \boldsymbol {C} \cdot \operatorname {d i a g} \left(\boldsymbol {m} _ {C} ^ {- 1}\right)\right)
$$
$$
\boldsymbol {W} = \operatorname {d i a g} \left(\left(+ \boldsymbol {m} _ {B} ^ {2}; - \boldsymbol {m} _ {C} ^ {2}\right)\right), \tag {2}
$$
the constraint on  $V$  is satisfied. Further,  $\mathbf{V}\mathbf{W}\mathbf{V}^{\top} = \mathbf{B}\mathbf{B}^{\top} - \mathbf{C}\mathbf{C}^{\top}$ , so
$$
\tilde {\boldsymbol {A}} := \sigma \left(\boldsymbol {B} \boldsymbol {B} ^ {\top} - \boldsymbol {C} \boldsymbol {C} ^ {\top}\right) = \sigma \left(\boldsymbol {V} \boldsymbol {W} \boldsymbol {V} ^ {\top}\right). \tag {3}
$$
Here, if  $\mathbf{v}_i\in [0,1]^k$  is the  $i$ -th row of matrix  $\pmb{V}$ , then  $\mathbf{v}_i$  is the soft (normalized) assignment of node  $i$  to the  $k$  communities. The diagonal entries of  $\pmb{W}$  represent the strength of the homophily (if positive) or heterophily (if negative) of the communities. For each entry,  $\tilde{\pmb{A}}_{i,j} = \sigma (\mathbf{v}_i\pmb {W}\mathbf{v}_j^\top)$ . We use these two forms interchangeably throughout this work.
Interpretation The edge probabilities output by this model have an intuitive interpretation. Recall that there are bijections between probability  $p \in [0,1]$ , odds  $o = \frac{p}{1 - p} \in [0,\infty)$ , and logit  $\ell = \log(o) \in (-\infty, +\infty)$ . The logit of the link probability between nodes  $i$  and  $j$  is  $\mathbf{v}_i^\top \mathbf{W} \mathbf{v}_j$ , which is a summation of terms  $\mathbf{v}_{ic} \mathbf{v}_{jc} \mathbf{W}_{cc}$  over all communities  $c \in [k]$ . If the nodes both fully participate in community  $c$ , that is,  $\mathbf{v}_{ic} = \mathbf{v}_{jc} = 1$ , then the edge logit is changed by  $\mathbf{W}_{cc}$  starting from a baseline of 0, or equivalently, the odds of an edge is multiplied by  $\exp(\mathbf{W}_{cc})$  starting from a baseline odds of 1; if either of the nodes participates only partially in community  $c$ , then the change in logit and odds is accordingly prorated. Homophily and heterophily also have a clear interpretation in this model: homophilous communities, which are expressed in  $B$ , are those with  $W_{cc} > 0$ , where two nodes both participating in the community increases the odds of a link, whereas communities with  $W_{cc} < 0$ , which are expressed in  $C$ , are heterophilous, and coparticipation decreases the odds of a link.
Community detection via interpretable factorizations There is extensive prior work on community detection and node clustering (Schaeffer, 2007; Aggarwal & Wang, 2010; Nascimento & De Carvalho, 2011), perhaps the most well-known being the normalized cuts algorithm of Shi & Malik (2000), which produces a clustering based on the entrywise signs of an eigenvector of the graph Laplacian matrix. However, the clustering algorithms which are most relevant to our work are those based on non-negative matrix factorization (NMF) (Lee & Seung, 1999; Berry et al., 2007; Wang & Zhang, 2012; Gillis, 2020), many of which can be seen as integrating nonnegativity constraints into
the broader, well-studied random dot product graph (RDPG) model (Young & Scheinerman, 2007; Scheinerman & Tucker, 2010; Athreya et al., 2017). One such algorithm is that of Yu et al. (2005), which approximately factors a graph's adjacency matrix  $\mathbf{A} \in \{0,1\}^{n \times n}$  into two positive matrices  $\mathbf{H}$  and  $\boldsymbol{\Lambda}$ , where  $\mathbf{H} \in \mathbb{R}_{+}^{n \times k}$  is left-stochastic and  $\boldsymbol{\Lambda} \in \mathbb{R}_{+}^{k \times k}$  is diagonal, such that  $\mathbf{H} \boldsymbol{\Lambda} \mathbf{H}^{\top} \approx \mathbf{A}$ . Here  $\mathbf{H}$  represents a soft clustering of the  $n$  nodes into  $k$  clusters, while the diagonal entries of  $\boldsymbol{\Lambda}$  represent the prevalence of edges within clusters. Note the similarity of the factorization to our model, save for the lack of a nonlinearity. Other NMF approaches include those of Ding et al. (2008), Yang et al. (2012), Kuang et al. (2012), and Kuang et al. (2015) (SYMNMF).
Modeling heterophily Much of the existing work on graph models has an underlying assumption of network homophily (Johnson et al., 2010; Noldus & Van Mieghem, 2015). There has been significant recent interest in the limitations of graph neural network (GNN) models (Duvenaud et al., 2015; Kipf & Welling, 2017; Hamilton et al., 2017) at addressing network heterophily (NT & Maehara, 2019; Zhu et al., 2020; Zheng et al., 2022), as well as proposed solutions (Pei et al., 2020; Yan et al., 2021), but relatively less work for models applicable to clustering. Some existing NMF approaches to clustering do naturally model heterophilous structure in networks. The model of Nourbakhsh et al. (2014), for example, is similar to that of Yu et al. (2005), but allows the cluster affinity matrix  $\Lambda$  to be non-diagonal; this allows for inter-cluster edge affinity to exceed intra-cluster edge affinity, so heterophily can arise in this model, though it is not a focus of their work. Another example is the model in Miller et al. (2009), which is similar to ours, though it restricts the cluster assignment matrix  $V$  to be binary; additionally, their training algorithm is not based on gradient descent as ours is, and it does not scale to larger networks. More recently, Rubin-Delanchy et al. (2017) and Peysakhovich & Bottou (2021) propose simple decompositions which allow for representation of non-PSD adjacency matrices. The model in the latter work is a factorization of the form  $A \approx D + BB^{\top} - CC^{\top}$ , where  $D \in \mathbb{R}^{n \times n}$  is diagonal and  $B, C \in \mathbb{R}^{n \times k}$  are low-rank; excluding the diagonal  $D$  term, the model in the former work is algebraically identical. The authors discuss how, interestingly, this model separates the homophilous and heterophilous structure into different factors, namely  $B$  and  $C$ , corresponding to positive and negative eigenvalues, respectively. Note that these decompositions do not include a nonlinear linking function, which is crucial to our exact factorization results, and the respective works do not investigate constraining the factors to be nonnegative.
Overlapping communities and exact embeddings Many models discussed above focus on the single-label clustering task and thus involve highly-constrained factorizations (e.g., sum-to-one conditions). We are interested in the closely related but distinct task of multi-label clustering, also known as overlapping community detection (Xie et al., 2013; Javed et al., 2018), which involves less constrained, more expressive factorizations. The BIGCLAM algorithm of Yang & Leskovec (2013) uses the following generative model for this task: the probability of a link between two nodes  $i$  and  $j$  is given by  $1 - \exp(-\pmb{f}_i \cdot \pmb{f}_j)$ , where  $\pmb{f}_i, \pmb{f}_j \in \mathbb{R}_+^k$  represent the intensities with which the nodes participate in each of the  $k$  communities. Note that BIGCLAM assumes strict homophily of the communities: two nodes participating in the same community always increase the probability of a link. However, this model allows for expression of very dense intersections of communities, which the authors observe is generally a characteristic of real-world networks. To ensure that output entries are probabilities, BIGCLAM's factorization includes a nonlinear linking function (namely,  $f(x) = 1 - e^x$ ), like our model and LPCA. Recent work outside clustering and community detection on graph generative models (Rendsburg et al., 2020; Chanpuriya et al., 2020) suggests that incorporating a nonlinear linking function can greatly increase the expressiveness of factorization-based graph models, to the point of being able to exactly represent a graph. This adds to a growing body of literature on expressiveness guarantees for embeddings on relational data (Sala et al., 2018; Bhattacharjee & Dasgupta, 2020; Boratko et al., 2021). Most relevant to our work, as previously discussed, Chanpuriya et al. (2020) provide a guarantee for exact low-rank representation of graphs with bounded max degree when using the LPCA factorization model. In this work, we provide a new such guarantee, except for bounded arboricity, which is more applicable to real-world networks, and extend these guarantees to our community-based factorization.
We first restate the main result from Chanpuriya et al. (2020) on exact representation of graphs with bounded max degree using the logistic principal components analysis (LPCA) model, which
reconstructs a graph  $\mathbf{A} \in \{0,1\}^{n \times n}$  using logit factors  $\mathbf{X}, \mathbf{Y} \in \mathbb{R}^{n \times k}$  via
$$
\boldsymbol {A} \approx \sigma \left(\boldsymbol {X} \boldsymbol {Y} ^ {\top}\right). \tag {4}
$$
Note that unlike our community-based factorization, the factors of the LPCA model are not nonnegative, and the factorization does not reflect the symmetry of the undirected graph's adjacency matrix. Regardless of the model's interpretability, the following theorem provides a significant guarantee on its expressiveness. We use the following notation: given a matrix  $M$ , let  $H(M)$  denote the matrix resulting from entrywise application of the Heaviside step function to  $M$ , that is, setting all positive entries to 1, negative entries to 0, and zero entries to  $1/2$ .
Theorem 5.1 (Exact LPCA Factorization for Bounded-Degree Graphs). Let  $\mathbf{A} \in \{0,1\}^{n \times n}$  be the adjacency matrix of a graph  $G$  with maximum degree  $c$ . Then there exist matrices  $\mathbf{X}, \mathbf{Y} \in \mathbb{R}^{n \times (2c + 1)}$  such that  $\mathbf{A} = H(\mathbf{X}\mathbf{Y}^{\top})$ .
This corresponds to arbitrarily small approximation error in the LPCA model (Equation 4) because, provided such factors  $\mathbf{X}, \mathbf{Y}$  for some graph  $\mathbf{A}$ , we have that  $\lim_{s\to \infty}\sigma \left(s\mathbf{X}\mathbf{Y}^{\top}\right) = H(\mathbf{X}\mathbf{Y}^{\top}) = \mathbf{A}$ . That is, we can scale the factors larger to reduce the error to an arbitrary extent.