Title: Exact Representation of Sparse Networks with Symmetric Nonnegative Embeddings
Venue: NeurIPS 2023
Accepted: yes
Keywords: graph

network

embeddings

arboricity

factorization

model

community

nonnegative
Abstract: Graph models based on factorization of the adjacency matrix often fail to capture network structures related to links between dissimilar nodes (heterophily). We introduce a novel graph factorization model that leverages two nonnegative vectors per node to interpretably account for links between both similar and dissimilar nodes. We prove that our model can exactly represent any graph with low *arboricity*, a property that many real-world networks satisfy; our proof also applies to related models but has much greater scope than the closest prior bound, which is based on low *max degree*. Our factorization also has compelling properties besides expressiveness: due to its symmetric structure and nonnegativity, fitting the model inherently finds node communities, and the model's link predictions can be interpreted in terms of these communities. In experiments on real-world networks, we demonstrate our factorization's effectiveness on a variety of tasks, including community detection and link prediction.

Exact Representation of Sparse Networks with Symmetric Nonnegative Embeddings

Abstract

Graph models based on factorization of the adjacency matrix often fail to capture network structures related to links between dissimilar nodes (heterophily). We introduce a novel graph factorization model that leverages two nonnegative vectors per node to interpretably account for links between both similar and dissimilar nodes. We prove that our model can exactly represent any graph with low arboricity , a property that many real-world networks satisfy; our proof also applies to related models but has much greater scope than the closest prior bound, which is based on low max degree . Our factorization also has compelling properties besides expressiveness: due to its symmetric structure and nonnegativity, fitting the model inherently finds node communities, and the model's link predictions can be interpreted in terms of these communities. In experiments on real-world networks, we demonstrate our factorization's effectiveness on a variety of tasks, including community detection and link prediction.

1 Introduction

Graphs data naturally arises in a variety of fields including sociology (Mason & Verwoerd, 2007), biology (Scott, 1988), and computer networking (Bonato, 2004). A key task in machine learning for graph data is forming models of graphs that can predict edges between nodes, form useful representations of nodes, and reveal interpretable structure in the graph, such as detecting clusters of nodes. Many graph models fall under the framework of edge-independent graph generative models, which output the probabilities of edges existing between any pair of nodes. The parameters of such models can be trained iteratively on the network, or some fraction of the network which is known, in the link prediction task, i.e., by minimizing a predictive loss. To choose among these models, one must consider two criteria: 1) whether the model can express structures of interest in the graph, 2) whether the model expresses these structures in an interpretable way.

Expressiveness of low-dimensional embeddings As real-world graphs are high-dimensional objects, graph models generally compress information about the graph. For example, dot product models associate each node with a real-valued 'embedding' vector; the predicted probability of a link between two nodes increases with the similarity of their embeddings. These models can alternatively be seen as factorizing the graph's adjacency matrix to approximate it with a low-rank matrix. Recent work of Seshadhri et al. (2020) has shown that dot product models are limited in their ability to model common structures in real-world graphs, such as triangles incident only on low-degree nodes. In response, Chanpuriya et al. (2020) showed that with the logistic principal components analysis (LPCA) model, which has two embeddings per node (i.e., using the dot product of the 'left' embedding of one node and the 'right' embedding of another), not only can such structures be represented, but further, any graph can be exactly represented with embedding vectors whose lengths are linear in the

max degree of the graph. There are two keys to this result. First is the presence of a nonlinear linking function in LPCA; since adjacency matrices are generally not low-rank, exact low-rank factorization is impossible without a linking function. Second is that having two embeddings rather than one allows for expression of non-positive semidefinite (PSD) matrices. As discussed in Peysakhovich & Bottou (2021), that the single-embedding models can only represent PSD matrices precludes representation of 'heterophilous' structures in graphs; heterophilous structures are those wherein dissimilar nodes are linked, in contrast to more intuitive 'homophilous' linking between similar nodes.

Interpretability and node clustering Beyond being able to capture a given network accurately, it is often desirable for a graph model to form interpretable representations of nodes and to produce edge probabilities in an interpretable fashion. Dot product models can achieve this by restricting the node embeddings to be nonnegative. Nonnegative factorization has long been used to decompose data into parts (Donoho & Stodden, 2003). In the context of graphs, this entails decomposing the set of nodes of the network into clusters or communities. In particular, each entry of the nonnegative embedding vector of a node represents the intensity with which the node participates in a community. This allows the edge probabilities output by dot product models to be interpretable in terms of coparticipation in communities. Depending on the model, these vectors may have restrictions such as a sum-to-one requirement, meaning the node is assigned a categorical distribution over communities. The least restrictive and most expressive case is that of soft assignments to overlapping communities, where the entries can vary totally independently. In such models, which include the BIGCLAM model of Yang &Leskovec (2013), the output of the dot product may be mapped through a nonlinear link function (as in LPCA) to produce a probability for each edge, i.e., to ensure the values lie in [0 , 1] .

Summary of contributions The key contributions of this work are as follows:

We prove that LPCA admits exact low-rank factorizations of graphs with bounded arboricity , which is the minimum number of forests into which a graph's edges can be partitioned. By the Nash-Williams theorem, arboricity is a measure of a graph's density in that, letting S denote an induced subgraph and n S and m S denote the number of nodes and edges in S , arboricity is the maximum over all S of ⌈ m S n S -1 ⌉ . Our result is more applicable to real-world graphs than the prior result of Chanpuriya et al. (2020) for graphs with bounded max degree, since sparsity is a common feature of real networks, whereas low max degree is not.

We introduce a graph model that extends LPCA and is both highly expressive and interpretable. Our model incorporates two embeddings per node and a nonlinear linking function, and hence is able to express both heterophily and overlapping communities. At the same time, our model is based on symmetric nonnegative matrix factorization, so it outputs link probabilities that are interpretable in terms of the communities it detects. While prior graph factorizations incorporate some aspects of nonnegativity, heterophily, and/or nonlinearity, our proposed model lies at the intersection of all three.

We show how any graph with a low-rank factorization in the LPCA model also admits a low-rank factorization in our community-based model. This means that the guarantees on low-rank representation for bounded max degree and arboricity also apply to our model.

In experiments, we show that our method is competitive with and often outperforms other comparable models on real-world graphs in terms of representing the network, doing interpretable link prediction, and detecting communities that align with ground-truth.

2 Motivating Example

To demonstrate how heterophily can manifest in networks, as well as how models which assume homophily can fail to represent such networks, we provide a simple synthetic example involving a graph of matches between users of a dating app. Suppose that users of this app are generally seeking partners of a different gender (to simplify this example, we assume that each user of the app is either a man or a woman), and suppose that each user comes from one of ten cities. Members from the same city are likely to match with each other; this typifies homophily, wherein links occur between similar nodes. Furthermore, users having the same gender are unlikely to match with each other; this typifies heterophily. Figure 1 shows an instantiation of such an adjacency matrix with 1000 nodes, which are randomly assigned to man or woman and to one of the ten cities. We recreate this network with our proposed embedding model and with BIGCLAM, which explicitly assumes

homophily. We also compare with the SVD of the adjacency matrix, which outputs the best (lowest Frobenius error) low-rank approximation that is possible without a nonlinear linking function. Since SVD lacks nonnegativity constraints on the factors, we do not expect intepretability. In Figure 2, we show how BIGCLAM captures only the ten communities based on city, i.e., only the homophilous structure, and fails to capture the heterophilous distinction between men and women. We also plot the error of the reconstructions as the embedding length increases. There are 10 · 2 = 20 different kinds of nodes, meaning the expected adjacency matrix is rank20 , and our model maintains the lowest error up to this embedding length; by contrast, BIGCLAM is unable to decrease error after capturing city information with length10 embeddings. In Figure 3, we visualize the features generated by the three methods, i.e., the factors returned by each factorization. Our model's factors capture the relevant latent structure in an interpretable way. By contrast, SVD's factors are harder to interpret, and BIGCLAM does not represent the heterophilous structure.

0

200

400

600

800

0

200

400

600

800

Expected Adjacency

0

200

400

600

800

0

200

400

600

800

Sampled Adjacency

0.0

0.2

0.4

0.6

0.8

1.0

0

200 400 600 800

0

200

400

600

800

SVD

0

200 400 600 800

0

200

400

600

800

BigClam

0

200 400 600 800

0

200

400

600

800

Ours

5

10

15

20

Embedding Length

0.7

0.8

0.9

SVD

BigClam

Ours

0

200

400

600

800

0

2

4

6

8

SVD

0

200

400

600

800

0

2

4

6

8

10

BigClam

0

200

400

600

800

0

2

4

6

8

Ours

0

200

400

600

800

0

1

2

0

200

400

600

800

0

1

0.5

0.0

0.5

0.5

0.0

0.5

0.0

0.5

1.0

0

2

0

2

4

3 Community-Based Graph Factorization

Consider the set of undirected, unweighted graphs on n nodes, i.e., the set of graphs with symmetric adjacency matrices in { 0 , 1 } n × n . We propose an edge-independent generative model for such graphs. Given nonnegative parameter matrices B ∈ R n × k B + and C ∈ R n × k C + , we set the probability of an edge existing between nodes i and j to be the ( i, j ) -th entry of matrix ˜ A :

where σ is the logistic function. Here k B , k C are the number of homophilous/heterophilous clusters. Intuitively, if b i ∈ R k B + is the i -th row of matrix B , then b i is the affinity of node i to each of the k B homophilous communities. Similarly, c i ∈ R k C + is the affinity of node i to the k C heterophilous communities. As an equivalent statement, for each pair of nodes i and j , ˜ A i,j := σ ( b i b j -c i c ⊤ j ) . We will soon discuss the precise interpretation of this model, but the idea is roughly similar to the attract-repel framework of Peysakhovich & Bottou (2021). When nodes i and j have similar 'attractive' b embeddings, i.e., when b i b ⊤ j is high, the likelihood of an edge between them increases, hence why the B factor is homophilous. By contrast, the C factor is 'repulsive'/heterophilous since, when c i c ⊤ j is high, the likelihood of an edge between i and j decreases.

Alternate expression We note that the model above can also be expressed in a form that normalizes cluster assignments and is more compact, in that it combines the homophilous and heterophilous cluster assignments. Instead of B and C , this form uses a matrix V ∈ [0 , 1] n × k and a diagonal matrix W ∈ R k × k , where k = k B + k C is the total number of clusters. In particular, let m B and m C be the vectors containing the maximums of each column of B and C . By setting

the constraint on V is satisfied. Further, V WV ⊤ = BB ⊤ -CC ⊤ , so

Here, if v i ∈ [0 , 1] k is the i -th row of matrix V , then v i is the soft (normalized) assignment of node i to the k communities. The diagonal entries of W represent the strength of the homophily (if positive) or heterophily (if negative) of the communities. For each entry, ˜ A i,j = σ ( v i W v ⊤ j ) . We use these two forms interchangeably throughout this work.

Interpretation The edge probabilities output by this model have an intuitive interpretation. Recall that there are bijections between probability p ∈ [0 , 1] , odds o = p 1 -p ∈ [0 , ∞ ) , and logit ℓ = log( o ) ∈ ( -∞ , + ∞ ) . The logit of the link probability between nodes i and j is v ⊤ i W v j , which is a summation of terms v ic v jc W cc over all communities c ∈ [ k ] . If the nodes both fully participate in community c , that is, v ic = v jc = 1 , then the edge logit is changed by W cc starting from a baseline of 0 , or equivalently, the odds of an edge is multiplied by exp( W cc ) starting from a baseline odds of 1 ; if either of the nodes participates only partially in community c , then the change in logit and odds is accordingly prorated. Homophily and heterophily also have a clear interpretation in this model: homophilous communities, which are expressed in B , are those with W cc > 0 , where two nodes both participating in the community increases the odds of a link, whereas communities with W cc < 0 , which are expressed in C , are heterophilous, and coparticipation decreases the odds of a link.

4 Related Work

Community detection via interpretable factorizations There is extensive prior work on community detection and node clustering (Schaeffer, 2007; Aggarwal & Wang, 2010; Nascimento & De Carvalho, 2011), perhaps the most well-known being the normalized cuts algorithm of Shi & Malik (2000), which produces a clustering based on the entrywise signs of an eigenvector of the graph Laplacian matrix. However, the clustering algorithms which are most relevant to our work are those based on non-negative matrix factorization (NMF) (Lee & Seung, 1999; Berry et al., 2007; Wang & Zhang, 2012; Gillis, 2020), many of which can be seen as integrating nonnegativity constraints into

the broader, well-studied random dot product graph (RDPG) model (Young & Scheinerman, 2007; Scheinerman & Tucker, 2010; Athreya et al., 2017). One such algorithm is that of Yu et al. (2005), which approximately factors a graph's adjacency matrix A ∈ { 0 , 1 } n × n into two positive matrices H and Λ , where H ∈ R n × k + is left-stochastic and Λ ∈ R k × k + is diagonal, such that H Λ H ⊤ ≈ A . Here H represents a soft clustering of the n nodes into k clusters, while the diagonal entries of Λ represent the prevalence of edges within clusters. Note the similarity of the factorization to our model, save for the lack of a nonlinearity. Other NMF approaches include those of Ding et al. (2008), Yang et al. (2012), Kuang et al. (2012), and Kuang et al. (2015) (SYMNMF).

Modeling heterophily Much of the existing work on graph models has an underlying assumption of network homophily (Johnson et al., 2010; Noldus & Van Mieghem, 2015). There has been significant recent interest in the limitations of graph neural network (GNN) models (Duvenaud et al., 2015; Kipf & Welling, 2017; Hamilton et al., 2017) at addressing network heterophily (NT & Maehara, 2019; Zhu et al., 2020; Zheng et al., 2022), as well as proposed solutions (Pei et al., 2020; Yan et al., 2021), but relatively less work for models applicable to clustering. Some existing NMF approaches to clustering do naturally model heterophilous structure in networks. The model of Nourbakhsh et al. (2014), for example, is similar to that of Yu et al. (2005), but allows the cluster affinity matrix Λ to be non-diagonal; this allows for inter-cluster edge affinity to exceed intra-cluster edge affinity, so heterophily can arise in this model, though it is not a focus of their work. Another example is the model in Miller et al. (2009), which is similar to ours, though it restricts the cluster assignment matrix V to be binary; additionally, their training algorithm is not based on gradient descent as ours is, and it does not scale to larger networks. More recently, Rubin-Delanchy et al. (2017) and Peysakhovich & Bottou (2021) propose simple decompositions which allow for representation of non-PSD adjacency matrices. The model in the latter work is a factorization of the form A ≈ D + BB ⊤ -CC ⊤ , where D ∈ R n × n is diagonal and B , C ∈ R n × k are low-rank; excluding the diagonal D term, the model in the former work is algebraically identical. The authors discuss how, interestingly, this model separates the homophilous and heterophilous structure into different factors, namely B and C , corresponding to positive and negative eigenvalues, respectively. Note that these decompositions do not include a nonlinear linking function, which is crucial to our exact factorization results, and the respective works do not investigate constraining the factors to be nonnegative.

Overlapping communities and exact embeddings Many models discussed above focus on the single-label clustering task and thus involve highly-constrained factorizations (e.g., sum-to-one conditions). We are interested in the closely related but distinct task of multi-label clustering, also known as overlapping community detection (Xie et al., 2013; Javed et al., 2018), which involves less constrained, more expressive factorizations. The BIGCLAM algorithm of Yang & Leskovec (2013) uses the following generative model for this task: the probability of a link between two nodes i and j is given by 1 -exp( -f i · f j ) , where f i , f j ∈ R k + represent the intensities with which the nodes participate in each of the k communities. Note that BIGCLAM assumes strict homophily of the communities: two nodes participating in the same community always increases the probability of a link. However, this model allows for expression of very dense intersections of communities, which the authors observe is generally a characteristic of real-world networks. To ensure that output entries are probabilities, BIGCLAM's factorization includes a nonlinear linking function (namely, f ( x ) = 1 -e x ), like our model and LPCA. Recent work outside clustering and community detection on graph generative models (Rendsburg et al., 2020; Chanpuriya et al., 2020) suggests that incorporating a nonlinear linking function can greatly increase the expressiveness of factorization-based graph models, to the point of being able to exactly represent a graph. This adds to a growing body of literature on expressiveness guarantees for embeddings on relational data (Sala et al., 2018; Bhattacharjee & Dasgupta, 2020; Boratko et al., 2021). Most relevant to our work, as previously discussed, Chanpuriya et al. (2020) provide a guarantee for exact low-rank representation of graphs with bounded max degree when using the LPCA factorization model. In this work, we provide a new such guarantee, except for bounded arboricity, which is more applicable to real-world networks, and extend these guarantees to our community-based factorization.

5 Theoretical Results

We first restate the main result from Chanpuriya et al. (2020) on exact representation of graphs with bounded max degree using the logistic principal components analysis (LPCA) model, which

reconstructs a graph A ∈ { 0 , 1 } n × n using logit factors X , Y ∈ R n × k via

Note that unlike our community-based factorization, the factors of the LPCA model are not nonnegative, and the factorization does not reflect the symmetry of the undirected graph's adjacency matrix. Regardless of the model's interpretability, the following theorem provides a significant guarantee on its expressiveness. We use the following notation: given a matrix M , let H ( M ) denote the matrix resulting from entrywise application of the Heaviside step function to M , that is, setting all positive entries to 1 , negative entries to 0 , and zero entries to 1 / 2 .

Theorem 5.1 (Exact LPCA Factorization for Bounded-Degree Graphs ) . Let A ∈ { 0 , 1 } n × n be the adjacency matrix of a graph G with maximum degree c . Then there exist matrices X , Y ∈ R n × (2 c +1) such that A = H ( XY ⊤ ) .

This corresponds to arbitrarily small approximation error in the LPCA model (Equation 4) because, provided such factors X , Y for some graph A , we have that lim s →∞ σ ( s XY ⊤ ) = H ( XY ⊤ ) = A . That is, we can scale the factors larger to reduce the error to an arbitrary extent.

We expand on this result in two ways. First, give a new bound for exact embedding in terms of arboricity, rather than max degree. This increases the applicability to real-world networks, which often are sparse (i.e., low arboricity) and have right-skewed degree distributions (i.e., high max degree). Second, we show that any rankk LPCA factorization can be converted to our model's symmetric nonnegative factorization with O ( k ) communities. This extends the guarantees on the LPCA model's power for exact representation of graphs, both the prior guarantee in terms of max degree and our new one in terms of arboricity, to our community-based model as well. In Appendix A.1, we also introduce a natural family of graphs - Community Overlap Threshold (COT) graphs - for which our model's community-based factorization not only exactly represents the graph, but also must capture some latent structure to do so with sufficiently low embedding dimensionality.

Arboricity bound for exact representation We will use the following well-known fact: the rank of the entrywise product of two matrices is at most the product of their individual ranks, that is,

Theorem 5.2 (Exact LPCA Factorization for Bounded-Arboricity Graphs) . Let A ∈ { 0 , 1 } n × n be the adjacency matrix of an undirected graph G with arboricity α . Then there exist embeddings X , Y ∈ R n × (4 α 2 +1) such that A = H ( XY ⊤ ) .

Proof. Let the undirected graph A have arboricity α , i.e., the edges can be partitioned into α forests. We produce a directed graph B from A by orienting the edges in these forests so that each node's edges point towards its children. Now A = B + B ⊤ , and every node in B has in-degree at most α .

Let V ∈ R n × 2 α be the Vandermonde matrix with V t,j = t j -1 . For any c ∈ R 2 α , [ V c ]( t ) = ∑ 2 α j =1 c ( j ) · t j -1 , that is, V c ∈ R n is a degree(2 α ) polynomial with coefficients c evaluated at the integers t ∈ [ n ] = { 1 , . . . , n } . Let b i be the i th column of B . We seek to construct a polynomial such that for t with b i ( t ) = 1 , [ V c i ]( t ) = 0 , and [ V c i ]( t ) < 0 elsewhere; that is, when inputting an index t ∈ [ n ] such that the t th node is an in-neighbor of the i th node, we want the polynomial to output 0 , and for all other indices in [ n ] , we want it to have a negative output. Letting N ( i ) denote the in-neighbors of the i th node, a simple instantiation of such a polynomial in t is -1 · ∏ j ∈ N ( i ) ( t -j ) 2 . Note that since all nodes have in-degree at most α , this polynomial's degree is at most 2 α , and hence there exists a coefficient vector c i ∈ R 2 α encoding this polynomial.

Let C ∈ R n × 2 α be the matrix resulting from stacking such coefficient vectors for each of the n nodes. Consider P = V C ∈ R n × n : P i,j is 0 if B i,j = 1 and negative otherwise. Then ( P ◦ P ⊤ ) i,j is 0 when either B i,j = 1 or ( B ⊤ ) i,j = 1 and positive otherwise; equivalently, since A = B + B ⊤ , ( P ◦ P ⊤ ) i,j = 0 iff A i,j = 1 . Take any positive ϵ less than the smallest positive entry of P ◦ P ⊤ . Letting J be an all-ones matrix, define M = ϵ J -( P ◦ P ⊤ ) . Note that M i,j > 0 if A = 1 and M i,j < 0 if A = 0 , that is, M = H ( A ) as desired. Since rank ( J ) = 1 and rank ( P ) ≤ 2 α , by the bound on the rank of entrywise products of matrices, the rank of M is at most (2 α ) 2 +1 . ■

Exact representation with community factorization LPCA factors X , Y ∈ R n × k can be processed into nonnegative factors B ∈ R n × k B + and C ∈ R n × k C + such that k B + k C = 6 k and

As a rough outline of the argument that follows, we will need 6 k columns in the new factors B , C , up from the k columns in X , Y , because accounting for the possible asymmetry in XY ⊤ will double the required columns, and accounting for the nonnegativity of B , C will triple the required columns. Observe that the left-hand side can only represent symmetric matrices, but XY ⊤ is not necessarily symmetric even if H ( XY ⊤ ) = A for a symmetric A . For this reason, we use a symmetrization: let L = 1 2 ( XY ⊤ + Y X ⊤ ) . Note that H ( L ) = H ( XY ⊤ ) , so if XY ⊤ constitutes an exact representation of A in that H ( XY ⊤ ) = A , so too do both expressions for L in Equation 5. Pseudocode for the procedure of constructing B , C given X , Y is given in Algorithm 1. The concept of this algorithm is to first separate the logit matrix L into a sum and difference of rank1 components via eigendecomposition. Each of these components can be written as + vv ⊤ or -vv ⊤ with v ∈ R n , where the sign depends on the sign of the eigenvalue. Each component is then separated into a sum and difference of three outer products of nonnegative vectors, via the following Lemma 5.3.

Lemma 5.3. Let ϕ : R → R denote the ReLU function, i.e., ϕ ( z ) = max { z, 0 } . For any vector v ,

Proof. Take any v ∈ R k . Then

where the first step follows from v = ϕ ( v ) -ϕ ( -v ) , and the last step from | v | = ϕ ( v )+ ϕ ( -v ) . ■

Algorithm 1 follows from Lemma 5.3 and constitutes a constructive proof of the following theorem: Theorem 5.4 (Exact Community Factorization from Exact LPCA Factorization) . Given a symmetric matrix A ∈ { 0 , 1 } and X , Y ∈ R n × k such that A = H ( XY ⊤ ) , there exist nonnegative matrices B ∈ R n × k B + and C ∈ R n × k C + such that k B + k C = 6 k and A = H ( BB ⊤ -CC ⊤ ) .

Algorithm 1 Converting LPCA Factors to Community Factors

Input logit factors X , Y ∈ R n × k

Output

B

∈

R

n

×

k

B

+

,

C

∈

R

n

×

k

C

+

such that

k

B

+

k

1:

Set

Q

∈

R

n

×

2

k

and

λ

∈

R

2

k

by truncated eigendecomposition such that

2: B ∗ ← Q + × diag( + λ + ) , where λ + , Q + are the positive eigenvalues/vectors √

3:

C

∗

Q

-

×

diag(

-

λ

-

)

, where

λ

-

,

Q

-

are the negative eigenvalues/vectors

4: B ← ( 2 ϕ ( B ∗ ); 2 ϕ ( -B ∗ ); | C ∗ | ) ▷ ϕ and | · | are entrywise ReLU and absolute value 5: C ← ( √ 2 ϕ ( C ∗ ); √ 2 ϕ ( -C ∗ ); | B ∗ | )

√

6: return B , C

As stated in the introduction to this section, Theorem 5.4 extends any upper bound on the exact factorization dimensionality from the LPCA model to our community-based model. That is, up to a constant factor, the bound in terms of max degree from Theorem 5.1 and the bound in terms of arboricity from Theorem 5.2 also apply to our model; for brevity, we state just the latter here.

Corollary 5.5 (Exact Community Factorization for Bounded-Arboricity Graphs) . Let A ∈ { 0 , 1 } n × n be the adjacency matrix of an undirected graph G with arboricity α . Then there exist nonnegative embeddings B ∈ R n × k B + and C ∈ R n × k C + such that k B + k C = 6(4 α 2 + 1) and A = H ( BB ⊤ -CC ⊤ ) .

√

C

= 6

k

Note that Corollary 5.5 is purely a statement about the capacity of our model; Theorem 5.2 stems from a constructive proof based on polynomial interpolation, and therefore so too does this corollary. We do not expect this factorization to be informative about the graph's latent structure. In the following Section 6, we will fit the model with an entirely different algorithm for downstream applications.

6 Experiments

We now present a training algorithm to fit our model, then evaluate our method on a benchmark of five real-world networks.

6.1 Training Algorithm

Given an input graph A ∈ { 0 , 1 } n × n , we find low-rank nonnegative matrices B and C such that the model produces ˜ A = σ ( BB ⊤ -CC ⊤ ) ∈ (0 , 1) n × n as in Equation 1 which approximately matches A . In particular, we train the model to minimize the sum of binary cross-entropies of the link predictions over all pairs of nodes:

We fit the parameters by gradient descent over this loss, as well as L 2 regularization of the factors B and C , subject to the nonnegativity of B and C . This algorithm is fairly straightforward; pseudocode is given in Algorithm 2. This is quite similar to the training algorithm of Chanpuriya et al. (2020), but in contrast to that work, which only targets an exact fit, we explore the expression of graph structure in the factors and their utility in downstream tasks. Regularization of the factors is implemented to this end to avoid overfitting. Though in the main paper we outline and evaluate a non-stochastic version of the training algorithm, it can generalize straightforwardly to a more scalable stochastic version, e.g., by sampling links and non-links for the loss function and using projected SGD. In Appendix A.2, we discuss an industry application of our model to tabular dataset completion, for which we employ such stochastic training. Here, we use the simpler non-stochastic training to isolate the impact of model capacity, which is the focus of this work, as opposed to optimization.

Algorithm 2 Fitting the Constrained Model

Input adjacency matrix A ∈ { 0 , 1 } n × n , regularization weight λ ≥ 0 , # of iterations I , # of homo/heterophilous communities k B /k C

Output fitted factors B ∈ R n × k B + , C ∈ R n × k C + such that σ ( BB ⊤ -CC ⊤ ) ≈ A

1: Initialize B , C by setting entries to independent samples of Unif (0 , 1 / √ k B ) , Unif (0 , 1 / √ k C )

2: for i ← 1 to I do

3: ˜ A ← σ ( BB ⊤ -CC ⊤ )

4: R ←-∑ ij ( A ij log( ˜ A ij ) + (1 -A ij ) log(1 -˜ A ij ) )

5: R ← R + λ ( ∥ B ∥ 2 F + ∥ C ∥ 2 F )

6:

Calculate

∂

B

,

C

R

, the gradient of

R

w.r.t.

B

,

C

, via differentiation through Steps 2 to 4

7: Update B , C to minimize R using ∂ B , C R , subject to B , C ≥ 0

8: end for

9: return B , C

Implementation details Our implementation uses PyTorch (Paszke et al., 2019) for automatic differentiation and minimizes loss using the SciPy (Jones et al., 2001) implementation of the LBFGS (Liu & Nocedal, 1989; Zhu et al., 1997) algorithm with default hyperparameters and up to a max of 200 iterations of optimization. We set regularization weight λ = 10 as in Yang & Leskovec (2013). We include code in the form of a Jupyter notebook (Pérez & Granger, 2007) demo.

6.2 Datasets

We use five fairly common mid-size datasets ranging from around 1K to 10K nodes. The selection of these five datasets is partly based on the presence of ground-truth multi-labels, which allows for evaluating the overlapping clustering methods. Statistics for these datasets, including the degeneracy

of each network, are given in Table 1. Note that degeneracy is an upper bound on arboricity. We note that the mid-sized networks used in our empirical work actually underemphasize the significance of our theoretical arboricity bound: see, e.g., the real-world networks in Pashanasangi & Seshadhri (2021), which have up to tens of millions of nodes but still have degeneracies at most in the hundreds.

BLOG is a social network of relationships between online bloggers; the node labels represent interests of the bloggers. Similarly, YOUTUBE is a social network of YouTube users, and the labels represent groups that the users joined.

POS is a word co-occurrence network: nodes represent words, and there are edges between words which are frequently adjacent in a section of the Wikipedia corpus. Each node label represents the part-of-speech of the word. PPI is a subgraph of the protein-protein interaction network for Homo Sapiens. Labels represent biological states. Finally, AMAZON is a co-purchasing network: nodes represent products, and there are edges between products which are frequently purchased together. Labels represent categories of products.

While social networks like the former two in this list are generally dominated by homophily (McPherson et al., 2001), the latter three should exhibit significant heterophily. For co-purchasing networks like AMAZON, depending on the product, two of the same kind of product are generally not copurchased, e.g., Pepsi and Coke, as discussed in Peysakhovich & Bottou (2021). Though less intuitively accessible, there is also prior discussion of disassortativity in word adjacencies (Foster et al., 2010; Zweig, 2016), as well as in PPI networks (Newman, 2002; Hase et al., 2010).

6.3 Results

Expressiveness First, we investigate the expressiveness of our generative model, that is, the fidelity with which it can reproduce an input network. In Section 1, we used a simple synthetic network to show that our model is more expressive than others due to its ability to represent heterophilous structures in addition to homophilous ones. We now evaluate the expressiveness of our model on realworld networks. As with the synthetic graph, we fix the number of communities or singular vectors, fit the model, then evaluate the reconstruction error. In Figure 4, we compare the results of our model with those of SVD, BIGCLAM (which is discussed in detail in Section 4), and SYMNMF (Kuang et al., 2015). SYMNMF simply factors the adjacency matrix as A ≈ HH ⊤ , where H ∈ R n × k + ; note that, like SVD, SYMNMF does not necessarily output a matrix whose entries are probabilities (i.e., bounded in [0 , 1] ), and hence it is not a natural graph generative model like ours and BIGCLAM.



























































For each method, we fix the number of communities or singular vectors at the ground-truth number. For this experiment only, we are not concerned with learning the latent structure of the graph; the only goal is accurate representation of the network with limited parameters. So, for a fair comparison with SVD, we do not regularize the training of the other methods. Our method consistently has the lowest reconstruction error, both in terms of Frobenius error and entrywise cross-entropy (Equation 6). We particularly highlight the improvement over BIGCLAM; the salient difference between these models, both of which are factorization-based, include a nonlinear link function, and assign node communities using nonnegative factors, is the presence of the heterophilous -CC ⊤ term in our model. Thus, the improvement directly reflects the value of incorporating heterophily into the model. Interestingly, we find the most significant improvement exactly on the three datasets which have been noted to exhibit significant heterophily: POS, PPI, and AMAZON.

Similarity to ground-truth communities To assess the interpretability of clusters generated by our method, we evaluate the similarity of these clusters to ground-truth communities (i.e., class labels), and we compare other methods for overlapping clustering. We additionally compare to another recent but non-generative approach, the VGRAPH method of Sun et al. (2019), which is based on link clustering; the authors found their method to generally achieve state-of-the-art results in this task. For all methods, we set the number of communities to be detected as the number of ground-truth communities. We report F1-Score as computed in Yang & Leskovec (2013). See Figure 5 (left): the performance of our method is competitive with SYMNMF, BIGCLAM, and vGraph.







































































Interpretable link prediction We assess the predictive power of our generative model on the link prediction task. As discussed in Section 3, the link probabilities output by our model are interpretable in terms of a clustering of nodes that it generates; we compare results with our method to those with other models which permit similar interpretation, namely BIGCLAM and SYMNMF. We randomly select 10% of node pairs to hold out, fit the models on the remaining 90%, then use the trained models to predict links between node pairs in the held out 10%. As a baseline, we also show results for randomly predicting link or no link with equal probability. See Figure 5 (right). The performance of our method is competitive with or exceeds those of the other methods in terms of F1 Score.

7 Conclusion

We introduce a community-based graph generative model based on symmetric nonnegative matrix factorization which can represent both homophily and heterophily. We add to a prior guarantee of exact representation for bounded degree graphs with a broader guarantee for bounded arboricity graphs, and we show that both of these guarantees apply to our more interpretable graph model. We illustrate our model's capabilities with experiments on a synthetic motivating example. Experiments on real-world networks show its effectiveness on several key tasks. Broadly, our results suggest that incorporating heterophily into methods for networks can improve both theoretical grounding and empirical performance, while maintaining interpretability. Future directions include deeper understanding of the expressiveness of low-rank logit models and convergence of training algorithms.

Acknowledgments and Disclosure of Funding

This project was partially supported by an Adobe Research grant, along with NSF Grants 2046235 and 1763618.

References

Aggarwal, C. C. and Wang, H. A survey of clustering algorithms for graph data. In Managing and Mining Graph Data , pp. 275-301. Springer, 2010.

Athreya, A., Fishkind, D. E., Tang, M., Priebe, C. E., Park, Y., V ogelstein, J. T., Levin, K., Lyzinski, V., and Qin, Y. Statistical inference on random dot product graphs: a survey. The Journal of Machine Learning Research , 18(1):8393-8484, 2017.

Berry, M. W., Browne, M., Langville, A. N., Pauca, V. P., and Plemmons, R. J. Algorithms and applications for approximate nonnegative matrix factorization. Computational Statistics & Data Analysis , 52(1):155-173, 2007.

Bhattacharjee, R. and Dasgupta, S. What relations are reliably embeddable in euclidean space? In Algorithmic Learning Theory , pp. 174-195. PMLR, 2020.

Bonato, A. A survey of models of the web graph. In Workshop on Combinatorial and Algorithmic Aspects of Networking , pp. 159-172. Springer, 2004.

Boratko, M., Zhang, D., Monath, N., Vilnis, L., Clarkson, K. L., and McCallum, A. Capacity and bias of learned geometric embeddings for directed graphs. Advances in Neural Information Processing Systems , 34:16423-16436, 2021.

Breitkreutz, B.-J., Stark, C., Reguly, T., Boucher, L., Breitkreutz, A., Livstone, M., Oughtred, R., Lackner, D. H., Bähler, J., Wood, V., et al. The biogrid interaction database: 2008 update. Nucleic acids research , 36(suppl_1):D637-D640, 2007.

Chanpuriya, S., Musco, C., Sotiropoulos, K., and Tsourakakis, C. Node embeddings and exact lowrank representations of complex networks. Advances in Neural Information Processing Systems , 33, 2020.

Ding, C., Li, T., and Jordan, M. I. Nonnegative matrix factorization for combinatorial optimization: Spectral clustering, graph matching, and clique finding. In 2008 Eighth IEEE International Conference on Data Mining , pp. 183-192. IEEE, 2008.

Donoho, D. L. and Stodden, V. When does non-negative matrix factorization give a correct decomposition into parts? In Advances in Neural Information Processing Systems 16 , pp. 1141-1148. MIT Press, 2003.

Duvenaud, D., Maclaurin, D., Aguilera-Iparraguirre, J., Gómez-Bombarelli, R., Hirzel, T., AspuruGuzik, A., and Adams, R. P. Convolutional networks on graphs for learning molecular fingerprints. In Advances in Neural Information Processing Systems 28 , pp. 2224-2232, 2015.

Foster, J. G., Foster, D. V ., Grassberger, P., and Paczuski, M. Edge direction and the structure of networks. Proceedings of the National Academy of Sciences , 107(24):10815-10820, 2010.

Gillis, N. Nonnegative Matrix Factorization . SIAM, 2020.

Hamilton, W., Ying, Z., and Leskovec, J. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Hase, T., Niimura, Y., and Tanaka, H. Difference in gene duplicability may explain the difference in overall structure of protein-protein interaction networks among eukaryotes. BMC Evolutionary Biology , 10(1):1-15, 2010.

Javed, M. A., Younis, M. S., Latif, S., Qadir, J., and Baig, A. Community detection in networks: A multidisciplinary review. Journal of Network and Computer Applications , 108:87-111, 2018.

Johnson, S., Torres, J. J., Marro, J., and Munoz, M. A. Entropic origin of disassortativity in complex networks. Physical Review Letters , 104(10):108702, 2010.

Jones, E., Oliphant, T., Peterson, P., et al. SciPy: Open source scientific tools for Python, 2001. URL http://www.scipy.org/ .

Kipf, T. N. and Welling, M. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations , 2017.

Kuang, D., Ding, C., and Park, H. Symmetric nonnegative matrix factorization for graph clustering. In Proceedings of the 2012 SIAM International Conference on Data Mining , pp. 106-117. SIAM, 2012.

Kuang, D., Yun, S., and Park, H. Symnmf: nonnegative low-rank approximation of a similarity matrix for graph clustering. Journal of Global Optimization , 62(3):545-574, 2015.

Lee, D. D. and Seung, H. S. Learning the parts of objects by non-negative matrix factorization. Nature , 401(6755):788-791, 1999.

Liu, D. C. and Nocedal, J. On the limited memory BFGS method for large scale optimization. Mathematical Programming , 45(1-3):503-528, 1989.

Mason, O. and Verwoerd, M. Graph theory and networks in biology. IET Systems Biology , 1(2): 89-119, 2007.

McPherson, M., Smith-Lovin, L., and Cook, J. M. Birds of a feather: Homophily in social networks. Annual review of sociology , 27(1):415-444, 2001.

Miller, K. T., Griffiths, T. L., and Jordan, M. I. Nonparametric latent feature models for link prediction. In Advances in Neural Information Processing Systems 22 , pp. 1276-1284. Curran Associates, Inc., 2009.

Nascimento, M. C. and De Carvalho, A. C. Spectral methods for graph clustering-a survey. European Journal of Operational Research , 211(2):221-231, 2011.

Newman, M. E. Assortative mixing in networks. Physical Review Letters , 89(20):208701, 2002.

Noldus, R. and Van Mieghem, P. Assortativity in complex networks. Journal of Complex Networks , 3(4):507-542, 2015.

Nourbakhsh, F., Bulo, S. R., and Pelillo, M. A matrix factorization approach to graph compression. In 2014 22nd International Conference on Pattern Recognition , pp. 76-81. IEEE, 2014.

NT, H. and Maehara, T. Revisiting graph neural networks: All we have is low-pass filters. arXiv preprint arXiv:1905.09550 , 2019.

Pashanasangi, N. and Seshadhri, C. Faster and generalized temporal triangle counting, via degeneracy ordering. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pp. 1319-1328, 2021.

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pp. 8024-8035. Curran Associates, Inc., 2019.

Pei, H., Wei, B., Chang, K. C., Lei, Y., and Yang, B. Geom-gcn: Geometric graph convolutional networks. In 8th International Conference on Learning Representations, ICLR 2020 , 2020.

Pérez, F. and Granger, B. E. IPython: a system for interactive scientific computing. Computing in Science and Engineering , 9(3):21-29, May 2007. ISSN 1521-9615. doi: 10.1109/MCSE.2007.53. URL https://ipython.org .

Peysakhovich, A. and Bottou, L. An attract-repel decomposition of undirected networks. arXiv preprint arXiv:2106.09671 , 2021.

Qiu, J., Dong, Y., Ma, H., Li, J., Wang, K., and Tang, J. Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vec. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining , pp. 459-467. ACM, 2018.

Rendsburg, L., Heidrich, H., and Von Luxburg, U. Netgan without gan: From random walks to low-rank approximations. In International Conference on Machine Learning , pp. 8073-8082. PMLR, 2020.

Rubin-Delanchy, P., Cape, J., Tang, M., and Priebe, C. E. A statistical interpretation of spectral embedding: the generalised random dot product graph. arXiv preprint arXiv:1709.05506 , 2017.

Sala, F., De Sa, C., Gu, A., and Ré, C. Representation tradeoffs for hyperbolic embeddings. In International Conference on Machine Learning , pp. 4460-4469. PMLR, 2018.

Schaeffer, S. E. Graph clustering. Computer Science Review , 1(1):27-64, 2007.

Scheinerman, E. R. and Tucker, K. Modeling graphs using dot product representations. Computational statistics , 25(1):1-16, 2010.

Scott, J. Social network analysis. Sociology , 22(1):109-127, 1988.

Seshadhri, C., Sharma, A., Stolman, A., and Goel, A. The impossibility of low-rank representations for triangle-rich complex networks. Proceedings of the National Academy of Sciences , 117(11): 5631-5637, 2020.

Shi, J. and Malik, J. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 22(8):888-905, 2000.

Sun, F., Qu, M., Hoffmann, J., Huang, C., and Tang, J. vgraph: A generative model for joint community detection and node representation learning. In Advances in Neural Information Processing Systems 32 , pp. 512-522, 2019.

Tang, L. and Liu, H. Relational learning via latent social dimensions. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pp. 817-826. ACM, 2009.

Wang, Y.-X. and Zhang, Y.-J. Nonnegative matrix factorization: A comprehensive review. IEEE Transactions on Knowledge and Data Engineering , 25(6):1336-1353, 2012.

Xie, J., Kelley, S., and Szymanski, B. K. Overlapping community detection in networks: The state-of-the-art and comparative study. ACM Computing Surveys , 45(4):43:1-43:35, 2013.

Yan, Y., Hashemi, M., Swersky, K., Yang, Y., and Koutra, D. Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks. arXiv preprint arXiv:2102.06462 , 2021.

Yang, J. and Leskovec, J. Overlapping community detection at scale: a nonnegative matrix factorization approach. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining , pp. 587-596, 2013.

Yang, J. and Leskovec, J. Defining and evaluating network communities based on ground-truth. Knowledge and Information Systems , 42(1):181-213, 2015.

Yang, Z., Hao, T., Dikmen, O., Chen, X., and Oja, E. Clustering by nonnegative matrix factorization using graph random walk. In Advances in Neural Information Processing Systems , pp. 1079-1087, 2012.

Young, S. J. and Scheinerman, E. R. Random dot product graph models for social networks. In International Workshop on Algorithms and Models for the Web-Graph , pp. 138-149. Springer, 2007.

Yu, K., Yu, S., and Tresp, V . Soft clustering on graphs. In Advances in Neural Information Processing Systems , pp. 1553-1560, 2005.

Zheng, X., Liu, Y., Pan, S., Zhang, M., Jin, D., and Yu, P. S. Graph neural networks for graphs with heterophily: A survey. arXiv:2202.07082 , 2022.

Zhu, C., Byrd, R. H., Lu, P., and Nocedal, J. Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Transactions on Mathematical Software (TOMS) , 23(4):550-560, 1997.

Zhu, J., Yan, Y., Zhao, L., Heimann, M., Akoglu, L., and Koutra, D. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems , 33, 2020.

Zweig, K. A. Are word-adjacency networks networks? In Towards a theoretical framework for analyzing complex linguistic networks , pp. 153-163. Springer, 2016.

A Appendix

A.1 COT Graph Exact Representation

As a theoretical demonstration of the capability of our model to learn latent structure, we additionally show that our model can exactly represent a natural family of graphs, which exhibits both homophily and heterophily, with small k and interpretably. The family of graphs is specified below in Definition 1; roughly speaking, nodes in such graphs share an edge iff they coparticipate in some number of homophilous communities and don't coparticipate in a number of heterophilous communities. For example, the motivating graph described in Section 2 would be an instance of such a graph if an edge occurs between two users iff the two users are from the same city and have different genders.

Definition 1 (Community Overlap Threshold (COT) Graph) . An unweighted, undirected graph whose edges are determined by an overlapping clustering and a 'thresholding' integer t ∈ Z as follows: for each vertex i , there are two latent binary vectors b i ∈ { 0 , 1 } k b and c i ∈ { 0 , 1 } k c , and there is an edge between vertices i and j iff b i · b j -c i · c j ≥ t .

Theorem A.1 (Compact Representation of COT Graphs) . Suppose A is the adjacency matrix of a COT graph on n nodes with latent vectors b i ∈ { 0 , 1 } k b and c i ∈ { 0 , 1 } k c for i ∈ { 1 , 2 , . . . , n } . Let k = k b + k c . Then, for any ϵ > 0 , there exist V ∈ [0 , 1] n × ( k +1) and diagonal W ∈ R ( k +1) × ( k +1) such that ∥ ∥ σ ( V WV ⊤ ) -A ∥ ∥ F < ϵ .

Proof. Let t be the thresholding integer of the graph, and let the rows of B ∈ { 0 , 1 } n × k b and C ∈ { 0 , 1 } n × k c contain the vectors b and c of all nodes. Via Equation 2, we can find V ∗ ∈ [0 , 1] n × k and diagonal W ∗ ∈ R k × k such that V ∗ W ∗ V ∗⊤ = BB ⊤ -CC ⊤ . Now let

Then ( V WV ⊤ ) ij = b i · b j -c i · c j + 1 2 -t . Hence ( V WV ⊤ ) ij > 0 iff b i · b j -c i · c j > t -1 2 , which is true iff A ij = 1 by the assumption on the graph. Similarly, ( V WV ⊤ ) ij < 0 iff A ij = 0 . It follows that

A.2 Application: Tabular Data Completion

We apply our link prediction algorithm to the task of data completion for categorical tabular data. We first transform such data to a graph as follows: We create a node for each row, as well as a node for each unique category for each of the columns. For each entry of the table, a link occurs between the node for the row and the node for the value in the entry, which is one of the possible categories of the column (e.g., if Bob is American, then a link occurs between the row node for 'Bob' and the category node for 'American'). No links occur between two nodes for rows, or between two nodes for category values. For the negative samples, we select only edges between nodes for rows and nodes for category values; the negative samples would otherwise be dominated by pairs of nodes for rows.

We use two proprietary company datasets datasets which we call COMPANY A and COMPANY B. We use the same experimental setup as described in Section 6.3. Unlike in Section 6.3, we have a restriction on the possible links: each row node must be linked to exactly one of the value nodes for each of the categories. Hence, for each combination of row node and column, we predict a link only with the value node with the highest predicted link probability. We set the number of communities for all methods to 20 and 50 for COMPANY A and COMPANY B, respectively. We employ the stochastic version of our algorithm, based on projected SGD, for which code is also provided. We generally use a batch size of 100 ; we find that the optimization of BIGCLAM often diverges on COMPANY B with this batch size, so we instead use batches of size 1000 for its optimization. The results are provided in Figure 6. Notably, our proposed approach outperforms comparable (community-based factorization) methods in terms of accuracy. As a baseline, we also show the accuracy when completing the tables by simply selecting the most common categorical value for each column ('PLURALITY'). All methods outperform this baseline on both datasets.





























Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends for by Chanpuriya et al (2020) for networks with homophilous and heterophilous edges. They show that their model is able to yield interesting results and they give theoretical underpinnings as well.\n\nSTRENGTHS:\nThe model is straightforward but seems to work well. Due do its relative simplicity and its relationship to previous work it is possible to give theoretical guarantees.\n\nWEAKNESSES:\nThe paper seems to ignore relationships to multilayer and multiplex networks which are also able to capture different types of relationships through different layers. Mason Porter, Peter Mucha and others have written many papers on such networks. There is also a literature on spectral methods for signed networks which can also capture heterophily; Mihai Cucuringu comes to mind. These areas should be mentioned in the related works section. \n\nSome of the presentation is not clear:\nIn (4) it seems that A is a graph but the right hand side is a probability matrix. How is a 0-1 matrix reconstructed this way? \n\nThe overall method is competitive but does not outperform any of the other methods. The conclusion does not give any indication of how the performance could be improved. Is the issue related to independence assumptions? A discussion would be helpful.\n\nQUESTIONS:\nIn the motivating example there are 10 cities but the model uses 12 communities. How is the number of communities chosen? \n\nEquation 4: what do the wriggly lines mean? \n\nTheorem 5.1: can it not happen that X Y^T has entries taking the value 0, in which case H would give the value 1/2 and not 0 or 1? \n\nProof of Theorem 5.2: In my understanding a forest is a collection of trees. What is a partition into forests? It would be good to give a worked example in the appendix. How are the children found? Are the trees supposed to be rooted, choosing a root at random, and then orienting the tree from there? The proof seems to be constructive but it is not clear what the resulting matrices X and Y would be; the final step in the proof seems to be missing.\n\nLIMITATIONS:\nThere is no mention of societal impact. For example when assessing credit risk by assigning customers to communities of different risk, would extra caution be advised?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes and studies a community-based factorization model for exact representation of sparse networks. The authors extend a prior result on exact factorization which is based on logistic principal component analysis (LPCA), and show that an LPCA factorization can be converted to the proposed community-based graph factorization. The new factorization model explicitly captures both homophily and heterophily among nodes in the graph, and hence it is more interpretable. The authors carry out experiments using a different training algorithm to fit the model, and illustrate the advantage of the proposed decomposition method over existing ones for community detection and link prediction tasks.\n\nSTRENGTHS:\n- The paper provides an intuitive community-based graph factorization model that is able to explicitly capture both homophily and heterophily among nodes in the graph. The expressiveness of the new factorization is backed by a reduction from LPCA which is shown to be able to exactly represent any sparse graph with bounded arboricity. Although it may not be something of extremely high theoretical significance, I liked this aspect of technical development. It is an interesting result to have.\n\nWEAKNESSES:\n- There is a clear gap between the exact factorization studied up to Section 5 and the actual training algorithm evaluated in Section 6. Although the authors have mentioned the algorithmic gap explicitly at the end of Section 5. In order to well approximate the heavyside function which is used to achieve exact representation by LPCA, the input of the logit function should have entries that are arbitrarily large in magnitude. However, the training algorithm uses regularization to prevent this. The regularization does exactly the opposite of what is needed for exact representation. Therefore, there is a significant gap between theoretical development and empirical verifications.\n\n- Since the paper studies exact representation, there should be at least an experiment (e.g. the synthetic one) which shows that the reconstruction error goes to 0.\n\n- The overall writing and clarity can be improved. There are a few typos. For example, Line 32, principal components analysis -> principal component analysis. I also could not find where in the paper the term arboricity is clearly defined.\n\nQUESTIONS:\n- For the synthetic experiment shown in Figure 2, what happens if you increase the embedding length further? Will SVD beat your method afterwards? Can you make your method has close to 0 error (since it is supposed to be exact)?\n\n- What happens if you don't regularize your training algorithm for the experiments in Section 6, and how would the methods compare in that case?\n\nLIMITATIONS:\nI cannot find where the authors discuss the limitations or potential negative societal impact of their work, but I think it is fine.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to use the logistic PCA model to represent. Paper's main contribution is theoretical – improving the embedding dimensionality bounds from maximum degree to the arboricity of a graph. Overall, I believe the contribution is significant, but currently the paper does not highlight why too much – see more in the weaknesses section. The evaluation is severely limited, which might be forgivable for a more theory-oriented paper.\n\nSTRENGTHS:\n* Theoretical guarantees are interesting to (potentially) broader research community.\n* The methods offers grounded interpretability of node embeddings in terms of the community structure.\n* The paper is written clearly and should be easy to understand to the general audience.\n\nWEAKNESSES:\n* While the theoretical results are interesting, they are not well connected to the literature on either:\n ** Spanning trees and effective resistances and graph sparsification would be exciting to sparsification folks.\n ** Curvature (through bounds on Ollivier-Ricci curvature from effective resistance) would yield interesting connections to GNNs\n* The proposed algorithm is not really practical computationally speaking and does not scale to large graphs.\n* Experiments are not really indicative of real-world performance of the algorithm and rather focus on provide some free-form support for the theoretical claims of the paper.\n\nQUESTIONS:\n* Would node classification experiments on the datasets like POS/PPI/Blog (these are already present in the experimental section) strengthen the paper? Suggested comparison is some early neural embedding such as DeepWalk. This would open up interest from the subfield of more heuristic node embedding methods.\n\n* Can you detail the comparison with Chanpuriya et al., especially in terms of the proof?\n\nLIMITATIONS:\nYes; although I would stress limited practicality of the proposed algorithm due to its time complexity.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper concerns several main results/themes: \n\n- the authors show a theoretical result on a prior model known as LPCA. Their result indicates that exact factorizations for graphs under the LPCA model is possible under a bounded arboricity assumption, which is more generally applicable than the prior degree-based assumption.\n\n- the authors propose a graph generative model  that uses two non-negative vectors (one for homophily, one for heterophily) per node as well as a non-linear linking function to embed a given graph in an interpretable manner. This models generalizes the prior LPCA model, and has nice properties such as having factorizations that are non-linear, captures heterophily in addition to homophily, and is non-negative. \n\n- they show that the theoretical results that are available for LPCA (including the one that they derived) is applicable to their more general model as well. \n\n- they show promising experimental results on datasets.\n\nSTRENGTHS:\n- Originality: The two original contributions are mainly as follows. 1. the theoretical result on LPCA representations which uses a new arboricity criteria rather than a max degree criteria. 2. The proposed generative model (which is more general than LPCA). To the best of my knowledge, these two contributions have not been priorly proposed. I believe that the level of originality of this article, while not groundbreaking, is sufficient for publication at NeurIPS. \n\n- Quality: The quality of the experiments and the theoretical results are sound and technically solid, to the best of my knowledge. The results of the experiments supports the authors' claims that their proposed approach capture heterophily in an interpretable manner and that in specific cases it outperforms competitors. \n\n- Clarity: the paper is presented in a clear and easy to read manner, which is much appreciated. The authors motivated their results well. \n\n- Significance: Graph embeddings are certainly a relevant area of research nowadays. The authors' contributions have moderate impact, given their expressive, interpretable and more generally applicable model. It is significant enough for publication at NeurIPS.\n\nWEAKNESSES:\n- The authors covered virtually all the bases. Overall, I do not see any obvious shortcomings/omissions in the paper. I do have some comments/questions below: \n\n-  From a theoretical and originality perspective, the advances that they make over the prior result (the arboricity vs degree assumption) appears rather marginal (similar to Chanpuriya 2020). Yet a new result is a new result. I do not find this to disqualify this article from being accepted. \n\n- I find a small disconnect between the theoretical results and the proposed methods. The theory says that exact representations exist. The methods provide one way to find a representation. But there seems to be little linking the two: there seems to be no guarantee that what the algorithm actually finds in practice is the exact representation. It is possible that I might be missing something here, and any clarifications from the authors would be appreciated.\n\nQUESTIONS:\n- See above section\n\nLIMITATIONS:\nThe main limitations of these classes of methods is scalability and computational complexity. The authors briefly address this in the paper in A.3 for example by mentioning a stochastic training approach. I think the author's discussion of the limitations is sufficient but could be improved, for example, by discussing computational complexity (big Oh run time) in more explicit manners in both the deterministic and the stochastic case.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}