SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Aggarwal, C. C. and Wang, H. A survey of clustering algorithms for graph data. In Managing and Mining Graph Data, pp. 275-301. Springer, 2010.
Athreya, A., Fishkind, D. E., Tang, M., Priebe, C. E., Park, Y., Vogelstein, J. T., Levin, K., Lyzinski, V., and Qin, Y. Statistical inference on random dot product graphs: a survey. The Journal of Machine Learning Research, 18(1):8393-8484, 2017.
Berry, M. W., Browne, M., Langville, A. N., Pauca, V. P., and Plemmons, R. J. Algorithms and applications for approximate nonnegative matrix factorization. Computational Statistics & Data Analysis, 52(1):155-173, 2007.
Bhattacharjee, R. and Dasgupta, S. What relations are reliably embeddable in euclidean space? In Algorithmic Learning Theory, pp. 174-195. PMLR, 2020.
Bonato, A. A survey of models of the web graph. In Workshop on Combinatorial and Algorithmic Aspects of Networking, pp. 159-172. Springer, 2004.
Boratko, M., Zhang, D., Monath, N., Vilnis, L., Clarkson, K. L., and McCallum, A. Capacity and bias of learned geometric embeddings for directed graphs. Advances in Neural Information Processing Systems, 34:16423-16436, 2021.
Breitkreutz, B.-J., Stark, C., Reguly, T., Boucher, L., Breitkreutz, A., Livstone, M., Oughtred, R., Lackner, D. H., Bahler, J., Wood, V., et al. The biogrid interaction database: 2008 update. *Nucleic acids research*, 36(suppl_1):D637–D640, 2007.
Chanpuriya, S., Musco, C., Sotiropoulos, K., and Tsourakakis, C. Node embeddings and exact low-rank representations of complex networks. Advances in Neural Information Processing Systems, 33, 2020.
Ding, C., Li, T., and Jordan, M. I. Nonnegative matrix factorization for combinatorial optimization: Spectral clustering, graph matching, and clique finding. In 2008 Eighth IEEE International Conference on Data Mining, pp. 183-192. IEEE, 2008.
Donoho, D. L. and Stodden, V. When does non-negative matrix factorization give a correct decomposition into parts? In Advances in Neural Information Processing Systems 16, pp. 1141-1148. MIT Press, 2003.
Duvenaud, D., Maclaurin, D., Aguilera-Iparraguirre, J., Gomez-Bombarelli, R., Hirzel, T., Aspuru-Guzik, A., and Adams, R. P. Convolutional networks on graphs for learning molecular fingerprints. In Advances in Neural Information Processing Systems 28, pp. 2224-2232, 2015.
Foster, J. G., Foster, D. V., Grassberger, P., and Paczuski, M. Edge direction and the structure of networks. Proceedings of the National Academy of Sciences, 107(24):10815-10820, 2010.
Gillis, N. Nonnegative Matrix Factorization. SIAM, 2020.
Hamilton, W., Ying, Z., and Leskovec, J. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Hase, T., Niimura, Y., and Tanaka, H. Difference in gene duplicability may explain the difference in overall structure of protein-protein interaction networks among eukaryotes. BMC Evolutionary Biology, 10(1):1-15, 2010.
Javed, M. A., Younis, M. S., Latif, S., Qadir, J., and Baig, A. Community detection in networks: A multidisciplinary review. Journal of Network and Computer Applications, 108:87-111, 2018.
Zheng, X., Liu, Y., Pan, S., Zhang, M., Jin, D., and Yu, P. S. Graph neural networks for graphs with heterophily: A survey. arXiv:2202.07082, 2022.
Zhu, C., Byrd, R. H., Lu, P., and Nocedal, J. Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Transactions on Mathematical Software (TOMS), 23(4):550-560, 1997.
Zhu, J., Yan, Y., Zhao, L., Heimann, M., Akoglu, L., and Koutra, D. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, 33, 2020.
Zweig, K. A. Are word-adjacency networks networks? In Towards a theoretical framework for analyzing complex linguistic networks, pp. 153–163. Springer, 2016.
A.1 COT Graph Exact Representation
As a theoretical demonstration of the capability of our model to learn latent structure, we additionally show that our model can exactly represent a natural family of graphs, which exhibits both homophily and heterophily, with small  $k$  and interpretably. The family of graphs is specified below in Definition 1; roughly speaking, nodes in such graphs share an edge iff they coparticipate in some number of homophilous communities and don't coparticipate in a number of heterophilous communities. For example, the motivating graph described in Section 2 would be an instance of such a graph if an edge occurs between two users iff the two users are from the same city and have different genders.
Definition 1 (Community Overlap Threshold (COT) Graph). An unweighted, undirected graph whose edges are determined by an overlapping clustering and a "thresholding" integer  $t \in \mathbb{Z}$  as follows: for each vertex  $i$ , there are two latent binary vectors  $\pmb{b}_i \in \{0,1\}^{k_b}$  and  $\pmb{c}_i \in \{0,1\}^{k_c}$ , and there is an edge between vertices  $i$  and  $j$  iff  $\pmb{b}_i \cdot \pmb{b}_j - \pmb{c}_i \cdot \pmb{c}_j \geq t$ .
Theorem A.1 (Compact Representation of COT Graphs). Suppose  $\mathbf{A}$  is the adjacency matrix of a COT graph on  $n$  nodes with latent vectors  $\pmb{b}_i\in \{0,1\}^{k_b}$  and  $\pmb {c}_i\in \{0,1\}^{k_c}$  for  $i\in \{1,2,\dots ,n\}$ . Let  $k = k_{b} + k_{c}$ . Then, for any  $\epsilon >0$ , there exist  $\pmb {V}\in [0,1]^{n\times (k + 1)}$  and diagonal  $\pmb {W}\in \mathbb{R}^{(k + 1)\times (k + 1)}$  such that  $\| \sigma (\pmb {V}\pmb {W}\pmb {V}^{\top}) - \pmb {A}\| _F < \epsilon$ .
Proof. Let  $t$  be the thresholding integer of the graph, and let the rows of  $B \in \{0,1\}^{n \times k_b}$  and  $C \in \{0,1\}^{n \times k_c}$  contain the vectors  $b$  and  $c$  of all nodes. Via Equation 2, we can find  $V^* \in [0,1]^{n \times k}$  and diagonal  $W^* \in \mathbb{R}^{k \times k}$  such that  $V^* W^* V^{*\top} = BB^{\top} - CC^{\top}$ . Now let
$$
\boldsymbol {V} = (\boldsymbol {V} ^ {*} \quad \boldsymbol {1}) \qquad \boldsymbol {W} = \left( \begin{array}{c c} \boldsymbol {W} ^ {*} & 0 \\ 0 & \frac {1}{2} - t \end{array} \right).
$$
Then  $(\boldsymbol{V}\boldsymbol{W}\boldsymbol{V}^{\top})_{ij} = \boldsymbol{b}_i \cdot \boldsymbol{b}_j - \boldsymbol{c}_i \cdot \boldsymbol{c}_j + \frac{1}{2} - t$ . Hence  $(\boldsymbol{V}\boldsymbol{W}\boldsymbol{V}^{\top})_{ij} > 0$  iff  $\boldsymbol{b}_i \cdot \boldsymbol{b}_j - \boldsymbol{c}_i \cdot \boldsymbol{c}_j > t - \frac{1}{2}$ , which is true iff  $\boldsymbol{A}_{ij} = 1$  by the assumption on the graph. Similarly,  $(\boldsymbol{V}\boldsymbol{W}\boldsymbol{V}^{\top})_{ij} < 0$  iff  $\boldsymbol{A}_{ij} = 0$ . It follows that
$$
\lim  _ {s \rightarrow \infty} \sigma (V (s W) V ^ {\top}) = \lim  _ {s \rightarrow \infty} \sigma (s V W W ^ {\top}) = A.
$$
A.2 Application: Tabular Data Completion
We apply our link prediction algorithm to the task of data completion for categorical tabular data. We first transform such data to a graph as follows: We create a node for each row, as well as a node for each unique category for each of the columns. For each entry of the table, a link occurs between the node for the row and the node for the value in the entry, which is one of the possible categories of the column (e.g., if Bob is American, then a link occurs between the row node for 'Bob' and the category node for 'American'). No links occur between two nodes for rows, or between two nodes for category values. For the negative samples, we select only edges between nodes for rows and nodes for category values; the negative samples would otherwise be dominated by pairs of nodes for rows.
We use two proprietary company datasets datasets which we call COMPANY A and COMPANY B. We use the same experimental setup as described in Section 6.3. Unlike in Section 6.3, we have a restriction on the possible links: each row node must be linked to exactly one of the value nodes for each of the categories. Hence, for each combination of row node and column, we predict a link only with the value node with the highest predicted link probability. We set the number of communities for all methods to 20 and 50 for COMPANY A and COMPANY B, respectively. We employ the stochastic version of our algorithm, based on projected SGD, for which code is also provided. We generally use a batch size of 100; we find that the optimization of BIGCLAM often diverges on COMPANY B with this batch size, so we instead use batches of size 1000 for its optimization. The results are provided in Figure 6. Notably, our proposed approach outperforms comparable (community-based factorization) methods in terms of accuracy. As a baseline, we also show the accuracy when completing the tables by simply selecting the most common categorical value for each column ("PLURALITY"). All methods outperform this baseline on both datasets.
Figure 6: Test accuracy of tabular data completion on two proprietary company datasets (COMPANY A and COMPANY B).