SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The upper and lower bounds are formally established by Theorems 2 and 3 in Appendix A, respectively, and are generalized to input graphs with directed edges and multiple edge types in Appendix D. Theorem 1 implies that, the  $(L - 1)$ -walk index of  $\mathcal{I}$  in graph prediction and its  $(L - 1,t)$ -walk index in vertex prediction control the separation rank with respect to  $\mathcal{I}$ , and are thus paramount for
Almost all in the sense of all weight assignments but a set of Lebesgue measure zero.
Low Walk Index
Low separation rank
High Walk Index
High separation rank
Figure 2: Depth  $L$  GNNs can model stronger interactions between sides of partitions that have a higher walk index (Definition 2). The partition  $(\mathcal{I}_1, \mathcal{I}_1^c)$  (left) divides the vertices into two separate cliques, connected by a single edge. Only two vertices reside in  $\mathcal{C}_{\mathcal{I}_1}$  — the set of vertices with an edge crossing the partition. Taking for example depth  $L = 3$ , the 2-walk index of  $\mathcal{I}_1$  is  $\Theta(|\mathcal{V}|^2)$  and its  $(2, t)$ -walk index is  $\Theta(|\mathcal{V}|)$ , for  $t \in \mathcal{V}$ . In contrast, the partition  $(\mathcal{I}_2, \mathcal{I}_2^c)$  (right) equally divides the vertices in each clique to different sides. All vertices reside in  $\mathcal{C}_{\mathcal{I}_2}$ , meaning the 2-walk index of  $\mathcal{I}_2$  is  $\Theta(|\mathcal{V}|^3)$  and its  $(2, t)$ -walk index is  $\Theta(|\mathcal{V}|^2)$ , for  $t \in \mathcal{V}$ . Hence, in both graph and vertex prediction scenarios, the walk index of  $\mathcal{I}_1$  is relatively low compared to that of  $\mathcal{I}_2$ . Our analysis (Section 4.1 and Appendix A) states that a higher separation rank can be attained with respect to  $\mathcal{I}_2$ , meaning stronger interaction can be modeled across  $(\mathcal{I}_2, \mathcal{I}_2^c)$  than across  $(\mathcal{I}_1, \mathcal{I}_1^c)$ . We empirically confirm this prospect in Section 4.2.
modeling interaction between  $\mathcal{I}$  and  $\mathcal{I}^c$  — see Figure 2 for an illustration. It thereby formalizes the conventional wisdom by which GNNs can model stronger interaction between areas of the input graph that are more interconnected. We support this finding empirically with common GNN architectures (e.g. GCN and GIN with ReLU non-linearity) in Section 4.2.
One may interpret Theorem 1 as encouraging addition of edges to an input graph. Indeed, the theorem states that such addition can enhance the GNN's ability to model interactions between input vertices. This accords with existing evidence by which increasing connectivity can improve the performance of GNNs in practice (see, e.g., [40, 1]). However, special care needs to be taken when adding edges: it may distort the semantic meaning of the input graph, and may lead to plights known as over-smoothing and over-squashing [68, 78, 22, 1, 8]. Rather than employing Theorem 1 for adding edges, we use it to select which edges to preserve in a setting where some must be removed. That is, we employ it for designing an edge sparsification algorithm. The algorithm, named Walk Index Sparsification (WIS), is simple, computationally efficient, and in our experiments has markedly outperformed alternative methods in terms of induced prediction accuracy. We present and evaluate it in Section 5.
4.2 Empirical Demonstration
Our theoretical analysis establishes that, the strength of interaction GNNs can model across a partition of input vertices is primarily determined by the partition's walk index — a graph-theoretical characteristic defined by the number of walks originating from the boundary of the partition (see Definition 2). The analysis formally applies to GNNs with product aggregation (see Section 3), yet we empirically demonstrate that its conclusions carry over to various other message-passing GNN architectures, namely GCN [59], GAT [96], and GIN [103] (with ReLU non-linearity). Specifically, through controlled experiments, we show that such models perform better on tasks in which the partitions that require strong interaction are ones with higher walk index, given that all other aspects of the tasks are the same. A description of these experiments follows. For brevity, we defer some implementation details to Appendix H.2.
We constructed two graph prediction datasets, in which the vertex features of each input graph are patches of pixels from two randomly sampled Fashion-MNIST [102] images, and the goal is to predict whether the two images are of the same class. In both datasets, all input graphs have the same structure: two separate cliques with 16 vertices each, connected by a single edge. The datasets differ in how the image patches are distributed among the vertices: in the first dataset each clique holds all the patches of a single image, whereas in the second dataset each clique holds half of the patches from the first image and half of the patches from the second image. Figure 2 illustrates how
Images are sampled such that the amount of positive and negative examples are roughly balanced.
Table 1: In accordance with our theory (Section 4.1 and Appendix A), GNNs can better fit datasets in which the partitions (of input vertices) that require strong interaction are ones with higher walk index (Definition 2). Table reports means and standard deviations, taken over five runs, of train and test accuracies obtained by GNNs of depth 3 and width 16 on two datasets: one in which the essential partition — i.e. the main partition requiring strong interaction — has low walk index, and another in which it has high walk index (see Section 4.2 for a detailed description of the datasets). For all GNNs, the train accuracy attained over the second dataset is considerably higher than that attained over the first dataset. Moreover, the better train accuracy translates to better test accuracy. See Appendix H.2 for further implementation details.
Essential Partition Walk Index
Low	High
GCN	Train Acc. (%)	70.4 ± 1.7	81.4 ± 2.0
Test Acc. (%)	52.7 ± 1.9	66.2 ± 1.1
GAT	Train Acc. (%)	82.8 ± 2.6	88.5 ± 1.1
Test Acc. (%)	69.6 ± 0.6	72.1 ± 1.2
GIN	Train Acc. (%)	83.2 ± 0.8	94.2 ± 0.8
Test Acc. (%)	53.7 ± 1.8	64.8 ± 1.4
image patches are distributed in the first (left hand side of the figure) and second (right hand side of the figure) datasets, with blue and red marking assignment of vertices to images.
Each dataset requires modeling strong interaction across the partition separating the two images, referred to as the essential partition of the dataset. In the first dataset the essential partition separates the two cliques, thus it has low walk index. In the second dataset each side of the essential partition contains half of the vertices from the first clique and half of the vertices from the second clique, thus the partition has high walk index. For an example illustrating the gap between these walk indices see Figure 2.
Table 1 reports train and test accuracies achieved by GCN, GAT, and GIN (with ReLU non-linearity) over both datasets. In compliance with our theory, the GNNs fit the dataset whose essential partition has high walk index significantly better than they fit the dataset whose essential partition has low walk index. Furthermore, the improved train accuracy translates to improvements in test accuracy.
5 Practical Application: Expressivity Preserving Edge Sparsification
Section 4 theoretically characterizes the ability of a GNN to model interactions between input vertices. It reveals that this ability is controlled by a graph-theoretical property we call walk index (Definition 2). The current section derives a practical application of our theory, specifically, an edge sparsification algorithm named Walk Index Sparsification (WIS), which preserves the ability of a GNN to model interactions when input edges are removed. We present WIs, and show that it yields high predictive performance for GNNs over standard vertex prediction benchmarks of various scales, even when removing a significant portion of edges. In particular, we evaluate WIs using GCN [59], GIN [103], and ResGCN [66] over multiple datasets, including: Cora [89], which contains thousands of edges, DBLP [16], which contains tens of thousands of edges, and OGBN-ArXiv [52], which contains more than a million edges. WIs is simple, computationally efficient, and in our experiments has markedly outperformed alternative methods in terms of prediction accuracy across edge sparsity levels. We believe its further empirical investigation is a promising direction for future research.
5.1 Walk Index Sparsification (WIS)
Running GNNs over large-scale graphs can be prohibitively expensive in terms of runtime and memory. A natural way to tackle this problem is edge sparsification — removing edges from an input graph while attempting to maintain prediction accuracy (cf. [67, 24]).<sup>9,10</sup>
As opposed to edge rewiring methods that add or remove only a few edges with the goal of improving prediction accuracy (e.g., [106, 71, 95, 8]).
An alternative approach is to remove vertices from an input graph (see, e.g., [60]). However, this approach is unsuitable for vertex prediction tasks, so we limit our attention to edge sparsification.
Algorithm 1  $(L - 1)$ -Walk Index Sparsification (WIS) (instance of a general scheme described in Appendix F)
Input:  $\mathcal{G}$  graph,  $L\in \mathbb{N}$  GNN depth,  $N\in \mathbb{N}$  number of edges to remove   
Result: Sparsified graph obtained by removing  $N$  edges from  $\mathcal{G}$    
for  $n = 1,\dots ,N$  do # per edge, compute walk indices of partitions induced by  $\{t\}$  , for  $t\in \mathcal{V}$  , after its removal for  $e\in \mathcal{E}$  (excluding self-loops) do initialize  $\mathbf{s}^{(e)} = (0,\ldots ,0)\in \mathbb{R}^{|\mathcal{V}|}$  remove  $e$  from  $\mathcal{G}$  (temporarily) for every  $t\in \mathcal{V}$  , set  $\mathbf{s}_t^{(e)} = \mathrm{WI}_{L - 1,t}(\{t\})$  # = number of length  $L - 1$  walks from  $\mathcal{C}_{\{t\}}$  to t add e back to  $\mathcal{G}$    
end for # prune edge whose removal harms walk indices the least according to an order over  $(s^{(e)})_{e\in \mathcal{E}}$  for  $e\in \mathcal{E}$  , sort the entries of  $\mathbf{s}^{(e)}$  in ascending order let  $e^{\prime}\in \operatorname {argmax}_{e\in \mathcal{E}}\mathbf{s}^{(e)}$  according to lexicographic order over tuples remove  $e^{\prime}$  from  $\mathcal{G}$  (permanently)   
end for
Algorithm 2 1-Walk Index Sparsification (WIS) (efficient version of Algorithm 1 for  $L = 2$ )
Input:  $\mathcal{G}$  graph,  $N\in \mathbb{N}$  number of edges to remove   
Result: Sparsified graph obtained by removing  $N$  edges from  $\mathcal{G}$    
for  $n = 1,\dots ,N$  do for  $\{i,j\} \in \mathcal{E}$  (excluding self-loops) do let  $\deg_{min}(i,j)\coloneqq \min \{|N(i)|,|\mathcal{N}(j)|\}$  let  $\deg_{max}(i,j)\coloneqq \max \{|N(i)|,|\mathcal{N}(j)|\}$  end for # prune edge  $\{i,j\} \in \mathcal{E}$  with maximal  $\deg_{min}(i,j)$  , breaking ties using  $\deg_{max}(i,j)$  let  $e^{\prime}\in \mathrm{argmax}_{\{i,j\} \in \mathcal{E}}\bigl (\deg_{min}(i,j),\deg_{max}(i,j)\bigr)$  according to lexicographic order over pairs remove  $e^{\prime}$  from  $\mathcal{G}$    
end for
Our theory (Section 4) establishes that, the strength of interaction a depth  $L$  GNN can model across a partition of input vertices is determined by the partition's walk index, a quantity defined by the number of length  $L - 1$  walks originating from the partition's boundary. This brings forth a recipe for pruning edges. First, choose partitions across which the ability to model interactions is to be preserved. Then, for every input edge (excluding self-loops), compute a tuple holding what the walk indices of the chosen partitions will be if the edge is to be removed. Lastly, remove the edge whose tuple is maximal according to a preselected order over tuples (e.g. an order based on the sum, min, or max of a tuple's entries). This process repeats until the desired number of edges are removed. The idea behind the above-described recipe, which we call General Walk Index Sparsification, is that each iteration greedily prunes the edge whose removal takes the smallest toll in terms of ability to model interactions across chosen partitions — see Algorithm 3 in Appendix F for a formal outline. Below we describe a specific instantiation of the recipe for vertex prediction tasks, which are particularly relevant with large-scale graphs, yielding our proposed algorithm — Walk Index Sparsification (WIS). Exploration of other instantiations is regarded as a promising avenue for future work.
In vertex prediction tasks, the interaction between an input vertex and the remainder of the input graph is of central importance. Thus, it is natural to choose the partitions induced by singletons (i.e. the partitions  $(\{t\}, \mathcal{V} \setminus \{t\})$ , where  $t \in \mathcal{V}$ ) as those across which the ability to model interactions is to be preserved. We would like to remove edges while avoiding a significant deterioration in the ability to model interaction under any of the chosen partitions. To that end, we compare walk index tuples according to their minimal entries, breaking ties using the second smallest entries, and so
Figure 3: Comparison of GNN accuracies following sparsification of input edges — WIS, the edge sparsification algorithm brought forth by our theory (Algorithm 1), markedly outperforms alternative methods. Plots present test accuracies achieved by a depth  $L = 3$  GCN of width 64 over the Cora (left), DBLP (middle), and OGBN-ArXiv (right) vertex prediction datasets, with increasing percentage of removed edges (for each combination of dataset, edge sparsification algorithm, and percentage of removed edges, a separate GCN was trained and evaluated). WIS, designed to maintain the ability of a GNN to model interactions between input vertices, is compared against: (i) removing edges uniformly at random; (ii) a spectral sparsification method [93]; and (iii) an adaptation of UGS [24]. For Cora, we run both 2-WIS, which is compatible with the GNN's depth, and 1-WIS, which can be viewed as an approximation that admits a particularly efficient implementation (Algorithm 2). For DBLP and OGBN-ArXiv, due to their larger scale only 1-WIS is evaluated. Markers and error bars report means and standard deviations, respectively, taken over ten runs per configuration. Note that 1-WIS achieves results similar to 2-WIS, suggesting that the efficiency it brings does not come at a significant cost in performance. Appendix H provides further implementation details and experiments with additional GNN architectures (GIN and ResGCN) and datasets (Chameleon, Squirrel, and Amazon Computers). Code for reproducing the experiment is available at https://github.com/noamrazin/gnn_interactions.
forth. This is equivalent to sorting (in ascending order) the entries of each tuple separately, and then ordering the tuples lexicographically.
Algorithm 1 provides a self-contained description of the method attained by the foregoing choices. We refer to this method as  $(L - 1)$ -Walk Index Sparsification (WIS), where the " $(L - 1)$ " indicates that only walks of length  $L - 1$  take part in the walk indices. Since  $(L - 1)$ -walk indices can be computed by taking the  $(L - 1)$ 'th power of the graph's adjacency matrix,  $(L - 1)$ -WIS runs in  $\mathcal{O}(N|\mathcal{E}||\mathcal{V}|^3\log (L))$  time and requires  $\mathcal{O}(|\mathcal{E}||\mathcal{V}| + |\mathcal{V}|^2)$  memory, where  $N$  is the number of edges to be removed. For large graphs a runtime cubic in the number of vertices can be restrictive. Fortunately, 1-WIS, which can be viewed as an approximation for  $(L - 1)$ -WIS with  $L > 2$ , facilitates a particularly simple and efficient implementation based solely on vertex degrees, requiring only linear time and memory — see Algorithm 2 (whose equivalence to 1-WIS is explained in Appendix G). Specifically, 1-WIS runs in  $\mathcal{O}(N|\mathcal{E}| + |\mathcal{V}|)$  time and requires  $\mathcal{O}(|\mathcal{E}| + |\mathcal{V}|)$  memory.
5.2 Empirical Evaluation
Below is an empirical evaluation of WIS. For brevity, we defer to Appendix H some implementation details, as well as experiments with additional GNN architectures (GIN and ResGCN) and datasets (Chameleon [82], Squirrel [82], and Amazon Computers [92]).
Using depth  $L = 3$  GNNs (with ReLU non-linearity), we evaluate over the Cora dataset both 2-WIS, which is compatible with the GNNs' depth, and 1-WIS, which can be viewed as an efficient approximation. Over the DBLP and OGBN-ArXiv datasets, due to their larger scale only 1-WIS is evaluated. Figure 3 (and Figure 8 in Appendix H) shows that WIS significantly outperforms the following alternative methods in terms of induced prediction accuracy: (i) a baseline in which edges are removed uniformly at random; (ii) a well-known spectral algorithm [93] designed to preserve the spectrum of the sparsified graph's Laplacian; and (iii) an adaptation of UGS [24] — a recent supervised approach for learning to prune edges.[11] Both 2-WIS and 1-WIS lead to higher test accuracies, while (as opposed to UGS) avoiding the need for labels, and for training a GNN over the original (non-sparsified) graph — a procedure which in some settings is prohibitively expensive in terms of runtime and memory. Interestingly, 1-WIS performs similarly to 2-WIS, indicating that the efficiency it brings does not come at a sizable cost in performance.
<sup>11</sup>UGS [24] jointly prunes input graph edges and GNN weights. For fair comparison, we adapt it to only remove edges.
6.1 Summary
GNNs are designed to model complex interactions between entities represented as vertices of a graph. The current paper provides the first theoretical analysis for their ability to do so. We proved that, given a partition of input vertices, the strength of interaction that can be modeled between its sides is controlled by the walk index — a graph-theoretical characteristic defined by the number of walks originating from the boundary of the partition. Experiments with common GNN architectures, e.g. GCN [59] and GIN [103], corroborated this result.
Our theory formalizes conventional wisdom by which GNNs can model stronger interaction between regions of the input graph that are more interconnected. More importantly, we showed that it facilitates a novel edge sparsification algorithm which preserves the ability of a GNN to model interactions when edges are removed. Our algorithm, named Walk Index Sparsification (WIS), is simple, computationally efficient, and in our experiments has markedly outperformed alternative methods in terms of induced prediction accuracy. More broadly, WIS showcases the potential of improving GNNs by theoretically analyzing the interactions they can model.
6.2 Limitations and Future Work
The theoretical analysis considers GNNs with product aggregation, which are less common in practice (cf. Section 3). We empirically demonstrated that its conclusions apply to more popular GNNs (Section 4.2), and derived a practical edge sparsification algorithm based on the theory (Section 5). Nonetheless, extending our analysis to additional aggregation functions is a worthy avenue to explore.
Our work also raises several interesting directions concerning WIS. A naive implementation of  $(L - 1)$ -WIS has runtime cubic in the number of vertices (cf. Section 5.1). Since this can be restrictive for large-scale graphs, the evaluation in Section 5.2 mostly focused on 1-WIS, which can be viewed as an efficient approximation of  $(L - 1)$ -WIS (its runtime and memory requirements are linear — see Section 5.1). Future work can develop efficient exact implementations of  $(L - 1)$ -WIS (e.g. using parallelization) and investigate regimes where it outperforms 1-WIS in terms of induced prediction accuracy. Additionally,  $(L - 1)$ -WIS is a specific instantiation of the general WIS scheme (given in Appendix F), tailored for preserving the ability to model interactions across certain partitions. Exploring other instantiations, as well as methods for automatically choosing the partitions across which the ability to model interactions is preserved, are valuable directions for further research.
We thank Eshbal Hezroni for aid in preparing illustrative figures. This work was supported by a Google Research Scholar Award, a Google Research Gift, the Yandex Initiative in Machine Learning, the Israel Science Foundation (grant 1780/21), the Tel Aviv University Center for AI and Data Science, the Adelis Research Fund for Artificial Intelligence, Len Blavatnik and the Blavatnik Family Foundation, and Amnon and Anat Shashua. NR is supported by the Apple Scholars in AI/ML PhD fellowship.
[1] Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. International Conference on Learning Representations (ICLR), 2021.
[2] Arash Amini, Amin Karbasi, and Farokh Marvasti. Low-rank matrix approximation using point-wise operators. IEEE transactions on information theory, 58(1):302-310, 2011.
[3] Waiss Azizian and Marc Lelarge. Expressive power of invariant and equivariant graph neural networks. International Conference on Learning Representations (ICLR), 2021.
[4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
A Formal Analysis: Quantifying the Ability of Graph Neural Networks to Model Interactions
We begin by upper bounding the separation ranks a GNN can achieve.
Theorem 2. For an undirected graph  $\mathcal{G}$  and  $t\in \mathcal{V}$ , let  $f^{(\theta ,\mathcal{G})}$  and  $f^{(\theta ,\mathcal{G},t)}$  be the functions realized by depth  $L$  graph and vertex prediction GNNs, respectively, with width  $D_{h}$ , learnable weights  $\theta$ , and product aggregation (Equations (2) to (5)). Then, for any  $\mathcal{I}\subseteq \mathcal{V}$  and assignment of weights  $\theta$  it holds that:
$$
\left(\operatorname {g r a p h} \text {p r e d i c t i o n}\right) \quad \log \left(\operatorname {s e p} \left(f ^ {\left(\theta , \mathcal {G}\right)}; \mathcal {I}\right)\right) \leq \log \left(D _ {h}\right) \cdot \left(4 \underbrace {\rho_ {L - 1} \left(\mathcal {C} _ {\mathcal {I}} , \mathcal {V}\right)} _ {\mathrm {W I} _ {L - 1} (\mathcal {I})} + 1\right), \tag {6}
$$
$$
\text {(v e r t e x p r e d i c t i o n)} \quad \log \left(\operatorname {s e p} \left(f ^ {\left(\theta , \mathcal {G}, t\right)}; \mathcal {I}\right)\right) \leq \log \left(D _ {h}\right) \cdot 4 \underbrace {\rho_ {L - 1} \left(\mathcal {C} _ {\mathcal {I}} , \{t \}\right)} _ {\mathrm {W I} _ {L - 1, t} (\mathcal {I})}. \tag {7}
$$
Proof sketch (proof in Appendix I.2). In Appendix E, we show that the computations performed by a GNN with product aggregation can be represented as a tensor network. In brief, a tensor network is a weighted graph that describes a sequence of arithmetic operations known as tensor contractions (see Appendices E.1 and E.2 for a self-contained introduction to tensor networks). The tensor network corresponding to a GNN with product aggregation adheres to a tree structure — its leaves are associated with input vertex features and interior nodes embody the operations performed by the GNN. Importing machinery from tensor analysis literature, we prove that  $\mathrm{sep}(f^{(\theta,\mathcal{G})};\mathcal{I})$  is upper bounded by a minimal cut weight in the corresponding tensor network, among cuts separating leaves associated with input vertices in  $\mathcal{I}$  from leaves associated with input vertices in  $\mathcal{I}^c$ . Equation (6) then follows by finding such a cut in the tensor network with sufficiently low weight. Equation (7) is established analogously.
A natural question is whether the upper bounds in Theorem 2 are tight, i.e. whether separation ranks close to them can be attained. We show that nearly matching lower bounds hold for almost all assignments of weights  $\theta$ . To this end, we define admissible subsets of  $\mathcal{C}_{\mathcal{I}}$ , based on a notion of vertex subsets with no repeating shared neighbors.