Title: On the Ability of Graph Neural Networks to Model Interactions Between Vertices
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph Neural Networks

Expressivity

Interactions

Edge Sparsification
Abstract: Graph neural networks (GNNs) are widely used for modeling complex interactions between entities represented as vertices of a graph. Despite recent efforts to theoretically analyze the expressive power of GNNs, a formal characterization of their ability to model interactions is lacking. The current paper aims to address this gap. Formalizing strength of interactions through an established measure known as separation rank, we quantify the ability of certain GNNs to model interaction between a given subset of vertices and its complement, i.e. between the sides of a given partition of input vertices. Our results reveal that the ability to model interaction is primarily determined by the partition's walk index --- a graph-theoretical characteristic defined by the number of walks originating from the boundary of the partition. Experiments with common GNN architectures corroborate this finding. As a practical application of our theory, we design an edge sparsification algorithm named Walk Index Sparsification (WIS), which preserves the ability of a GNN to model interactions when input edges are removed. WIS is simple, computationally efficient, and in our experiments has markedly outperformed alternative methods in terms of induced prediction accuracy. More broadly, it showcases the potential of improving GNNs by theoretically analyzing the interactions they can model.

On the Ability of Graph Neural Networks to Model Interactions Between Vertices

Abstract

Graph neural networks (GNNs) are widely used for modeling complex interactions between entities represented as vertices of a graph. Despite recent efforts to theoretically analyze the expressive power of GNNs, a formal characterization of their ability to model interactions is lacking. The current paper aims to address this gap. Formalizing strength of interactions through an established measure known as separation rank , we quantify the ability of certain GNNs to model interaction between a given subset of vertices and its complement, i.e. between the sides of a given partition of input vertices. Our results reveal that the ability to model interaction is primarily determined by the partition's walk index -a graph-theoretical characteristic defined by the number of walks originating from the boundary of the partition. Experiments with common GNN architectures corroborate this finding. As a practical application of our theory, we design an edge sparsification algorithm named Walk Index Sparsification ( WIS ), which preserves the ability of a GNN to model interactions when input edges are removed. WIS is simple, computationally efficient, and in our experiments has markedly outperformed alternative methods in terms of induced prediction accuracy. 1 More broadly, it showcases the potential of improving GNNs by theoretically analyzing the interactions they can model.

1 Introduction

Graph neural networks ( GNNs ) are a family of deep learning architectures, designed to model complex interactions between entities represented as vertices of a graph. In recent years, GNNs have been successfully applied across a wide range of domains, including social networks, biochemistry, and recommender systems (see, e.g. , [36, 59, 45, 49, 96, 104, 101, 18]). Consequently, significant interest in developing a mathematical theory behind GNNs has arisen.

One of the fundamental questions a theory of GNNs should address is expressivity , which concerns the class of functions a given architecture can realize. Existing studies of expressivity largely fall into three categories. First, and most prominent, are characterizations of ability to distinguish nonisomorphic graphs [103, 74, 72, 70, 6, 15, 10, 17, 43, 42, 80], as measured by equivalence to classical Weisfeiler-Leman graph isomorphism tests [99]. Second, are proofs for universal approximation of continuous permutation invariant or equivariant functions, possibly up to limitations in distinguishing some classes of graphs [73, 55, 25, 69, 3, 42]. Last, are works examining specific properties of GNNs such as frequency response [77, 5] or computability of certain graph attributes, e.g. moments, shortest paths, and substructure multiplicity [35, 9, 26, 39, 69, 23, 17, 105].

A major drawback of many existing approaches - in particular proofs of equivalence to WeisfeilerLeman tests and those of universality - is that they operate in asymptotic regimes of unbounded

example: length two walks

Example: length two walks

walk #1

walk #2

walk #48

can model between     and

Interaction depth     GNN

-walk index of     :

walks from

# length

network width or depth. Moreover, to the best of our knowledge, none of the existing approaches formally characterize the strength of interactions GNNs can model between vertices, and how that depends on the structure of the input graph and the architecture of the neural network.

The current paper addresses the foregoing gaps. Namely, it theoretically quantifies the ability of fixed-size GNNs to model interactions between vertices, delineating the impact of the input graph structure and the neural network architecture (width and depth). Strength of modeled interactions is formalized via separation rank [12] - a commonly used measure for the interaction a function models between a subset of input variables and its complement (the rest of the input variables). Given a function and a partition of its input variables, the higher the separation rank, the more interaction the function models between the sides of the partition. Separation rank is prevalent in quantum mechanics, where it can be viewed as a measure of entanglement [62]. It was previously used for analyzing variants of convolutional, recurrent, and self-attention neural networks, yielding both theoretical insights and practical tools [30, 33, 61, 62, 64, 100, 65, 85]. We employ it for studying GNNs.

Key to our theory is a widely studied correspondence between neural networks with polynomial nonlinearity and tensor networks 2 [32, 29, 30, 34, 90, 61, 62, 7, 56, 57, 63, 64, 83, 100, 84, 85, 65]. We extend this correspondence, and use it to analyze message-passing GNNs with product aggregation. We treat both graph prediction, where a single output is produced for an entire input graph, and vertex prediction, in which the network produces an output for every vertex. For graph prediction, we prove that the separation rank of a depth L GNN with respect to a partition of vertices is primarily determined by the partition's ( L -1) -walk index -a graph-theoretical characteristic defined to be the number of length L -1 walks originating from vertices with an edge crossing the partition. The same holds for vertex prediction, except that there walk index is defined while only considering walks ending at the target vertex. Our result, illustrated in Figure 1, implies that for a given input graph, the ability of GNNs to model interaction between a subset of vertices I and its complement I c , predominantly depends on the number of walks originating from the boundary between I and I c . We corroborate this proposition through experiments with standard GNN architectures, such as Graph Convolutional Network (GCN) [59] and Graph Isomorphism Network (GIN) [103].

Our theory formalizes conventional wisdom by which GNNs can model stronger interaction between regions of the input graph that are more interconnected. More importantly, we show that it facilitates an edge sparsification algorithm that preserves the expressive power of GNNs (in terms of ability to model interactions). Edge sparsification concerns removal of edges from a graph for reducing computational and/or memory costs, while attempting to maintain selected properties of the graph ( cf. [11, 93, 48, 20, 86, 98, 67, 24]). In the context of GNNs, our interest lies in maintaining prediction accuracy as the number of edges removed from the input graph increases. We propose an algorithm for removing edges, guided by our separation rank characterization. The algorithm, named Walk Index Sparsification ( WIS ), is demonstrated to yield high predictive performance for GNNs ( e.g. GCN and GIN) over standard benchmarks of various scales, even when removing a significant portion of

edges. WIS is simple, computationally efficient, and in our experiments has markedly outperformed alternative methods in terms of induced prediction accuracies across edge sparsity levels. More broadly, WIS showcases the potential of improving GNNs by theoretically analyzing the interactions they can model, and we believe its further empirical investigation is a promising direction for future research.

The remainder of the paper is organized as follows. Section 2 introduces notation and the concept of separation rank. Section 3 presents the theoretically analyzed GNN architecture. Section 4 theoretically quantifies (via separation rank) its ability to model interactions between vertices of an input graph. Section 5 proposes and evaluates WIS - an edge sparsification algorithm for arbitrary GNNs, born from our theory. Lastly, Section 6 concludes. Related work is discussed throughout, and for the reader's convenience, is recapitulated in Appendix B.

2 Preliminaries

2.1 Notation

For N ∈ N , let [ N ] := { 1 , . . . , N } . We consider an undirected input graph G = ( V , E ) with vertices V = [ |V| ] and edges E ⊆ {{ i, j } : i, j ∈ V} . Vertices are equipped with features X := ( x (1) , . . . , x ( |V| ) ) ∈ R D x ×|V| -one D x -dimensional feature vector per vertex ( D x ∈ N ). For i ∈ V , we use N ( i ) := { j ∈ V : { i, j } ∈ E} to denote its set of neighbors, and, as customary in the context of GNNs, assume the existence of all self-loops, i.e. i ∈ N ( i ) for all i ∈ V ( cf. [59, 50]). Furthermore, for I ⊆ V we let N ( I ) := ∪ i ∈I N ( i ) be the neighbors of vertices in I , and I c := V \ I be the complement of I . Weuse C I to denote the boundary of the partition ( I , I c ) , i.e. the set of vertices with an edge crossing the partition, defined by C I := { i ∈ I : N ( i ) ∩I c = ∅}∪{ j ∈ I c : N ( j ) ∩I glyph[negationslash] = ∅} . 3 Lastly, we denote the number of length l ∈ N ≥ 0 walks from any vertex in I ⊆ V to any vertex in J ⊆ V by ρ l ( I , J ) . 4 In particular, ρ l ( I , J ) = ∑ i ∈I ,j ∈J ρ l ( { i } , { j } ) .

glyph[negationslash]

Note that we focus on undirected graphs for simplicity of presentation. As discussed in Section 4, our results are extended to directed graphs in Appendix D.

2.2 Separation Rank: A Measure of Modeled Interaction

A prominent measure quantifying the interaction a multivariate function models between a subset of input variables and its complement ( i.e. all other variables) is known as separation rank . The separation rank was introduced in [12], and has since been employed for various applications [51, 47, 13]. It is also a common measure of entanglement , a profound concept in quantum physics quantifying interaction between particles [62]. In the context of deep learning, it enabled analyses of expressivity and generalization in certain convolutional, recurrent, and self-attention neural networks, resulting in theoretical insights and practical methods (guidelines for neural architecture design, pretraining schemes, and regularizers - see [30, 33, 61, 62, 64, 100, 65, 85]).

Given a multivariate function f : ( R D x ) N → R , its separation rank with respect to a subset of input variables I ⊆ [ N ] is the minimal number of summands required to express it, where each summand is a product of two functions - one that operates over variables indexed by I , and another that operates over the remaining variables. Formally:

Definition 1. The separation rank of f : ( R D x ) N → R with respect to I ⊆ [ N ]

where X := ( x (1) , . . . , x ( N ) ) , X I := ( x ( i ) ) i ∈I , and X I c := ( x ( j ) ) j ∈I c . By convention, if f is identically zero then sep( f ; I ) = 0 , and if the set on the right hand side of Equation (1) is empty then sep( f ; I ) = ∞ .

Interpretation If sep( f ; I ) = 1 , the function is separable, meaning it does not model any interaction between X I and X I c , i.e. between the sides of the partition ( I , I c ) . Specifically, it can be represented as f ( X ) = g ( X I ) · ¯ g ( X I c ) for some functions g and ¯ g . In a statistical setting, where f is a probability density function, this would mean that X I and X I c are statistically independent. The higher sep( f ; I ) is, the farther f is from separability, implying stronger modeling of interaction between X I and X I c .

3 Graph Neural Networks

Modern GNNs predominantly follow the message-passing paradigm [45, 50], whereby each vertex is associated with a hidden embedding that is updated according to its neighbors. The initial embedding of i ∈ V is taken to be its input features: h (0 ,i ) := x ( i ) ∈ R D x . Then, in a depth L message-passing GNN, a common update scheme for the hidden embedding of i ∈ V at layer l ∈ [ L ] is:

where { {·} } denotes a multiset, W (1) ∈ R D h × D x , W (2) ∈ R D h × D h , . . . , W ( L ) ∈ R D h × D h are learnable weight matrices, with D h ∈ N being the network's width ( i.e. hidden dimension), and AGGREGATE is a function combining multiple input vectors into a single vector. A notable special case is GCN [59], in which AGGREGATE performs a weighted average followed by a non-linear activation function ( e.g. ReLU). 5 Other aggregation operators are also viable, e.g. element-wise sum, max, or product ( cf. [49, 53]). We note that distinguishing self-loops from other edges, and more generally, treating multiple edge types, is possible through the use of different weight matrices for different edge types [49, 88]. For conciseness, we hereinafter focus on the case of a single edge type, and treat multiple edge types in Appendix D.

After L layers, the GNN generates hidden embeddings h ( L, 1) , . . . , h ( L, |V| ) ∈ R D h . For graph prediction, where a single output is produced for the whole graph, the hidden embeddings are usually combined into a single vector through the AGGREGATE function. A final linear layer with weights W ( o ) ∈ R 1 × D h is then applied to the resulting vector. 6 Overall, the function realized by a depth L graph prediction GNN receives an input graph G with vertex features X := ( x (1) , . . . , x ( |V| ) ) ∈ R D x ×|V| , and returns:

with θ := ( W (1) , . . . , W ( L ) , W ( o ) ) denoting the network's learnable weights. For vertex prediction tasks, where the network produces an output for every t ∈ V , the final linear layer is applied to each h ( L,t ) separately. That is, for a target vertex t ∈ V , the function realized by a depth L vertex prediction GNN is given by:

Our aim is to investigate the ability of GNNs to model interactions between vertices. Prior studies of interactions modeled by different deep learning architectures have focused on neural networks with polynomial non-linearity, building on their representation as tensor networks [32, 30, 34, 90, 61, 62, 7, 56, 63, 64, 83, 100, 84, 85, 65]. Although neural networks with polynomial non-linearity are less common in practice, they have demonstrated competitive performance [28, 31, 91, 94, 27, 37, 53], and hold promise due to their compatibility with quantum computation [46, 14] and fully homomorphic encryption [44]. More importantly, their analyses brought forth numerous insights that were demonstrated empirically and led to development of practical tools for widespread deep learning models (with non-linearities such as ReLU).

Following the above, in our theoretical analysis (Section 4) we consider GNNs with (element-wise) product aggregation, which are polynomial functions of their inputs. Namely, the AGGREGATE operator from Equations (2) and (3) is taken to be:

where glyph[circledot] stands for the Hadamard product and X is a multiset of vectors. The resulting architecture can be viewed as a variant of the GNN proposed in [53], where it was shown to achieve competitive performance in practice. Central to our proofs are tensor network representations of GNNs with product aggregation (formally established in Appendix E), analogous to those used for analyzing other types of neural networks. We empirically demonstrate our theoretical findings on popular GNNs (Section 4.2), such as GCN and GIN with ReLU non-linearity, and use them to derive a practical edge sparsification algorithm (Section 5).

We note that some of the aforementioned analyses of neural networks with polynomial non-linearity were extended to account for additional non-linearities, including ReLU, through constructs known as generalized tensor networks [29]. We thus believe our theory may be similarly extended, and regard this as an interesting direction for future work.

4 Theoretical Analysis: The Effect of Input Graph Structure and Neural Network Architecture on Modeled Interactions

In this section, we employ separation rank (Definition 1) to theoretically quantify how the input graph structure and network architecture (width and depth) affect the ability of a GNN with product aggregation to model interactions between input vertices. We overview the main results and their implications in Section 4.1, while deferring the formal analysis to Appendix A due to lack of space. Section 4.2 provides experiments demonstrating our theory's implications on common GNNs, such as GCN and GIN with ReLU non-linearity.

4.1 Overview and Implications

Consider a depth L GNN with width D h and product aggregation (Section 3). Given a graph G , any assignment to the weights of the network θ induces a multivariate function f ( θ, G ) for graph prediction (Equation (3)) and f ( θ, G ,t ) for prediction over a given vertex t ∈ V (Equation (4)) - whose variables correspond to feature vectors of input vertices. The separation rank of this function with respect to I ⊆ V thus measures the interaction modeled across the partition ( I , I c ) , i.e. between the vertices in I and those in I c . The higher the separation rank is, the stronger the modeled interaction.

Key to our analysis are the following notions of walk index , defined by the number of walks emanating from the boundary of the partition ( I , I c ) , i.e. from vertices with an edge crossing the partition induced by I (see Figure 1 for an illustration).

glyph[negationslash]

Definition 2. Let I ⊆ V . Denote by C I the set of vertices with an edge crossing the partition ( I , I c ) , i.e. C I := { i ∈ I : N ( i ) ∩ I c = ∅} ∪ { j ∈ I c : N ( j ) ∩ I glyph[negationslash] = ∅} , and recall that ρ l ( C I , J ) denotes the number of length l ∈ N ≥ 0 walks from any vertex in C I to any vertex in J ⊆ V . For L ∈ N :

(graph prediction) we define the ( L -1) -walk index of I , denoted WI L -1 ( I ) , to be the number of length L -1 walks originating from C I , i.e. WI L -1 ( I ) := ρ L -1 ( C I , V ) ; and

(vertex prediction) for t ∈ V we define the ( L -1 , t ) -walk index of I , denoted WI L -1 ,t ( I ) , to be the number of length L -1 walks from C I that end at t , i.e. WI L -1 ,t ( I ) := ρ L -1 ( C I , { t } ) .

As our main theoretical contribution, we prove:

Theorem 1 (informally stated) . For all weight assignments θ and t ∈ V :

Moreover, nearly matching lower bounds hold for almost all weight assignments. 7

The upper and lower bounds are formally established by Theorems 2 and 3 in Appendix A, respectively, and are generalized to input graphs with directed edges and multiple edge types in Appendix D. Theorem 1 implies that, the ( L -1) -walk index of I in graph prediction and its ( L -1 , t ) -walk index in vertex prediction control the separation rank with respect to I , and are thus paramount for

Low Walk Index

High Walk Index

Low separation rank

High separation rank

modeling interaction between I and I c -see Figure 2 for an illustration. It thereby formalizes the conventional wisdom by which GNNs can model stronger interaction between areas of the input graph that are more interconnected. We support this finding empirically with common GNN architectures ( e.g. GCN and GIN with ReLU non-linearity) in Section 4.2.

One may interpret Theorem 1 as encouraging addition of edges to an input graph. Indeed, the theorem states that such addition can enhance the GNN's ability to model interactions between input vertices. This accords with existing evidence by which increasing connectivity can improve the performance of GNNs in practice (see, e.g. , [40, 1]). However, special care needs to be taken when adding edges: it may distort the semantic meaning of the input graph, and may lead to plights known as over-smoothing and over-squashing [68, 78, 22, 1, 8]. Rather than employing Theorem 1 for adding edges, we use it to select which edges to preserve in a setting where some must be removed. That is, we employ it for designing an edge sparsification algorithm. The algorithm, named Walk Index Sparsification ( WIS ), is simple, computationally efficient, and in our experiments has markedly outperformed alternative methods in terms of induced prediction accuracy. We present and evaluate it in Section 5.

4.2 Empirical Demonstration

Our theoretical analysis establishes that, the strength of interaction GNNs can model across a partition of input vertices is primarily determined by the partition's walk index - a graph-theoretical characteristic defined by the number of walks originating from the boundary of the partition (see Definition 2). The analysis formally applies to GNNs with product aggregation (see Section 3), yet we empirically demonstrate that its conclusions carry over to various other message-passing GNN architectures, namely GCN [59], GAT [96], and GIN [103] (with ReLU non-linearity). Specifically, through controlled experiments, we show that such models perform better on tasks in which the partitions that require strong interaction are ones with higher walk index, given that all other aspects of the tasks are the same. A description of these experiments follows. For brevity, we defer some implementation details to Appendix H.2.

We constructed two graph prediction datasets, in which the vertex features of each input graph are patches of pixels from two randomly sampled Fashion-MNIST [102] images, and the goal is to predict whether the two images are of the same class. 8 In both datasets, all input graphs have the same structure: two separate cliques with 16 vertices each, connected by a single edge. The datasets differ in how the image patches are distributed among the vertices: in the first dataset each clique holds all the patches of a single image, whereas in the second dataset each clique holds half of the patches from the first image and half of the patches from the second image. Figure 2 illustrates how

image patches are distributed in the first (left hand side of the figure) and second (right hand side of the figure) datasets, with blue and red marking assignment of vertices to images.

Each dataset requires modeling strong interaction across the partition separating the two images, referred to as the essential partition of the dataset. In the first dataset the essential partition separates the two cliques, thus it has low walk index. In the second dataset each side of the essential partition contains half of the vertices from the first clique and half of the vertices from the second clique, thus the partition has high walk index. For an example illustrating the gap between these walk indices see Figure 2.

Table 1 reports train and test accuracies achieved by GCN, GAT, and GIN (with ReLU non-linearity) over both datasets. In compliance with our theory, the GNNs fit the dataset whose essential partition has high walk index significantly better than they fit the dataset whose essential partition has low walk index. Furthermore, the improved train accuracy translates to improvements in test accuracy.

5 Practical Application: Expressivity Preserving Edge Sparsification

Section 4 theoretically characterizes the ability of a GNN to model interactions between input vertices. It reveals that this ability is controlled by a graph-theoretical property we call walk index (Definition 2). The current section derives a practical application of our theory, specifically, an edge sparsification algorithm named Walk Index Sparsification ( WIS ), which preserves the ability of a GNN to model interactions when input edges are removed. We present WIS, and show that it yields high predictive performance for GNNs over standard vertex prediction benchmarks of various scales, even when removing a significant portion of edges. In particular, we evaluate WIS using GCN [59], GIN [103], and ResGCN [66] over multiple datasets, including: Cora [89], which contains thousands of edges, DBLP [16], which contains tens of thousands of edges, and OGBN-ArXiv [52], which contains more than a million edges. WIS is simple, computationally efficient, and in our experiments has markedly outperformed alternative methods in terms of prediction accuracy across edge sparsity levels. We believe its further empirical investigation is a promising direction for future research.

5.1 Walk Index Sparsification (WIS)

Running GNNs over large-scale graphs can be prohibitively expensive in terms of runtime and memory. A natural way to tackle this problem is edge sparsification - removing edges from an input graph while attempting to maintain prediction accuracy ( cf. [67, 24]). 9,10

Our theory (Section 4) establishes that, the strength of interaction a depth L GNN can model across a partition of input vertices is determined by the partition's walk index, a quantity defined by the number of length L -1 walks originating from the partition's boundary. This brings forth a recipe for pruning edges. First, choose partitions across which the ability to model interactions is to be preserved. Then, for every input edge (excluding self-loops), compute a tuple holding what the walk indices of the chosen partitions will be if the edge is to be removed. Lastly, remove the edge whose tuple is maximal according to a preselected order over tuples ( e.g. an order based on the sum, min, or max of a tuple's entries). This process repeats until the desired number of edges are removed. The idea behind the above-described recipe, which we call General Walk Index Sparsification , is that each iteration greedily prunes the edge whose removal takes the smallest toll in terms of ability to model interactions across chosen partitions - see Algorithm 3 in Appendix F for a formal outline. Below we describe a specific instantiation of the recipe for vertex prediction tasks, which are particularly relevant with large-scale graphs, yielding our proposed algorithm - Walk Index Sparsification (WIS). Exploration of other instantiations is regarded as a promising avenue for future work.

In vertex prediction tasks, the interaction between an input vertex and the remainder of the input graph is of central importance. Thus, it is natural to choose the partitions induced by singletons ( i.e. the partitions ( { t } , V \ { t } ) , where t ∈ V ) as those across which the ability to model interactions is to be preserved. We would like to remove edges while avoiding a significant deterioration in the ability to model interaction under any of the chosen partitions. To that end, we compare walk index tuples according to their minimal entries, breaking ties using the second smallest entries, and so

forth. This is equivalent to sorting (in ascending order) the entries of each tuple separately, and then ordering the tuples lexicographically.

Algorithm 1 provides a self-contained description of the method attained by the foregoing choices. We refer to this method as ( L -1) -Walk Index Sparsification (WIS), where the ' ( L -1) ' indicates that only walks of length L -1 take part in the walk indices. Since ( L -1) -walk indices can be computed by taking the ( L -1) 'th power of the graph's adjacency matrix, ( L -1) -WIS runs in O ( N |E||V| 3 log( L )) time and requires O ( |E||V| + |V| 2 ) memory, where N is the number of edges to be removed. For large graphs a runtime cubic in the number of vertices can be restrictive. Fortunately, 1 -WIS, which can be viewed as an approximation for ( L -1) -WIS with L > 2 , facilitates a particularly simple and efficient implementation based solely on vertex degrees, requiring only linear time and memory - see Algorithm 2 (whose equivalence to 1 -WIS is explained in Appendix G). Specifically, 1 -WIS runs in O ( N |E| + |V| ) time and requires O ( |E| + |V| ) memory.

5.2 Empirical Evaluation

Below is an empirical evaluation of WIS. For brevity, we defer to Appendix H some implementation details, as well as experiments with additional GNN architectures (GIN and ResGCN) and datasets (Chameleon [82], Squirrel [82], and Amazon Computers [92]).

Using depth L = 3 GNNs (with ReLU non-linearity), we evaluate over the Cora dataset both 2 -WIS, which is compatible with the GNNs' depth, and 1 -WIS, which can be viewed as an efficient approximation. Over the DBLP and OGBN-ArXiv datasets, due to their larger scale only 1 -WIS is evaluated. Figure 3 (and Figure 8 in Appendix H) shows that WIS significantly outperforms the following alternative methods in terms of induced prediction accuracy: (i) a baseline in which edges are removed uniformly at random; (ii) a well-known spectral algorithm [93] designed to preserve the spectrum of the sparsified graph's Laplacian; and (iii) an adaptation of UGS [24] - a recent supervised approach for learning to prune edges. 11 Both 2 -WIS and 1 -WIS lead to higher test accuracies, while (as opposed to UGS) avoiding the need for labels, and for training a GNN over the original (non-sparsified) graph - a procedure which in some settings is prohibitively expensive in terms of runtime and memory. Interestingly, 1 -WIS performs similarly to 2 -WIS, indicating that the efficiency it brings does not come at a sizable cost in performance.

6 Conclusion

6.1 Summary

GNNs are designed to model complex interactions between entities represented as vertices of a graph. The current paper provides the first theoretical analysis for their ability to do so. We proved that, given a partition of input vertices, the strength of interaction that can be modeled between its sides is controlled by the walk index -agraph-theoretical characteristic defined by the number of walks originating from the boundary of the partition. Experiments with common GNN architectures, e.g. GCN [59] and GIN [103], corroborated this result.

Our theory formalizes conventional wisdom by which GNNs can model stronger interaction between regions of the input graph that are more interconnected. More importantly, we showed that it facilitates a novel edge sparsification algorithm which preserves the ability of a GNN to model interactions when edges are removed. Our algorithm, named Walk Index Sparsification ( WIS ), is simple, computationally efficient, and in our experiments has markedly outperformed alternative methods in terms of induced prediction accuracy. More broadly, WIS showcases the potential of improving GNNs by theoretically analyzing the interactions they can model.

6.2 Limitations and Future Work

The theoretical analysis considers GNNs with product aggregation, which are less common in practice ( cf. Section 3). We empirically demonstrated that its conclusions apply to more popular GNNs (Section 4.2), and derived a practical edge sparsification algorithm based on the theory (Section 5). Nonetheless, extending our analysis to additional aggregation functions is a worthy avenue to explore.

Our work also raises several interesting directions concerning WIS. A naive implementation of ( L -1) -WIS has runtime cubic in the number of vertices ( cf. Section 5.1). Since this can be restrictive for large-scale graphs, the evaluation in Section 5.2 mostly focused on 1 -WIS, which can be viewed as an efficient approximation of ( L -1) -WIS (its runtime and memory requirements are linear see Section 5.1). Future work can develop efficient exact implementations of ( L -1) -WIS ( e.g. using parallelization) and investigate regimes where it outperforms 1 -WIS in terms of induced prediction accuracy. Additionally, ( L -1) -WIS is a specific instantiation of the general WIS scheme (given in Appendix F), tailored for preserving the ability to model interactions across certain partitions. Exploring other instantiations, as well as methods for automatically choosing the partitions across which the ability to model interactions is preserved, are valuable directions for further research.

Acknowledgments and Disclosure of Funding

We thank Eshbal Hezroni for aid in preparing illustrative figures. This work was supported by a Google Research Scholar Award, a Google Research Gift, the Yandex Initiative in Machine Learning, the Israel Science Foundation (grant 1780/21), the Tel Aviv University Center for AI and Data Science, the Adelis Research Fund for Artificial Intelligence, Len Blavatnik and the Blavatnik Family Foundation, and Amnon and Anat Shashua. NR is supported by the Apple Scholars in AI/ML PhD fellowship.

References

Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. International Conference on Learning Representations (ICLR) , 2021.

Arash Amini, Amin Karbasi, and Farokh Marvasti. Low-rank matrix approximation using point-wise operators. IEEE transactions on information theory , 58(1):302-310, 2011.

Waiss Azizian and Marc Lelarge. Expressive power of invariant and equivariant graph neural networks. International Conference on Learning Representations (ICLR) , 2021.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Muhammet Balcilar, Renton Guillaume, Pierre Héroux, Benoit Gaüzère, Sébastien Adam, and Paul Honeine. Analyzing the expressive power of graph neural networks in a spectral perspective. International Conference on Learning Representations (ICLR) , 2021.

Muhammet Balcilar, Pierre Héroux, Benoit Gauzere, Pascal Vasseur, Sébastien Adam, and Paul Honeine. Breaking the limits of message passing graph neural networks. In International Conference on Machine Learning (ICML) , 2021.

Emilio Rafael Balda, Arash Behboodi, and Rudolf Mathar. A tensor analysis on dense connectivity via convolutional arithmetic circuits. Preprint , 2018.

Pradeep Kr Banerjee, Kedar Karhadkar, Yu Guang Wang, Uri Alon, and Guido Montúfar. Oversquashing in gnns through the lens of information contraction and graph expansion. In 2022 58th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 1-8. IEEE, 2022.

Pablo Barceló, Egor Kostylev, Mikael Monet, Jorge Pérez, Juan Reutter, and Juan-Pablo Silva. The logical expressiveness of graph neural networks. International Conference on Learning Representations (ICLR) , 2020.

Pablo Barceló, Floris Geerts, Juan Reutter, and Maksimilian Ryschkov. Graph neural networks with local graph parameters. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Surender Baswana and Sandeep Sen. A simple and linear time randomized algorithm for computing sparse spanners in weighted graphs. Random Structures & Algorithms , 30(4):532-563, 2007.

Gregory Beylkin and Martin J Mohlenkamp. Numerical operator calculus in higher dimensions. Proceedings of the National Academy of Sciences , 99(16):10246-10251, 2002.

Gregory Beylkin, Jochen Garcke, and Martin J Mohlenkamp. Multivariate regression and machine learning with sums of separable functions. SIAM Journal on Scientific Computing , 31(3):1840-1857, 2009.

Amandeep Singh Bhatia, Mandeep Kaur Saggi, Ajay Kumar, and Sushma Jain. Matrix product state-based quantum classifier. Neural computation , 31(7):1499-1517, 2019.

Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yuguang Wang, Pietro Lio, Guido F Montufar, and Michael Bronstein. Weisfeiler and lehman go cellular: Cw networks. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. International Conference on Learning Representations (ICLR) , 2018.

Giorgos Bouritsas, Fabrizio Frasca, Stefanos P Zafeiriou, and Michael Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 , 2021.

Richard Caron and Tim Traynor. The zero set of a polynomial. WSMR Report , pages 05-02, 2005.

Alireza Chakeri, Hamidreza Farhidzadeh, and Lawrence O Hall. Spectral sparsification in spectral clustering. In 2016 23rd international conference on pattern recognition (icpr) , pages 2301-2306. IEEE, 2016.

Michail Chatzianastasis, Johannes F Lutzeyer, George Dasoulas, and Michalis Vazirgiannis. Graph ordering attention networks. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2023.

Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the oversmoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2020.

Lei Chen, Zhengdao Chen, and Joan Bruna. On graph neural networks versus graph-augmented mlps. International Conference on Learning Representations (ICLR) , 2021.

Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks. In International Conference on Machine Learning (ICML) , 2021.

Zhengdao Chen, Soledad Villar, Lei Chen, and Joan Bruna. On the equivalence between graph isomorphism testing and function approximation with gnns. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Zhengdao Chen, Lei Chen, Soledad Villar, and Joan Bruna. Can graph neural networks count substructures? In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Grigorios G Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Yannis Panagakis, Jiankang Deng, and Stefanos Zafeiriou. P-nets: Deep polynomial neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 7325-7335, 2020.

Nadav Cohen and Amnon Shashua. Simnets: A generalization of convolutional networks. Advances in Neural Information Processing Systems (NeurIPS), Deep Learning Workshop , 2014.

Nadav Cohen and Amnon Shashua. Convolutional rectifier networks as generalized tensor decompositions. International Conference on Machine Learning (ICML) , 2016.

Nadav Cohen and Amnon Shashua. Inductive bias of deep convolutional networks through pooling geometry. International Conference on Learning Representations (ICLR) , 2017.

Nadav Cohen, Or Sharir, and Amnon Shashua. Deep simnets. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

Nadav Cohen, Or Sharir, and Amnon Shashua. On the expressive power of deep learning: A tensor analysis. Conference On Learning Theory (COLT) , 2016.

Nadav Cohen, Or Sharir, Yoav Levine, Ronen Tamari, David Yakira, and Amnon Shashua. Analysis and design of convolutional networks via hierarchical tensor decompositions. Intel Collaborative Research Institute for Computational Intelligence (ICRI-CI) Special Issue on Deep Learning Theory , 2017.

Nadav Cohen, Ronen Tamari, and Amnon Shashua. Boosting dilated convolutional networks with mixed tensor decompositions. International Conference on Learning Representations (ICLR) , 2018.

Nima Dehmamy, Albert-László Barabási, and Rose Yu. Understanding the representation power of graph neural networks in learning graph topology. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. Advances in neural information processing systems , 28, 2015.

Timo Felser, Marco Trenti, Lorenzo Sestini, Alessio Gianelle, Davide Zuliani, Donatella Lucchesi, and Simone Montangero. Quantum-inspired machine learning on high-energy physics data. npj Quantum Information , 7(1):1-8, 2021.

Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.02428 , 2019.

Vikas Garg, Stefanie Jegelka, and Tommi Jaakkola. Generalization and representational limits of graph neural networks. In International Conference on Machine Learning (ICML) , 2020.

Johannes Gasteiger, Stefan Weißenberger, and Stephan Günnemann. Diffusion improves graph learning. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Floris Geerts. The expressive power of kth-order invariant graph networks. arXiv preprint arXiv:2007.12035 , 2020.

Floris Geerts and Juan L Reutter. Expressiveness and approximation properties of graph neural networks. International Conference on Learning Representations (ICLR) , 2022.

Floris Geerts, Filip Mazowiecki, and Guillermo Perez. Let's agree to degree: Comparing graph convolutional networks in the message-passing framework. In International Conference on Machine Learning (ICML) , 2021.

Ran Gilad-Bachrach, Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Naehrig, and John Wernsing. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In International Conference on Machine Learning (ICML) , 2016.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning (ICML) , 2017.

Edward Grant, Marcello Benedetti, Shuxiang Cao, Andrew Hallam, Joshua Lockhart, Vid Stojevic, Andrew G Green, and Simone Severini. Hierarchical quantum classifiers. npj Quantum Information , 4(1): 1-8, 2018.

Wolfgang Hackbusch. On the efficient evaluation of coalescence integrals in population balance models. Computing , 78(2):145-159, 2006.

Michael Hamann, Gerd Lindner, Henning Meyerhenke, Christian L Staudt, and Dorothea Wagner. Structure-preserving sparsification methods for social networks. Social Network Analysis and Mining , 6 (1):1-22, 2016.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems (NeurIPS) , 2017.

William L Hamilton. Graph representation learning. Synthesis Lectures on Artifical Intelligence and Machine Learning , 14(3):1-159, 2020.

Robert J Harrison, George I Fann, Takeshi Yanai, and Gregory Beylkin. Multiresolution quantum chemistry in multiwavelet bases. In International Conference on Computational Science , pages 103-110. Springer, 2003.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Chenqing Hua, Guillaume Rabusseau, and Jian Tang. High-order pooling for graph neural networks with tensor decomposition. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML) , 2015.

Nicolas Keriven and Gabriel Peyré. Universal invariant and equivariant graph neural networks. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Valentin Khrulkov, Alexander Novikov, and Ivan Oseledets. Expressive power of recurrent neural networks. International Conference on Learning Representations (ICLR) , 2018.

Valentin Khrulkov, Oleksii Hrinchuk, and Ivan Oseledets. Generalized tensor models for recurrent neural networks. International Conference on Learning Representations (ICLR) , 2019.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations (ICLR) , 2017.

Jure Leskovec and Christos Faloutsos. Sampling from large graphs. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 631-636, 2006.

Yoav Levine, Or Sharir, and Amnon Shashua. Benefits of depth for long-term memory of recurrent networks. International Conference on Learning Representations (ICLR) Workshop , 2018.

Yoav Levine, David Yakira, Nadav Cohen, and Amnon Shashua. Deep learning and quantum entanglement: Fundamental connections with implications to network design. International Conference on Learning Representations (ICLR) , 2018.

Yoav Levine, Or Sharir, Nadav Cohen, and Amnon Shashua. Quantum entanglement in deep learning architectures. Physical review letters , 122(6):065301, 2019.

Yoav Levine, Noam Wies, Or Sharir, Hofit Bata, and Amnon Shashua. Limits to depth efficiencies of self-attention. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Yoav Levine, Noam Wies, Daniel Jannai, Dan Navon, Yedid Hoshen, and Amnon Shashua. The inductive bias of in-context learning: Rethinking pretraining example design. International Conference on Learning Representations (ICLR) , 2022.

Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. Deepergcn: All you need to train deeper gcns. arXiv preprint arXiv:2006.07739 , 2020.

Jiayu Li, Tianyun Zhang, Hao Tian, Shengmin Jin, Makan Fardad, and Reza Zafarani. Sgcn: A graph sparsifier based on graph convolutional networks. In Pacific-Asia Conference on Knowledge Discovery and Data Mining , pages 275-287. Springer, 2020.

Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Thirty-Second AAAI conference on artificial intelligence (AAAI) , 2018.

Andreas Loukas. What graph neural networks cannot learn: depth vs width. International Conference on Learning Representations (ICLR) , 2020.

Andreas Loukas. How hard is to distinguish graphs with graph neural networks? In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Dongsheng Luo, Wei Cheng, Wenchao Yu, Bo Zong, Jingchao Ni, Haifeng Chen, and Xiang Zhang. Learning to drop: Robust graph neural network via topological denoising. In Proceedings of the 14th ACM international conference on web search and data mining , pages 779-787, 2021.

Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the universality of invariant networks. In International Conference on Machine Learning (ICML) , 2019.

Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 4602-4609, 2019.

Christopher Morris, Yaron Lipman, Haggai Maron, Bastian Rieck, Nils M Kriege, Martin Grohe, Matthias Fey, and Karsten Borgwardt. Weisfeiler and leman go machine learning: The story so far. arXiv preprint arXiv:2112.09992 , 2021.

Galileo Namata, Ben London, Lise Getoor, Bert Huang, and U Edu. Query-driven active surveying for collective classification. In 10th international workshop on mining and learning with graphs , volume 8, page 1, 2012.

Hoang NT and Takanori Maehara. Revisiting graph neural networks: All we have is low-pass filters. arXiv preprint arXiv:1905.09550 , 2019.

Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. International Conference on Learning Representations (ICLR) , 2020.

Román Orús. A practical introduction to tensor networks: Matrix product states and projected entangled pair states. Annals of physics , 349:117-158, 2014.

Pál András Papp and Roger Wattenhofer. A theoretical comparison of graph neural network extensions. In International Conference on Machine Learning (ICML) , 2022.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W , 2017.

Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gcn: Geometric graph convolutional networks. International Conference on Learning Representations (ICLR) , 2020.

Noam Razin and Nadav Cohen. Implicit regularization in deep learning may not be explainable by norms. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Noam Razin, Asaf Maman, and Nadav Cohen. Implicit regularization in tensor factorization. International Conference on Machine Learning (ICML) , 2021.

Noam Razin, Asaf Maman, and Nadav Cohen. Implicit regularization in hierarchical tensor factorization and deep convolutional neural networks. International Conference on Machine Learning (ICML) , 2022.

Veeru Sadhanala, Yu-Xiang Wang, and Ryan Tibshirani. Graph sparsification approaches for laplacian smoothing. In Artificial Intelligence and Statistics , pages 1250-1259. PMLR, 2016.

Venu Satuluri, Srinivasan Parthasarathy, and Yiye Ruan. Local graph sparsification for scalable clustering. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of data , pages 721-732, 2011.

Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European semantic web conference , pages 593-607. Springer, 2018.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. AI magazine , 29(3):93-93, 2008.

Or Sharir and Amnon Shashua. On the expressive power of overlapping architectures of deep learning. International Conference on Learning Representations (ICLR) , 2018.

Or Sharir, Ronen Tamari, Nadav Cohen, and Amnon Shashua. Tensorial mixture models. arXiv preprint , 2016.

Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868 , 2018.

Daniel A Spielman and Nikhil Srivastava. Graph sparsification by effective resistances. SIAM Journal on Computing , 40(6):1913-1926, 2011.

E Miles Stoudenmire. Learning relevant features of data with multi-scale tensor networks. Quantum Science and Technology , 3(3):034003, 2018.

Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. International Conference on Learning Representations (ICLR) , 2022.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. International Conference on Learning Representations (ICLR) , 2018.

Guifré Vidal. Class of quantum many-body states that can be efficiently simulated. Physical review letters , 101(11):110501, 2008.

Elli Voudigari, Nikos Salamanos, Theodore Papageorgiou, and Emmanuel J Yannakoudakis. Rank degree: An efficient algorithm for graph sampling. In 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) , pages 120-129. IEEE, 2016.

Boris Weisfeiler and Andrei Leman. The reduction of a graph to canonical form and the algebra which appears therein. NTI, Series , 2(9):12-16, 1968.

Noam Wies, Yoav Levine, Daniel Jannai, and Amnon Shashua. Which transformer architecture fits my data? a vocabulary bottleneck in self-attention. International Conference on Machine Learning (ICML) , 2021.

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems , 32(1):4-24, 2020.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? International Conference on Learning Representations (ICLR) , 2019.

Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 974-983, 2018.

Bohang Zhang, Shengjie Luo, Liwei Wang, and Di He. Rethinking the expressive power of gnns via graph biconnectivity. International Conference on Learning Representations (ICLR) , 2023.

Cheng Zheng, Bo Zong, Wei Cheng, Dongjin Song, Jingchao Ni, Wenchao Yu, Haifeng Chen, and Wei Wang. Robust graph representation learning via neural sparsification. In International Conference on Machine Learning (ICML) , 2020.

A Formal Analysis: Quantifying the Ability of Graph Neural Networks to Model Interactions

We begin by upper bounding the separation ranks a GNN can achieve.

Theorem 2. For an undirected graph G and t ∈ V , let f ( θ, G ) and f ( θ, G ,t ) be the functions realized by depth L graph and vertex prediction GNNs, respectively, with width D h , learnable weights θ , and product aggregation (Equations (2) to (5) ). Then, for any I ⊆ V and assignment of weights θ it holds that:

Proof sketch (proof in Appendix I.2). In Appendix E, we show that the computations performed by a GNN with product aggregation can be represented as a tensor network . In brief, a tensor network is a weighted graph that describes a sequence of arithmetic operations known as tensor contractions (see Appendices E.1 and E.2 for a self-contained introduction to tensor networks). The tensor network corresponding to a GNN with product aggregation adheres to a tree structure - its leaves are associated with input vertex features and interior nodes embody the operations performed by the GNN. Importing machinery from tensor analysis literature, we prove that sep( f ( θ, G ) ; I ) is upper bounded by a minimal cut weight in the corresponding tensor network, among cuts separating leaves associated with input vertices in I from leaves associated with input vertices in I c . Equation (6) then follows by finding such a cut in the tensor network with sufficiently low weight. Equation (7) is established analogously.

A natural question is whether the upper bounds in Theorem 2 are tight, i.e. whether separation ranks close to them can be attained. We show that nearly matching lower bounds hold for almost all assignments of weights θ . To this end, we define admissible subsets of C I , based on a notion of vertex subsets with no repeating shared neighbors .

Definition 3. We say that I , J ⊆ V have no repeating shared neighbors if every k ∈ N ( I ) ∩N ( J ) has only a single neighbor in each of I and J , i.e. |N ( k ) ∩ I| = |N ( k ) ∩ J | = 1 .

Definition 4. For I ⊆ V , we refer to C ⊆ C I as an admissible subset of C I if there exist I ′ ⊆ I , J ′ ⊆ I c with no repeating shared neighbors such that C = N ( I ′ ) ∩ N ( J ′ ) . We use S ( I ) to denote the set comprising all admissible subsets of C I :

Theorem 3 below establishes that almost all possible values for the network's weights lead the upper bounds in Theorem 2 to be tight, up to logarithmic terms and to the number of walks from C I being replaced with the number of walks from any single C ∈ S ( I ) . The extent to which C I can be covered by an admissible subset thus determines how tight the upper bounds are. Trivially, at least the shared neighbors of any i ∈ I , j ∈ I c can be covered, since N ( i ) ∩ N ( j ) ∈ S ( I ) . Appendix C shows that for various canonical graphs all of C I , or a large part of it, can be covered by an admissible subset.

Theorem 3. Consider the setting and notation of Theorem 2. Given I ⊆ V , for almost all assignments of weights θ , i.e. for all but a set of Lebesgue measure zero, it holds that:

where:

with D := min { D x , D h } . If ρ L -1 ( C , V ) = 0 or ρ L -1 ( C , { t } ) = 0 , the respective lower bound (right hand side of Equation (8) or Equation (9) ) is zero by convention.

Proof sketch (proof in Appendix I.3). Our proof follows a line similar to that used in [64, 100, 65] for lower bounding the separation rank of self-attention neural networks. The separation rank of any f : ( R D x ) |V| → R can be lower bounded by examining its outputs over a grid of inputs. Specifically, for M ∈ N template vectors v (1) , . . . , v ( M ) ∈ R D x , we can create a grid tensor for f by evaluating it over each point in { ( v ( d 1 ) , . . . , v ( d |V| ) ) } M d 1 ,...,d |V| =1 and storing the outcomes in a tensor with |V| axes of dimension M each. Arranging the grid tensor as a matrix B ( f ) where rows correspond to axes indexed by I and columns correspond to the remaining axes, we show that rank( B ( f )) ≤ sep( f ; I ) . The proof proceeds by establishing that for almost every assignment of θ , there exist template vectors with which log(rank( B ( f ( θ, G ) ))) and log(rank( B ( f ( θ, G ,t ) ))) are greater than (or equal to) the right hand sides of Equations (8) and (9), respectively.

Directed edges and multiple edge types Appendix D generalizes Theorems 2 and 3 to the case of graphs with directed edges and an arbitrary number of edge types.

B Related Work

Expressivity of GNNs The expressivity of GNNs has been predominantly evaluated through ability to distinguish non-isomorphic graphs, as measured by correspondence to Weisfeiler-Leman (WL) graph isomorphism tests (see [75] for a recent survey). [103, 74] instigated this thread of research, establishing that message-passing GNNs are at most as powerful as the WL algorithm, and can match it under certain technical conditions. Subsequently, architectures surpassing WL were proposed, with expressivity measured via higher-order WL variants (see, e.g. , [74, 72, 25, 41, 6, 15, 10, 42, 17, 80]). Another line of inquiry regards universality among continuous permutation invariant or equivariant functions [73, 55, 69, 3, 42]. [25] showed that distinguishing non-isomorphic graphs and universality are, in some sense, equivalent. Lastly, there exist analyses of expressivity focused on the frequency response of GNNs [77, 5] and their capacity to compute specific graph functions, e.g. moments, shortest paths, and substructure counting [35, 9, 39, 69, 26, 23, 17].

Although a primary purpose of GNNs is to model interactions between vertices, none of the past works formally characterize their ability to do so, as our theory does. 12 The current work thus provides a novel perspective on the expressive power of GNNs. Furthermore, a major limitation of existing approaches - in particular, proofs of equivalence to WL tests and universality - is that they often operate in asymptotic regimes of unbounded network width or depth. Consequently, they fall short of addressing which type of functions can be realized by GNNs of practical size. In contrast, we characterize how the modeled interactions depend on both the input graph structure and the neural network architecture (width and depth). As shown in Section 5, this facilitates designing an efficient and effective edge sparsification algorithm.

Measuring modeled interactions via separation rank Separation rank (Section 2.2) has been paramount to the study of interactions modeled by certain convolutional, recurrent, and self-attention neural networks. It enabled theoretically analyzing how different architectural parameters impact expressivity [32, 29, 30, 34, 7, 90, 62, 61, 56, 57, 64, 100, 65] and implicit regularization [83, 84, 85]. 13 On the practical side, insights brought forth by separation rank led to tools for improving performance, including: guidelines for architecture design [30, 62, 64, 100], pretraining schemes [65], and regularizers for countering locality in convolutional neural networks [85]. We employ separation rank

for studying the interactions GNNs model between vertices, and similarly provide both theoretical insights and a practical application - edge sparsification algorithm (Section 5).

Edge sparsification Computations over large-scale graphs can be prohibitively expensive in terms of runtime and memory. As a result, various methods were proposed for sparsifying graphs by removing edges while attempting to maintain structural properties, such as distances between vertices [11, 48], graph Laplacian spectrum [93, 86], and vertex degree distribution [98], or outcomes of graph analysis and clustering algorithms [87, 20]. Most relevant to our work are recent edge sparsification methods aiming to preserve the prediction accuracy of GNNs as the number of removed edges increases [67, 24]. These methods require training a GNN over the original (non-sparsified) graph, hence only inference costs are reduced. Guided by our theory, in Section 5 we propose Walk Index Sparsification ( WIS ) - an edge sparsification algorithm that preserves expressive power in terms of ability to model interactions. WIS improves efficiency for both training and inference. Moreover, comparisons with the spectral algorithm of [93] and a recent method from [24] demonstrate that WIS brings about higher prediction accuracies across edge sparsity levels.

C Tightness of Upper Bounds for Separation Rank

Theorem 2 upper bounds the separation rank with respect to I ⊆ V of a depth L GNN with product aggregation. According to it, under the setting of graph prediction, the separation rank is largely capped by the ( L -1) -walk index of I , i.e. the number of length L -1 walks from C I -the set of vertices with an edge crossing the partition ( I , I c ) . Similarly, for prediction over t ∈ V , separation rank is largely capped by the ( L -1 , t ) -walk index of I , which takes into account only length L -1 walks from C I ending at t . Theorem 3 provides matching lower bounds, up to logarithmic terms and to the number of walks from C I being replaced with the number of walks from any single admissible subset C ∈ S ( I ) (Definition 4). Hence, the match between the upper and lower bounds is determined by the portion of C I that can be covered by an admissible subset.

In this appendix, to shed light on the tightness of the upper bounds, we present several concrete examples on which a significant portion of C I can be covered by an admissible subset.

Complete graph Suppose that every two vertices are connected by an edge, i.e. E = {{ i, j } : i, j ∈ V} . For any non-empty I glyph[subsetnoteql] V , clearly C I = N ( I ) ∩ N ( I c ) = V . In this case, C I = V ∈ S ( I ) , meaning C I is an admissible subset of itself. To see it is so, notice that for any i ∈ I , j ∈ I c , all vertices are neighbors of both I ′ := { i } and J ′ := { j } , which trivially have no repeating shared neighbors (Definition 3). Thus, up to a logarithmic factor, the upper and lower bounds from Theorems 2 and 3 coincide.

Chain graph Suppose that E = {{ i, i +1 } : i ∈ [ |V| 1] } ∪ {{ i, i } : i ∈ V} . For any non-empty I glyph[subsetnoteql] V , at least half of the vertices in C I can be covered by an admissible subset. That is, there exists C ∈ S ( I ) satisfying |C| ≥ 2 -1 · |C I | . For example, such C can be constructed algorithmically as follows. Let I ′ , J ′ = ∅ . Starting from k = 1 , if { k, k +1 } ⊆ C I and one of { k, k +1 } is in I while the other is in I c , then assign I ′ ←I ′ ∪ ( { k, k +1 } ∩ I ) , J ′ ←J ′ ∪ ( { k, k +1 } ∩ I c ) , and k ← k + 3 . That is, add each of { k, k + 1 } to either I ′ if it is in I or J ′ if it is in I c , and skip vertex k + 2 . Otherwise, set k ← k + 1 . The process terminates once k > |V| 1 . By construction, I ′ ⊆ I and J ′ ⊆ I c , implying that N ( I ′ ) ∩ N ( J ′ ) ⊆ C I . Due to the chain graph structure, I ′ ∪J ′ ⊆ N ( I ′ ) ∩N ( J ′ ) and I ′ and J ′ have no repeating shared neighbors (Definition 3). Furthermore, for every pair of vertices from C I added to I ′ and J ′ , we can miss at most two other vertices from C I . Thus, C := N ( I ′ ) ∩N ( J ′ ) is an admissible subset of C I satisfying |C| ≥ 2 -1 ·|C I | .

glyph[negationslash]

General graph For an arbitrary graph and non-empty I glyph[subsetnoteql] V , an admissible subset of C I can be obtained by taking any sequence of pairs ( i 1 , j 1 ) , . . . , ( i M , j M ) ∈ I × I c with no shared neighbors, in the sense that [ N ( i m ) ∪ N ( j m )] ∩ [ N ( i m ′ ) ∪ N ( j m ′ )] = ∅ for all m = m ′ ∈ [ M ] . Defining I ′ := { i 1 , . . . , i M } and J ′ := { j 1 , . . . , j M } , by construction they do not have repeating shared neighbors (Definition 3), and so N ( I ′ ) ∩N ( J ′ ) ∈ S ( I ) . In particular, the shared neighbors of each pair are covered by N ( I ′ ) ∩ N ( J ′ ) , i.e. ∪ M m =1 N ( i m ) ∩ N ( j m ) ⊆ N ( I ′ ) ∩ N ( J ′ ) .

D Extension of Analysis to Directed Graphs With Multiple Edge Types

In this appendix, we generalize the separation rank bounds from Theorems 2 and 3 to directed graphs with multiple edge types.

Let G = ( V , E , τ ) be a directed graph with vertices V = [ |V| ] , edges E ⊆ { ( i, j ) : i, j ∈ V} , and a map τ : E → [ Q ] from edges to one of Q ∈ N edge types. For i ∈ V , let N in ( i ) := { j ∈ V : ( j, i ) ∈ E} be its incoming neighbors and N out ( i ) := { j ∈ V : ( i, j ) ∈ E} be its outgoing neighbors . For I ⊆ V , we denote N in ( I ) := ∪ i ∈I N in ( i ) and N out ( I ) := ∪ i ∈I N out ( i ) . As customary in the context of GNNs, we assume the existence of all self-loops ( cf. Section 2.1).

Message-passing GNNs (Section 3) operate identically over directed and undirected graphs, except that in directed graphs the hidden embedding of a vertex is updated only according to its incoming neighbors. For handling multiple edge types, common practice is to use different weight matrices per type in the GNN's update rule ( cf. [49, 88]). Hence, we consider the following update rule for directed graphs with multiple edge types, replacing that from Equation (2):

where ( W (1 ,q ) ∈ R D h × D x ) q ∈ [ Q ] and ( W ( l,q ) ∈ R D h × D h ) l ∈{ 2 ,...,L } ,q ∈ [ Q ] are learnable weight matrices.

glyph[negationslash]

In our analysis for undirected graphs (Appendix A), a central concept is C I -the set of vertices with an edge crossing the partition induced by I ⊆ V . Due to the existence of self-loops it is equal to the shared neighbors of I and I c , i.e. C I = N ( I ) ∩ N ( I c ) . We generalize this concept to directed graphs, defining C I → to be the set of vertices with an incoming edge from the other side of the partition induced by I , i.e. C I → := { i ∈ I : N in ( i ) ∩ I c = ∅} ∪ { j ∈ I c : N in ( j ) ∩ I glyph[negationslash] = ∅} . Due to the existence of self-loops it is given by C I → = N out ( I ) ∩ N out ( I c ) . Indeed, for undirected graphs C I → = C I .

With the definition of C I → in place, Theorem 4 upper bounds the separation ranks a GNN can achieve over directed graphs with multiple edge types. A technical subtlety is that the bounds depend on walks of lengths l = L -1 , L -2 , . . . , 0 , while those in Theorem 2 for undirected graphs depend only on walks of length L -1 . As shown in the proof of Theorem 2, this dependence exists in undirected graphs as well. Though, in undirected graphs with self-loops, the number of length l ∈ N walks from C I decays exponentially as l decreases. One can therefore replace the sum over walk lengths with walks of length L -1 (up to a multiplicative constant). By contrast, in directed graphs this is not true in general, e.g. , when C I → contains only vertices with no outgoing edges (besides self-loops).

Theorem 4. For a directed graph with multiple edge types G and t ∈ V , let f ( θ, G ) and f ( θ, G ,t ) be the functions realized by depth L graph and vertex prediction GNNs, respectively, with width D h , learnable weights θ , and product aggregation (Equations (3) to (5) and (10) ). Then, for any I ⊆ V and assignment of weights θ it holds that:

Proof sketch (proof in Appendix I.4). The proof follows a line identical to that of Theorem 2, only requiring adjusting definitions from undirected graphs to directed graphs with multiple edge types.

Towards lower bounding separation ranks, we generalize the definitions of vertex subsets with no repeating shared neighbors (Definition 3) and admissible subsets of C I (Definition 4) to directed graphs.

Definition 5. We say that I , J ⊆ V have no outgoing repeating shared neighbors if every k ∈ N out ( I ) ∩ N out ( J ) has only a single incoming neighbor in each of I and J , i.e. |N in ( k ) ∩ I| = |N in ( k ) ∩ J | = 1 .

Definition 6. For I ⊆ V , we refer to C ⊆ C I → as an admissible subset of C I → if there exist I ′ ⊆ I , J ′ ⊆ I c with no outgoing repeating shared neighbors such that C = N out ( I ′ ) ∩ N out ( J ′ ) . We use S → ( I ) to denote the set comprising all admissible subsets of C I → :

Theorem 5 generalizes the lower bounds from Theorem 3 to directed graphs with multiple edge types. Theorem 5. Consider the setting and notation of Theorem 4. Given I ⊆ V , for almost all assignments of weights θ , i.e. for all but a set of Lebesgue measure zero, it holds that:

with D := min { D x , D h } . If ρ L -1 ( C , V ) = 0 or ρ L -1 ( C , { t } ) = 0 , the respective lower bound (right hand side of Equation (13) or Equation (14) ) is zero by convention.

Proof sketch (proof in Appendix I.5). The proof follows a line identical to that of Theorem 3, only requiring adjusting definitions from undirected graphs to directed graphs with multiple edge types.

E Representing Graph Neural Networks With Product Aggregation as Tensor Networks

In this appendix, we prove that GNNs with product aggregation (Section 3) can be represented through tensor networks - a graphical language for expressing tensor contractions, widely used in quantum mechanics literature for modeling quantum states ( cf. [97]). This representation facilitates upper bounding the separation ranks of a GNN with product aggregation (proofs for Theorem 2 and its extension in Appendix D), and is delivered in Appendix E.3. We note that analogous tensor network representations were shown for variants of recurrent and convolutional neural networks [61, 62]. For the convenience of the reader, we lay out basic concepts from the field of tensor analysis in Appendix E.1 and provide a self-contained introduction to tensor networks in Appendix E.2 (see [79] for a more in-depth treatment).

E.1 Primer on Tensor Analysis

For our purposes, a tensor is simply a multi-dimensional array. The order of a tensor is its number of axes, which are typically called modes ( e.g. a vector is an order one tensor and a matrix is an order two tensor). The dimension of a mode refers to its length, i.e. the number of values it can be indexed with. For an order N ∈ N tensor A ∈ R D 1 ×···× D N with modes of dimensions D 1 , . . . , D N ∈ N , we will denote by A d 1 ,...,d N its ( d 1 , . . . , d N ) 'th entry, where ( d 1 , . . . , d N ) ∈ [ D 1 ] ×··· × [ D N ] .

It is possible to rearrange tensors into matrices - a process known as matricization . The matricization of A with respect to I ⊆ [ N ] , denoted J A ; I K ∈ R ∏ i ∈I D i × ∏ j ∈I c D j is its arrangement as a matrix where rows correspond to modes indexed by I and columns correspond to the remaining modes. Specifically, denoting the elements in I by i 1 < · · · < i |I| and those in I c by j 1 < · · · < j |I c | , the matricization J A ; I K holds the entries of A such that A d 1 ,...,d N is placed in row index 1 + ∑ |I| l =1 ( d i l -1) ∏ |I| l ′ = l +1 D i l ′ and column index 1 + ∑ |I c | l =1 ( d j l -1) ∏ |I c | l ′ = l +1 D j l ′ .

Tensors with modes of the same dimension can be combined via contraction -ageneralization of matrix multiplication. It will suffice to consider contractions where one of the modes being contracted is the last mode of its tensor.

Definition 7. Let A ∈ R D 1 ×···× D N , B ∈ R D ′ 1 ×···× D ′ N ′ for orders N,N ′ ∈ N and mode dimensions D 1 , . . . , D N , D ′ 1 , . . . , D ′ N ′ ∈ N satisfying D n = D ′ N ′ for some n ∈ [ N ] . The moden contraction of A with B , denoted A ∗ n B ∈ R D 1 ×···× D n -1 × D ′ 1 ×···× D ′ N ′ -1 × D n +1 ×···× D N , is given element-wise by:

where:

Vector

Matrix

Order

𝑁

Tensor

𝐯

1

𝐷

∈ ℝ

"

𝐌

1

𝐷

#

𝐷

$

2

∈ ℝ

"

!

×"

"

𝓣

𝐷

#

𝐷

$

𝐷

'

𝑁

1

Vector-Matrix Multiplication

𝐌

1

𝐷

#

𝐷

$

∈ ℝ

"

!

𝐯

2

1

2

∈ ℝ

"

!

×⋯×"

#

𝐌𝐯

For example, the mode2 contraction of A ∈ R D 1 × D 2 with B ∈ R D ′ 1 × D 2 boils down to multiplying A with B glyph[latticetop] from the right, i.e. A ∗ 2 B = AB glyph[latticetop] . It is oftentimes convenient to jointly contract multiple tensors. Given an order N tensor A and M ∈ N ≤ N tensors B (1) , . . . , B ( M ) , we use A ∗ i ∈ [ M ] B ( i ) to denote the contraction of A with B (1) , . . . , B ( M ) in modes 1 , . . . , M , respectively (assuming mode dimensions are such that the contractions are well-defined).

E.2 Tensor Networks

A tensor network is an undirected weighted graph T = ( V T , E T , w T ) that describes a sequence of tensor contractions (Definition 7), with vertices V T , edges E T , and a function mapping edges to natural weights w T : E T → N . We will only consider tensor networks that are connected. To avoid confusion with vertices and edges of a GNN's input graph, and in accordance with tensor network terminology, we refer by nodes and legs to the vertices and edges of a tensor network, respectively.

Every node in a tensor network is associated with a tensor, whose order is equal to the number of legs emanating from the node. Each end point of a leg is associated with a mode index, and the leg's weight determines the dimension of the corresponding tensor mode. That is, an end point of e ∈ E T is a pair ( A , n ) ∈ V T × N , with n ranging from one to the order of A , and w T ( e ) is the dimension of A in mode n . A leg can either connect two nodes or be connected to a node on one end and be loose on the other end. If two nodes are connected by a leg, their associated tensors are contracted together in the modes specified by the leg. Legs with a loose end are called open legs . The number of open legs is exactly the order of the tensor produced by executing all contractions in the tensor network, i.e. by contracting the tensor network. Figure 4 presents exemplar tensor network diagrams of a vector, matrix, order N ∈ N tensor, and vector-matrix multiplication.

E.3 Tensor Networks Corresponding to Graph Neural Networks With Product Aggregation

Fix some undirected graph G and learnable weights θ = ( W (1) , . . . , W ( L ) , W ( o ) ) . Let f ( θ, G ) and f ( θ, G ,t ) , for t ∈ V , be the functions realized by depth L graph and vertex prediction GNNs, respectively, with width D h and product aggregation (Equations (2) to (5)). For X = ( x (1) , . . . , x ( |V| ) ) ∈ R D x ×|V| , we construct tensor networks T ( X ) and T ( t ) ( X ) whose contraction yields f ( θ, G ) ( X ) and f ( θ, G ,t ) ( X ) , respectively. Both T ( X ) and T ( t ) ( X ) adhere to a tree structure, where each leaf node is associated with a vertex feature vector, i.e. one of x (1) , . . . , x ( |V| ) , and each interior node is associated with a weight matrix from W (1) , . . . , W ( L ) , W ( o ) or a δ -tensor with modes of dimension D h , holding ones on its hyper-diagonal and zeros elsewhere. We denote an order N ∈ N tensor of the latter type by δ ( N ) ∈ R D h ×···× D h , i.e. δ ( N ) d 1 ,...,d N = 1 if d 1 = · · · = d N and δ ( N ) d 1 ,...,d N = 0 otherwise for all d 1 , . . . , d N ∈ [ D h ] .

𝑾

(𝒍)

𝑾

(𝒍)

𝜹

( 𝓝 𝒊

&𝟏)

𝒉

(𝒍(𝟏,𝒋

𝟏

)

𝒉

(𝒍(𝟏,𝒋

𝑴

)

𝒉

(𝑳,𝟏)

𝒉

(𝑳, 𝓥 )

𝜹

( 𝓥 &𝟏)

𝑾

(𝒐)

𝒇

𝜽,𝓖

(𝑿)

𝒉

(𝑳,𝒕)

𝑾

(𝒐)

𝒇

𝜽,𝓖,𝒕

(𝑿)

𝒉

(𝒍,𝒊)

(a)

(b)

(c)

Intuitively, T ( X ) and T ( t ) ( X ) embody unrolled computation trees, describing the operations performed by the respective GNNs through tensor contractions. Let h ( l,i ) = glyph[circledot] j ∈N ( i ) ( W ( l ) h ( l -1 ,j ) ) be the hidden embedding of i ∈ V at layer l ∈ [ L ] (recall h (0 ,j ) = x ( j ) for j ∈ V ), and denote N ( i ) = { j 1 , . . . , j M } . We can describe h ( l,i ) as the outcome of contracting each h ( l -1 ,j 1 ) , . . . , h ( l -1 ,j M ) with W ( l ) , i.e. computing W ( l ) h ( l -1 ,j 1 ) , . . . , W ( l ) h ( l -1 ,j M ) , followed by contracting the resulting vectors with δ ( |N ( i ) | +1) , which induces product aggregation (see Figure 5(a)). Furthermore, in graph prediction, the output layer producing f ( θ, G ) ( X ) = W ( o ) ( glyph[circledot] i ∈V h ( L,i ) ) amounts to contracting h ( L, 1) , . . . , h ( L, |V| ) with δ ( |V| +1) , and subsequently contracting the resulting vector with W ( o ) (see Figure 5(b)); while for vertex prediction, f ( θ, G ,t ) ( X ) = W ( o ) h ( L,t ) is a contraction of h ( L,t ) with W ( o ) (see Figure 5(c)).

Overall, every layer in a GNN with product aggregation admits a tensor network formulation given the outputs of the previous layer. Thus, we can construct a tree tensor network for the whole GNN by starting from the output layer - Figure 5(b) for graph prediction or Figure 5(c) for vertex prediction and recursively expanding nodes associated with h ( l,i ) according to Figure 5(a), for l = L, . . . , 1 and i ∈ V . A technical subtlety is that each h ( l,i ) can appear multiple times during this procedure. In the language of tensor networks this translate to duplication of nodes. Namely, there are multiple copies of the sub-tree representing h ( l,i ) in the tensor network - one copy per appearance when unraveling the recursion. Figure 6 displays examples for tensor network diagrams of T ( X ) and T ( t ) ( X ) .

We note that, due to the node duplication mentioned above, the explicit definitions of T ( X ) and T ( t ) ( X ) entail cumbersome notation. Nevertheless, we provide them in Appendix E.3.1 for the interested reader.

E.3.1 Explicit Tensor Network Definitions

The tree tensor network representing f ( θ, G ) ( X ) consists of an initial input level - the leaves of the tree - comprising ρ L ( { i } , V ) copies of x ( i ) for each i ∈ V . We will use x ( i,γ ) to denote the copies of x ( i ) for i ∈ V and γ ∈ [ ρ L ( { i } , V )] . In accordance with the GNN inducing f ( θ, G ) , following the initial input level are L + 1 layers . Each layer l ∈ [ L ] includes two levels: one comprising ρ L -l +1 ( V , V ) nodes standing for copies of W ( l ) , and another containing δ -tensors ρ L -l ( { i } , V ) copies of δ ( |N ( i ) | +1) per i ∈ V . We associate each node in these layers with its layer index and a vertex of the input graph i ∈ V . Specifically, we will use W ( l,i,γ ) to denote copies of W ( l ) and δ ( l,i,γ ) to denote copies of δ ( |N ( i ) | +1) , for l ∈ [ L ] , i ∈ V , and γ ∈ N . In terms of connectivity, every leaf x ( i,γ ) has a leg to W (1 ,i,γ ) . The rest of the connections between nodes are such that each sub-tree whose root is δ ( l,i,γ ) represents h ( l,i ) , i.e. contracting the sub-tree results in the hidden

𝟏 𝟐 𝟑 𝒢 = 𝒱 = 1,2,3 ℰ = 1,2 , 1,1 , 2,2 , 3,3

𝐡

(𝟏,𝟑)

𝐡

(𝟏,𝟐)

𝐖

(𝟏)

𝐖

(𝟏)

𝜹

(𝟑)

𝐖

(𝟐)

𝐖

(𝟏)

𝐖

(𝟏)

𝜹

(𝟑)

𝐖

(𝟐)

𝜹

(𝟑)

𝐖

(𝟏)

𝜹

(𝟐)

𝐖

(𝟐)

𝐖

(𝒐)

𝐱

(𝟏)

𝐱

(𝟐)

𝐱

(𝟏)

𝐱

(𝟐)

𝐱

(𝟑)

𝒇

𝜽,𝓖

(𝑿)

depth

𝐿 = 2

𝐡

(𝟏,𝟏)

𝐡

(𝟐,𝟏)

𝐡

(𝟐,𝟑)

𝜹

(𝟐)

𝐡

(𝟏,𝟐)

𝐖

(𝟏)

𝐖

(𝟏)

𝜹

(𝟑)

𝐖

(𝟐)

𝐖

(𝟏)

𝐖

(𝟏)

𝜹

(𝟑)

𝐖

(𝟐)

𝐱

(𝟏)

𝐱

(𝟐)

𝐱

(𝟏)

𝐱

(𝟐)

𝐡

(𝟏,𝟏)

𝐡

(𝟐,𝟐)

𝜹

(𝟑)

𝜹

(𝟒)

𝓣(𝐗)

𝐖

(𝟏)

𝐖

(𝟏)

𝜹

(𝟑)

𝐖

(𝟐)

𝐖

(𝟏)

𝐖

(𝟏)

𝜹

(𝟑)

𝐖

(𝟐)

𝜹

(𝟑)

𝐖

(𝒐)

𝐱

(𝟏)

𝐱

(𝟐)

𝐱

(𝟏)

𝐱

(𝟐)

𝒇

𝜽,𝓖,𝟏

(𝑿)

depth

𝐿 = 2

𝓣 (𝟏) (𝐗)

𝟏 𝟐 𝟑 𝒢 = 𝒱 = 1,2,3 ℰ = 1,2 , 1,1 , 2,2 , 3,3

𝐡

(𝟏,𝟑)

𝐡

(𝟏,𝟐)

𝐖

(𝟏,𝟏,𝟏)

𝐖

(𝟏,𝟐,𝟏)

𝐖

(𝟐,𝟏,𝟏)

𝐖

(𝟏,𝟏,𝟑)

𝐖

(𝟏,𝟐,𝟑)

𝐖

(𝟐,𝟐,𝟏)

𝜹

(𝟐,𝟏,𝟏)

𝐖

(𝟏,𝟑,𝟏)

𝜹

(𝟏,𝟑,𝟏)

𝐖

(𝟐,𝟑,𝟏)

𝐖

(𝒐)

𝐱

(𝟏,𝟏)

𝐱

(𝟐,𝟏)

𝐱

(𝟏,𝟑)

𝐱

(𝟐,𝟑)

𝐱

(𝟑,𝟏)

𝒇

𝜽,𝓖

(𝑿)

depth

𝐿 = 2

𝐡

(𝟏,𝟏)

𝐡

(𝟐,𝟏)

𝐡

(𝟐,𝟑)

𝜹

(𝟐,𝟑,𝟏)

𝐡

(𝟏,𝟐)

𝐖

(𝟏,𝟏,𝟐)

𝐖

(𝟏,𝟐,𝟐)

𝜹

(𝟏,𝟏,𝟐)

𝐖

(𝟐,𝟏,𝟐)

𝐖

(𝟏,𝟏,𝟒)

𝐖

(𝟏,𝟐,𝟒)

𝜹

(𝟏,𝟐,𝟐)

𝐖

(𝟐,𝟐,𝟐)

𝐱

(𝟏,𝟐)

𝐱

(𝟐,𝟐)

𝐱

(𝟏,𝟒)

𝐱

(𝟐)

𝐡

(𝟏,𝟏)

𝐡

(𝟐,𝟐)

𝐱

(𝟐,𝟒)

𝜹

(𝟐,𝟐,𝟏)

𝜹

(𝟏,𝟏,𝟏)

𝜹

(𝟏,𝟐,𝟏)

𝜹

(𝟒)

𝓣(𝐗)

𝒇

𝜽,𝓖,𝟏

(𝑿)

depth

𝐿 = 2

𝐖

(𝟐,𝟏,𝟏)

𝜹

(𝟏,𝟏,𝟏)

𝐖

(𝟏,𝟏,𝟏)

𝐱

𝐖

(𝟏,𝟏)

(𝟏,𝟐,𝟏)

𝐱

𝐖

(𝟐,𝟐,𝟏)

𝜹

(𝟏,𝟐,𝟏)

𝐖

(𝟏,𝟏,𝟐)

(𝟐,𝟏)

𝐱

𝐖

(𝟏,𝟐)

𝓣 (𝟏) (𝐗)

(𝟏,𝟐,𝟐)

𝐱

(𝟐,𝟐)

𝐖

(𝒐)

𝜹

(𝟐,𝟏,𝟏)

embedding for i ∈ V at layer l ∈ [ L ] of the GNN inducing f ( θ, G ) . Last, is an output layer consisting of two connected nodes: a δ ( |V| +1) node, which has a leg to every δ -tensor from layer L , and a W ( o ) node. See Figure 7 (left) for an example of a tensor network diagram representing f ( θ, G ) ( X ) with this notation.

The tensor network construction for f ( θ, G ,t ) ( X ) is analogous to that for f ( θ, G ) ( X ) , comprising an initial input level followed by L +1 layers. Its input level and first L layers are structured the same, up to differences in the number of copies for each node. Specifically, the number of copies of x ( i ) is ρ L ( { i } , { t } ) instead of ρ L ( { i } , V ) , the number of copies of W ( l ) is ρ L -l +1 ( V , { t } ) instead of ρ L -l +1 ( V , V ) , and the number of copies of δ ( |N ( i ) | +1) is ρ L -l ( { i } , { t } ) instead of ρ L -l ( { i } , V ) , for i ∈ V and l ∈ [ L ] . The output layer consists only of a W ( o ) node, which is connected to the δ -tensor in layer L corresponding to vertex t . See Figure 7 (right) for an example of a tensor network diagram representing f ( θ, G ,t ) ( X ) with this notation.

Formally, the tensor network producing f ( θ, G ) ( X ) , denoted T ( X ) = ( V T ( X ) , E T ( X ) , w T ( X ) ) , is defined by:

where φ l,i,j ( γ ) := γ + ∑ k<i s.t. k ∈N ( j ) ρ L -l ( { k } , V ) , for l ∈ [ L ] , i ∈ V , and γ ∈ [ ρ L -l ( { i } , V )] , is used to map a δ -tensor copy corresponding to i in layer l to a W ( l ) copy, and N ( i ) j , for i ∈ V and j ∈ [ |N ( i ) | ] , denotes the j 'th neighbor of i according to an ascending order (recall vertices are represented by indices from 1 to |V| ).

Similarly, the tensor network T ( t ) ( X ) = ( V T ( t ) ( X ) , E T ( t ) ( X ) , w T ( t ) ( X ) ) , producing f ( θ, G ,t ) ( X ) , is defined by:

where φ ( t ) l,i,j ( γ ) := γ + ∑ k<i s.t. k ∈N ( j ) ρ L -l ( { k } , { t } ) , for l ∈ [ L ] , i ∈ V , and γ ∈ [ ρ L -l ( { i } , { t } )] , is used to map a δ -tensor copy corresponding to i in layer l to a W ( l ) copy.

Proposition 1 verifies that contracting T ( X ) and T ( t ) ( X ) yields f ( θ, G ) ( X ) and f ( θ, G ,t ) ( X ) , respectively.

Proposition 1. For an undirected graph G and t ∈ V , let f ( θ, G ) and f ( θ, G ,t ) be the functions realized by depth L graph and vertex prediction GNNs, respectively, with width D h , learnable weights θ , and product aggregation (Equations (2) to (5) ). For vertex features X = ( x (1) , . . . , x ( |V| ) ) ∈ R D x ×|V| , let the tensor networks T ( X ) = ( V T ( X ) , E T ( X ) , w T ( X ) ) and T ( t ) ( X ) = ( V T ( t ) ( X ) , E T ( t ) ( X ) , w T ( t ) ( X ) ) be as defined in Equations (15) to (20) , respectively. Then, performing the contractions described by T ( X ) produces f ( θ, G ) ( X ) , and performing the contractions described by T ( t ) ( X ) produces f ( θ, G ,t ) ( X ) .

Proof sketch (proof in Appendix I.6). For both T ( X ) and T ( t ) ( X ) , a straightforward induction over the layer l ∈ [ L ] establishes that contracting the sub-tree whose root is δ ( l,i,γ ) results in h ( l,i ) for all i ∈ V and γ , where h ( l,i ) is the hidden embedding for i at layer l of the GNNs inducing f ( θ, G ) and f ( θ, G ,t ) , given vertex features x (1) , . . . , x ( |V| ) . The proof concludes by showing that the contractions in the output layer of T ( X ) and T ( t ) ( X ) reproduce the operations defining f ( θ, G ) ( X ) and f ( θ, G ,t ) ( X ) in Equations (3) and (4), respectively.

F General Walk Index Sparsification

Our edge sparsification algorithm - Walk Index Sparsification (WIS) - was obtained as an instance of the General Walk Index Sparsification (GWIS) scheme described in Section 5. Algorithm 3 formally outlines this general scheme.

G Efficient Implementation of 1 -Walk Index Sparsification

Algorithm 2 (Section 5) provides an efficient implementation for 1 -WIS, i.e. Algorithm 1 with L = 2 . In this appendix, we formalize the equivalence between the two algorithms, meaning, we establish that Algorithm 2 indeed implements 1 -WIS.

Examining some iteration n ∈ [ N ] of 1 -WIS, let s ∈ R |V| be the tuple defined by s t = WI 1 ,t ( { t } ) = ρ 1 ( C { t } , { t } ) for t ∈ V . Recall that C { t } is the set of vertices with an edge crossing the partition induced by { t } . Thus, if t is not isolated, then C { t } = N ( t ) and s t = WI 1 ,t ( { t } ) = |N ( t ) | . Otherwise, if t is isolated, then C { t } = ∅ and s t = WI 1 ,t ( { t } ) = 0 . 1 -WIS computes for each e ∈ E (excluding self-loops) a tuple s ( e ) ∈ R |V| holding in its t 'th entry what the value of WI 1 ,t ( { t } ) would be if e is to be removed, for all t ∈ V . Notice that s ( e ) and s agree on all entries except for i, j ∈ e , since removing e from the graph only affects the degrees of i and j . Specifically, for i ∈ e ,

Algorithm 3 ( L -1) -General Walk Index Sparsification (GWIS)

Input:

G -graph

L ∈ N -GNNdepth

N ∈ N -number of edges to remove

I 1 , . . . , I M ⊆ V -vertex subsets specifying walk indices to maintain for graph prediction

J 1 , . . . , J M ′ ⊆ V and t 1 , . . . , t M ′ ∈ V -vertex subsets specifying walk indices to maintain with respect to target vertices, for vertex prediction

ARGMAX - operator over tuples ( s ( e ) ∈ R M + M ′ ) e ∈E that returns the edge whose tuple is maximal according to some order

Result: Sparsified graph obtained by removing N edges from G

either s ( e ) i = s i -1 = |N ( i ) | -1 if the removal of e did not isolate i , or s ( e ) i = s i -2 = 0 if it did (due to self-loops, if a vertex has a single edge to another then |N ( i ) | = 2 , so removing that edge changes WI 1 ,i ( { i } ) from two to zero). As a result, for any e = { i, j } , e ′ = { i ′ , j ′ } ∈ E , after sorting the entries of s ( e ) and s ( e ′ ) in ascending order we have that s ( e ′ ) is greater in lexicographic order than s ( e ) if and only if the pair (min {|N ( i ′ ) | , |N ( j ′ ) |} , max {|N ( i ′ ) | , |N ( j ′ ) |} ) is greater in lexicographic order than (min {|N ( i ) | , |N ( j ) |} , max {|N ( i ) | , |N ( j ) |} ) . Therefore, at every iteration n ∈ [ N ] Algorithm 2 and 1 -WIS (Algorithm 1 with L = 2 ) remove the same edge.

H Further Experiments and Implementation Details

H.1 Further Experiments

Figure 8 supplements Figure 3 from Section 5.2 by including experiments with additional: (i) GNN architectures - GIN and ResGCN; and (ii) datasets - Chameleon, Squirrel, and Amazon Computers. Overall, our evaluation includes six standard vertex prediction datasets in which we observed the graph structure to be crucial for accurate prediction, as measured by the difference between the test accuracy of a GCN trained and evaluated over the original graph and its test accuracy when trained and evaluated over the graph after all of the graph's edges were removed. We also considered, but excluded, the following datasets in which the accuracy difference was insignificant (less than five percentage points): Citeseer [89], PubMed [76], Coauthor CS and Physics [92], and Amazon Photo [92].

H.2 Further Implementation Details

We provide implementation details omitted from our experimental reports (Section 4.2, Section 5, and Appendix H.1). Source code for reproducing our results and figures, based on the PyTorch [81] and PyTorch Geometric [38] frameworks, can be found at https://github.com/noamrazin/

gnn_interactions . All experiments were run either on a single Nvidia RTX 2080 Ti GPU or a single Nvidia RTX A6000 GPU.

H.2.1 Empirical Demonstration of Theoretical Analysis (Table 1)

Models All models used, i.e. GCN, GAT, and GIN, had three layers of width 16 with ReLU nonlinearity. To ease optimization, we added layer normalization [4] after each one. Mean aggregation and a linear output layer were applied over the last hidden embeddings for prediction. As in the synthetic experiments of [1], each GAT layer consisted of four attention heads. Each GIN layer had its glyph[epsilon1] parameter fixed to zero and contained a two-layer feed-forward network, whose layers comprised a linear layer, batch normalization [54], and ReLU non-linearity.

Data The datasets consisted of 10000 train and 2000 test graphs. For every graph, we drew uniformly at random a label from { 0 , 1 } and an image from Fashion-MNIST. Then, depending on the chosen label, another image was sampled either from the same class (for label 1 ) or from all other classes (for label 0 ). We extracted patches of pixels from each image by flattening it into a vector and splitting the vector to 16 equally sized segments.

Optimization The binary cross-entropy loss was minimized via the Adam optimizer [58] with default β 1 , β 2 coefficients and full-batches ( i.e. every batch contained the whole training set). Optimiza-

tion proceeded until the train accuracy did not improve by at least 0 . 01 over 1000 consecutive epochs or 10000 epochs elapsed. The learning rates used for GCN, GAT, and GIN were 5 · 10 -3 , 5 · 10 -3 , and 10 -2 , respectively.

Hyperparameter tuning For each model separately, to tune the learning rate we carried out five runs (differing in random seed) with every value in the range { 10 -1 , 5 · 10 -2 , 10 -2 , 5 · 10 -3 , 10 -3 } over the dataset whose essential partition has low walk index. Since our interest resides in expressivity, which manifests in ability to fit the training set, for every model we chose the learning rate that led to the highest mean train accuracy.

H.2.2 Edge Sparsification (Figures 3 and 8)

Adaptations to UGS [24] [24] proposed UGS as a framework for jointly pruning input graph edges and weights of a GNN. At a high-level, UGS trains two differentiable masks, m g and m θ , that are multiplied with the graph adjacency matrix and the GNN's weights, respectively. Then, after a certain number of optimization steps, a predefined percentage p g of graph edges are removed according to the magnitudes of entries in m g , and similarly, p θ percent of the GNN's weights are fixed to zero according to the magnitudes of entries in m θ . This procedure continues in iterations, where each time the remaining GNN weights are rewinded to their initial values, until the desired sparsity levels are attained - see Algorithms 1 and 2 in [24]. To facilitate a fair comparison of our ( L -1) -WIS edge sparsification algorithm with UGS, we make the following adaptations to UGS.

We adapt UGS to only remove edges, which is equivalent to fixing the entries in the weight mask m θ to one and setting p θ = 0 in Algorithm 1 of [24].

For comparing performance across a wider range of sparsity levels, the number of edges removed at each iteration is changed from 5% of the current number of edges to 5% of the original number of edges.

Since our evaluation focuses on undirected graphs, we enforce the adjacency matrix mask m g to be symmetric.

Spectral sparsification [93] For Cora and DBLP, we used a Python implementation of the spectral sparsification algorithm from [93], based on the PyGSP library implementation. 14 To enable more efficient experimentation over the larger scale OGBN-ArXiv dataset, we used a Julia implementation based on that from the Laplacians library. 15

Models The GCN and GIN models had three layers of width 64 with ReLU non-linearity. As in the experiments of Section 4.2, we added layer normalization [4] after each one. Every GIN layer had a trainable glyph[epsilon1] parameter and contained a two-layer feed-forward network, whose layers comprised a linear layer, batch normalization [54], and ReLU non-linearity. For ResGCN, we used the implementation from [24] with ten layers of width 64 . In all models, a linear output layer was applied over the last hidden embeddings for prediction.

Data All datasets in our evaluation are multi-class vertex prediction tasks, each consisting of a single graph. In Cora, DBLP, and OGBN-ArXiv, vertices represent scientific publications and edges stand for citation links. In Chameleon and Squirrel, vertices represent web pages on Wikipedia and edges stand for mutual links between pages. In Amazon Computers, vertices represent products and edges indicate that two products are frequently bought together. For simplicity, we treat all graphs as undirected. Table 2 reports the number of vertices and undirected edges in each dataset. For all datasets, except OGBN-ArXiv, we randomly split the labels of vertices into train, validation, and test sets comprising 80% , 10% , and 10% of all labels, respectively. For OGBN-ArXiv, we used the default split from [52].

Optimization The cross-entropy loss was minimized via the Adam optimizer [58] with default β 1 , β 2 coefficients and full-batches ( i.e. every batch contained the whole training set). Optimization proceeded until the validation accuracy did not improve by at least 0 . 01 over 1000 consecutive epochs or 10000 epochs elapsed. The test accuracies reported in Figure 3 are those achieved during the epochs with highest validation accuracies. Table 3 specifies additional optimization hyperparameters.

Hyperparameter tuning For each combination of model and dataset separately, we tuned the learning rate, weight decay coefficient, and edge mask glyph[lscript] 1 regularization coefficient for UGS, and applied the chosen values for evaluating all methods without further tuning (note that the edge mask glyph[lscript] 1 regularization coefficient is relevant only for UGS). In particular, we carried out a grid search over learning rates { 10 -3 , 5 · 10 -4 , 10 -4 } , weight decay coefficients { 10 -3 , 10 -4 , 0 } , and edge mask glyph[lscript] 1 regularization coefficients { 10 -2 , 10 -3 , 10 -4 } . Per hyperparameter configuration, we ran ten repetitions of UGS (differing in random seed), each until all of the input graph's edges were removed. At every edge sparsity level ( 0% , 5% , 10% , . . . , 1 00% ), in accordance with [24], we trained a new model with identical hyperparameters, but a fixed edge mask, over each of the ten graphs. We chose the hyperparameters that led to the highest mean validation accuracy, taken over the sparsity levels and ten runs.

Due to the size of the ResGCN model, tuning its hyperparameters entails significant computational costs. Thus, over the Cora and DBLP datasets, per hyperparameter configuration we ran five repetitions of UGS with ResGCN instead of ten. For the large-scale OGBN-ArXiv dataset, we adopted the same hyperparameters used for GCN.

Other To allow more efficient experimentation, we compute the edge removal order of 2 -WIS (Algorithm 1) in batches of size 100 . Specifically, at each iteration of 2 -WIS, instead of removing the edge e ′ with maximal walk index tuple s ( e ′ ) , the 100 edges with largest walk index tuples are removed. For randomized edge sparsification algorithms - random pruning, the spectral sparsification method of [93], and the adaptation of UGS [24] - the evaluation runs for a given dataset and percentage of removed edges were carried over sparsified graphs obtained using different random seeds.

I Deferred Proofs

I.1 Additional Notation

For vectors, matrices, or tensors, parenthesized superscripts denote elements in a collection, e.g. ( a ( i ) ∈ R D ) N n =1 , while subscripts refer to entries, e.g. A d 1 ,d 2 ∈ R is the ( d 1 , d 2 ) 'th entry of A ∈ R D 1 × D 2 . A colon is used to indicate a range of entries, e.g. a : d is the first d entries of

a ∈ R D . We use ∗ to denote tensor contractions (Definition 7), ◦ to denote the Kronecker product, and glyph[circledot] to denote the Hadamard product. For P ∈ N ≥ 0 , the P 'th Hadamard power operator is denoted by glyph[circledot] P , i.e. [ glyph[circledot] P A ] d 1 ,d 2 = A P d 1 ,d 2 for A ∈ R D 1 × D 2 . Lastly, when enumerating over sets of indices an ascending order is assumed.

I.2 Proof of Theorem 2

We assume familiarity with the basic concepts from tensor analysis introduced in Appendix E.1, and rely on the tensor network representations established for GNNs with product aggregation in Appendix E. Specifically, we use the fact that for any X = ( x (1) , . . . , x ( |V| ) ) ∈ R D x ×|V| there exist tree tensor networks T ( X ) and T ( t ) ( X ) (described in Appendix E.3 and formally defined in Equations (15) to (20)) such that: (i) their contraction yields f ( θ, G ) ( X ) and f ( θ, G ,t ) ( X ) , respectively (Proposition 1); and (ii) each of their leaves is associated with a vertex feature vector, i.e. one of x (1) , . . . , x ( |V| ) , whereas all other aspects of the tensor networks do not depend on x (1) , . . . , x ( |V| ) .

The proof proceeds as follows. In Appendix I.2.1, by importing machinery from tensor analysis literature (in particular, adapting Claim 7 from [62]), we show that the separation ranks of f ( θ, G ) and f ( θ, G ,t ) can be upper bounded via cuts in their corresponding tensor networks. Namely, sep( f ( θ, G ) ; I ) is at most the minimal multiplicative cut weight in T ( X ) , among cuts separating leaves associated with vertices of the input graph in I from leaves associated with vertices of the input graph in I c , where multiplicative cut weight refers to the product of weights belonging to legs crossing the cut. Similarly, sep( f ( θ, G ,t ) ; I ) is at most the minimal multiplicative cut weight in T ( t ) ( X ) , among cuts of the same form. We conclude in Appendices I.2.2 and I.2.3 by applying this technique for upper bounding sep( f ( θ, G ) ; I ) and sep( f ( θ, G ,t ) ; I ) , respectively, i.e. by finding cuts in the respective tensor networks with sufficiently low multiplicative weights.

I.2.1 Upper Bounding Separation Rank via Multiplicative Cut Weight in Tensor Network

In a tensor network T = ( V T , E T , w T ) , every J T ⊆ V T induces a cut ( J T , J c T ) , i.e. a partition of the nodes into two sets. We denote by E T ( J T ) := {{ u, v } ∈ E T : u ∈ J T , v ∈ J c T } the set of legs crossing the cut, and define the multiplicative cut weight of J T to be the product of weights belonging to legs in E T ( J T ) , i.e. :

For X = ( x (1) , . . . , x ( |V| ) ) ∈ R D x ×|V| , let T ( X ) and T ( t ) ( X ) be the tensor networks corresponding to f ( θ, G ) ( X ) and f ( θ, G ,t ) ( X ) (detailed in Appendix E.3), respectively. Both T ( X ) and T ( t ) ( X ) adhere to a tree structure. Each leaf node is associated with a vertex feature vector ( i.e. one of x (1) , . . . , x ( |V| ) ), while interior nodes are associated with weight matrices or δ -tensors. The latter are tensors with modes of equal dimension holding ones on their hyper-diagonal and zeros elsewhere. The restrictions imposed by δ -tensors induce a modified notion of multiplicative cut weight, where legs incident to the same δ -tensor only contribute once to the weight product (note that weights of legs connected to the same δ -tensor are equal since they stand for mode dimensions).

Definition 8. For a tensor network T = ( V T , E T , w T ) and subset of nodes J T ⊆ V T , let E T ( J T ) be the set of edges crossing the cut ( J T , J c T ) . Denote by ˜ E T ( J T ) ⊆ E T ( J T ) a subset of legs containing for each δ -tensor in V T only a single leg from E T ( J T ) incident to it, along with all legs in E T ( J T ) not connected to δ -tensors. Then, the modified multiplicative cut weight of J T is:

Lemma 1 establishes that sep( f ( θ, G ) ; I ) and sep( f ( θ, G ,t ) ; I ) are upper bounded by the minimal modified multiplicative cut weights in T ( X ) and T ( t ) ( X ) , respectively, among cuts separating leaves associated with vertices in I from leaves associated vertices in I c .

Lemma 1. For any X = ( x (1) , . . . , x ( |V| ) ) ∈ R D x ×|V| , let T ( X ) = ( V T ( X ) , E T ( X ) , w T ( X ) ) and T ( t ) ( X ) = ( V T ( t ) ( X ) , E T ( t ) ( X ) , w T ( t ) ( X ) ) be the tensor network representations of f ( θ, G ) ( X ) and f ( θ, G ,t ) ( X ) (described in Appendix E.3 and formally defined in Equations (15) to (20) ), respectively.

Denote by V T ( X ) [ I ] ⊆ V T ( X ) and V T ( t ) ( X ) [ I ] ⊆ V T ( t ) ( X ) the sets of leaf nodes in T ( X ) and T ( t ) ( X ) , respectively, associated with vertices in I from the input graph G . Formally:

Similarly, denote by V T ( X ) [ I c ] ⊆ V T ( X ) and V T ( t ) ( X ) [ I c ] ⊆ V T ( t ) ( X ) the sets of leaf nodes in T ( X ) and T ( t ) ( X ) , respectively, associated with vertices in I c . Then, the following hold:

where ˜ w Π T ( X ) ( J T ( X ) ) is the modified multiplicative cut weight of J T ( X ) in T ( X ) and ˜ w Π T ( t ) ( X ) ( J T ( t ) ( X ) ) is the modified multiplicative cut weight of J T ( t ) ( X ) in T ( t ) ( X ) (Definition 8).

Proof. Wefirst prove Equation (21). Examining T ( X ) , notice that: (i) by Proposition 1 its contraction yields f ( θ, G ) ( X ) ; (ii) it has a tree structure; and (iii) each of its leaves is associated with a vertex feature vector, i.e. one of x (1) , . . . , x ( |V| ) , whereas all other aspects of the tensor network do not depend on x (1) , . . . , x ( |V| ) . Specifically, for any X and X ′ the nodes, legs, and leg weights of T ( X ) and T ( X ′ ) are identical, up to the assignment of features in the leaf nodes. Let F ∈ R D x ×···× D x be the order ρ L ( V , V ) tensor obtained by contracting all interior nodes in T ( X ) . The above implies that we may write f ( θ, G ) ( X ) as a contraction of F with x (1) , . . . , x ( |V| ) . Specifically, it holds that:

for any X = ( x (1) , . . . , x ( |V| ) ) ∈ R D x ×|V| , where µ : [ ρ L ( V , V )] → V maps a mode index of F to the appropriate vertex of G according to T ( X ) . Let µ -1 ( I ) := { n ∈ [ ρ L ( V , V )] : µ ( n ) ∈ I} be the mode indices of F corresponding to vertices in I . Invoking Lemma 2 leads to the following matricized form of Equation (23):

where ◦ denotes the Kronecker product.

We claim that sep( f ( θ, G ) ; I ) ≤ rank q F ; µ -1 ( I ) y . To see it is so, denote R := rank q F ; µ -1 ( I ) y and let u (1) , . . . , u ( R ) ∈ R D ρ L ( I , V ) x and ¯ u (1) , . . . , ¯ u ( R ) ∈ R D ρ L ( I c , V ) x be such that q F ; µ -1 ( I ) y = ∑ R r =1 u ( r ) (¯ u ( r ) ) glyph[latticetop] . Then, defining g ( r ) : ( R D x ) |I| → R and ¯ g ( r ) : ( R D x ) |I c | → R , for r ∈ [ R ] , as:

where X I := ( x ( i ) ) i ∈I and X I c := ( x ( j ) ) j ∈I c , we have that:

Since sep( f ( θ, G ) ; I ) is the minimal number of summands in a representation of this form of f ( θ, G ) , indeed, sep( f ( θ, G ) ; I ) ≤ R = rank q F ; µ -1 ( I ) y .

What remains is to apply Claim 7 from [62], which upper bounds the rank of a tensor's matricization with multiplicative cut weights in a tree tensor network. In particular, consider an order N ∈ N

tensor A produced by contracting a tree tensor network T . Then, for any K ⊆ [ N ] we have that rank J A ; K K is at most the minimal modified multiplicative cut weight in T , among cuts separating leaves corresponding to modes K from leaves corresponding to modes K c . Thus, invoking Claim 7 from [62] establishes Equation (21):

Equation (22) readily follows by steps analogous to those used above for proving Equation (21).

I.2.2 Cut in Tensor Network for Graph Prediction (Proof of Equation (6))

For X = ( x (1) , . . . , x ( |V| ) ) ∈ R D x ×|V| , let T ( X ) = ( V T ( X ) , E T ( X ) , w T ( X ) ) be the tensor network corresponding to f ( θ, G ) ( X ) (detailed in Appendix E.3 and formally defined in Equations (15) to (17)). By Lemma 1, to prove that

it suffices to find J T ( X ) ⊆ V T ( X ) satisfying: (i) leaves of T ( X ) associated with vertices in I are in J T ( X ) , whereas leaves associated with vertices in I c are not in J T ( X ) ; and (ii) ˜ w Π T ( X ) ( J T ( X ) ) ≤ D 4 ρ L -1 ( C I , V )+1 h , where ˜ w Π T ( X ) ( J T ( X ) ) is the modified multiplicative cut weight of J T ( X ) (Definition 8). To this end, define J T ( X ) to hold all nodes in V T ( X ) corresponding to vertices in I . Formally:

Clearly, J T ( X ) upholds (i) .

As for (ii) , there are two types of legs crossing the cut induced by J T ( X ) in T ( X ) . First, are those connecting a δ -tensor with a weight matrix in the same layer, where one is associated with a vertex in I and the other with a vertex in I c . That is, legs connecting δ ( l,i,γ ) with W ( l, N ( i ) j ,φ l,i,j ( γ )) , where i ∈ V and N ( i ) j ∈ V are on different sides of the partition ( I , I c ) in the input graph, for j ∈ [ |N ( i ) | ] , l ∈ [ L ] , γ ∈ [ ρ L -l ( { i } , V )] . The δ -tensors participating in these legs are exactly those associated with some i ∈ C I (recall C I is the set of vertices with an edge crossing the partition ( I , I c ) ). So, for every l ∈ [ L ] and i ∈ C I there are ρ L -l ( { i } , V ) such δ -tensors. Second, are legs from δ -tensors associated with i ∈ I in the L 'th layer to the δ -tensor in the output layer of T ( X ) . That is, legs connecting δ ( L,i, 1) with δ ( |V| +1) , for i ∈ I . Legs incident to the same δ -tensor only contribute once to ˜ w Π T ( X ) ( J T ( X ) ) . Thus, since the weights of all legs connected to δ -tensors are equal to D h , we have that:

Lastly, it remains to show that ∑ L l =1 ρ L -l ( C I , V ) ≤ 4 ρ L -1 ( C I , V ) , since in that case Lemma 1 implies:

which yields Equation (6) by taking the log of both sides.

The main idea is that, in an undirected graph with self-loops, the number of length l ∈ N walks from vertices with at least one neighbor decays exponentially when l decreases. Observe that ρ l ( C I , V ) ≤ ρ l +1 ( C I , V ) for all l ∈ N . Hence:

Furthermore, any length l ∈ N ≥ 0 walk i 0 , i 1 , . . . , i l ∈ V from C I induces at least two walks of length l +2 from C I , distinct from those induced by other length l walks - one which goes twice through the

self-loop of i 0 and then proceeds according to the length l walk, i.e. i 0 , i 0 , i 0 , i 1 , . . . , i l , and another that goes to a neighboring vertex (exists since i 0 ∈ C I ), returns to i 0 , and then proceeds according to the length l walk. This means that ρ L -l ( C I , V ) ≤ 2 -1 · ρ L -l +2 ( C I , V ) ≤ · · · ≤ 2 -glyph[floorleft] l/ 2 glyph[floorright] · ρ L -1 ( C I , V ) for all l ∈ { 3 , 5 , . . . , L -1 } . Going back to Equation (24), this leads to:

completing the proof of Equation (6).

I.2.3 Cut in Tensor Network for Vertex Prediction (Proof of Equation (7))

This part of the proof follows a line similar to that of Appendix I.2.2, with differences stemming from the distinction between the operation of a GNN over graph and vertex prediction tasks.

For X = ( x (1) , . . . , x ( |V| ) ) ∈ R D x ×|V| , let T ( t ) ( X ) = ( V T ( t ) ( X ) , E T ( t ) ( X ) , w T ( t ) ( X ) ) be the tensor network corresponding to f ( θ, G ,t ) ( X ) (detailed in Appendix E.3 and formally defined in Equations (18) to (20)). By Lemma 1, to prove that

it suffices to find J T ( t ) ( X ) ⊆ V T ( t ) ( X ) satisfying: (i) leaves of T ( t ) ( X ) associated with vertices in I are in J T ( t ) ( X ) , whereas leaves associated with vertices in I c are not in J T ( t ) ( X ) ; and (ii) ˜ w Π T ( t ) ( X ) ( J T ( t ) ( X ) ) ≤ D 4 ρ L -1 ( C I , { t } ) h , where ˜ w Π T ( t ) ( X ) ( J T ( t ) ( X ) ) is the modified multiplicative cut weight of J T ( t ) ( X ) (Definition 8). To this end, define J T ( t ) ( X ) to hold all nodes in V T ( t ) ( X ) corresponding to vertices in I . Formally:

where W ( o ) := { W ( o ) } if t ∈ I and W ( o ) := ∅ otherwise. Clearly, J T ( t ) ( X ) upholds (i) .

As for (ii) , the legs crossing the cut induced by J T ( t ) ( X ) in T ( t ) ( X ) are those connecting a δ -tensor with a weight matrix in the same layer, where one is associated with a vertex in I and the other with a vertex in I c . That is, legs connecting δ ( l,i,γ ) with W ( l, N ( i ) j ,φ ( t ) l,i,j ( γ )) , where i ∈ V and N ( i ) j ∈ V are on different sides of the partition ( I , I c ) in the input graph, for j ∈ [ |N ( i ) | ] , l ∈ [ L ] , γ ∈ [ ρ L -l ( { i } , { t } )] . The δ -tensors participating in these legs are exactly those associated with some i ∈ C I (recall C I is the set of vertices with an edge crossing the partition ( I , I c ) ). Hence, for every l ∈ [ L ] and i ∈ C I there are ρ L -l ( { i } , { t } ) such δ -tensors. Legs connected to the same δ -tensor only contribute once to ˜ w Π T ( t ) ( X ) ( J T ( t ) ( X ) ) . Thus, since the weights of all legs connected to δ -tensors are equal to D h , we have that:

Lastly, it remains to show that ∑ L l =1 ρ L -l ( C I , { t } ) ≤ 4 ρ L -1 ( C I , { t } ) , as in that case Lemma 1 implies:

which leads to Equation (7) by taking the log of both sides.

The main idea is that, in an undirected graph with self-loops, the number of length l ∈ N walks ending at t that originate from vertices with at least one neighbor decays exponentially when l decreases. First, clearly ρ l ( C I , { t } ) ≤ ρ l +1 ( C I , { t } ) for all l ∈ N . Therefore:

Furthermore, any length l ∈ N ≥ 0 walk i 0 , i 1 , . . . , i l -1 , t ∈ V from C I to t induces at least two walks of length l + 2 from C I to t , distinct from those induced by other length l walks - one which goes twice through the self-loop of i 0 and then proceeds according to the length l walk, i.e. i 0 , i 0 , i 0 , i 1 , . . . , i l -1 , t , and another that goes to a neighboring vertex (exists since i 0 ∈ C I ), returns to i 0 , and then proceeds according to the length l walk. This means that ρ L -l ( C I , { t } ) ≤ 2 -1 · ρ L -l +2 ( C I , { t } ) ≤ · · · ≤ 2 -glyph[floorleft] l/ 2 glyph[floorright] · ρ L -1 ( C I , { t } ) for all l ∈ { 3 , 5 , . . . , L -1 } . Going back to Equation (25), we have that:

concluding the proof of Equation (7).

I.2.4 Technical Lemma

Lemma 2. For any order N ∈ N tensor A ∈ R D ×···× D , vectors x (1) , . . . , x ( N ) ∈ R D , and subset of mode indices I ⊆ [ N ] , it holds that A ∗ i ∈ [ N ] x ( i ) = ( ◦ i ∈ I x ( i ) ) glyph[latticetop] J A ; I K ( ◦ j ∈I c x ( j ) ) ∈ R .

Proof. The identity follows directly from the definitions of tensor contraction, matricization, and Kronecker product (Appendix I.1):

I.3 Proof of Theorem 3

We assume familiarity with the basic concepts from tensor analysis introduced in Appendix E.1.

We begin by establishing a general technique for lower bounding the separation rank of a function through grid tensors , also used in [64, 100, 65, 85]. For any f : ( R D x ) N → R and M ∈ N template vectors v (1) , . . . , v ( M ) ∈ R D x , we can create a grid tensor of f , which is a form of function discretization, by evaluating it over each point in { ( v ( d 1 ) , . . . , v ( d N ) ) } M d 1 ,...,d N =1 and storing the outcomes in an order N tensor with modes of dimension M . That is, the grid tensor of f for templates v (1) , . . . , v ( M ) , denoted B ( f ) ∈ R M ×···× M , is defined by B ( f ) d 1 ,...,d N = f ( v ( d 1 ) , . . . , v ( d N ) ) for all d 1 , . . . , d N ∈ [ M ] . 16 Lemma 3 shows that sep( f ; I ) is lower bounded by the rank of B ( f ) 's matricization with respect to I .

Lemma 3. For f : ( R D x ) N → R and M ∈ N template vectors v (1) , . . . , v ( M ) ∈ R D x , let B ( f ) ∈ R M ×···× M be the corresponding order N grid tensor of f . Then, for any I ⊆ [ N ] :

Proof. If sep( f ; I ) is ∞ or zero, i.e. f cannot be represented as a finite sum of separable functions (with respect to I ) or is identically zero, then the claim is trivial. Otherwise, denote R := sep( f ; I ) , and let g (1) , . . . , g ( R ) : ( R D x ) |I| → R and ¯ g (1) , . . . , ¯ g ( R ) : ( R D x ) |I c | → R such that:

where X := ( x (1) , . . . , x ( N ) ) , X I := ( x ( i ) ) i ∈I , and X I c := ( x ( j ) ) j ∈I c . For r ∈ [ R ] , let B ( g ( r ) ) and B (¯ g ( r ) ) be the grid tensors of g ( r ) and ¯ g ( r ) over templates v (1) , . . . , v ( M ) , respectively. That is, B ( g ( r ) ) d i : i ∈I = g ( r ) (( v ( d i ) ) i ∈I ) and B (¯ g ( r ) ) d j : j ∈I c = ¯ g ( r ) (( v ( d j ) ) j ∈I c ) for all d 1 , . . . , d N ∈ [ M ] .

By Equation (26) we have that for any d 1 , . . . , d N ∈ [ M ] :

Denoting by u ( r ) ∈ R M |I| and ¯ u ( r ) ∈ R M |I c | the arrangements of B ( g ( r ) ) and B (¯ g ( r ) ) as vectors, respectively for r ∈ [ R ] , this implies that the matricization of B ( f ) with respect to I can be written as:

We have arrived at a representation of J B ( f ); I K as a sum of R outer products between two vectors. An outer product of two vectors is a matrix of rank at most one. Consequently, by sub-additivity of rank we conclude: rank J B ( f ); I K ≤ R = sep( f ; I ) .

In the context of graph prediction, let C ∗ ∈ argmax C∈S ( I ) log( α C ) · ρ L -1 ( C , V ) . By Lemma 3, to prove that Equation (8) holds for weights θ , it suffices to find template vectors for which log(rank q B ( f ( θ, G ) ) ; I y ) ≥ log( α C ∗ ) · ρ L -1 ( C ∗ , V ) . Notice that, since the outputs of f ( θ, G ) vary polynomially with the weights θ , so do the entries of q B ( f ( θ, G ) ) ; I y for any choice of template vectors. Thus, according to Lemma 9, by constructing weights θ and template vectors satisfying log(rank q B ( f ( θ, G ) ) ; I y ) ≥ log( α C ∗ ) · ρ L -1 ( C ∗ , V ) , we may conclude that this is the case for almost all assignments of weights, meaning Equation (8) holds for almost all assignments of weights. In Appendix I.3.1 we construct such weights and template vectors.

In the context of vertex prediction, let C ∗ t ∈ argmax C∈S ( I ) log( α C ,t ) · ρ L -1 ( C , { t } ) . Due to arguments analogous to those above, to prove that Equation (9) holds for almost all assignments of weights, we need only find weights θ and template vectors satisfying log(rank q B ( f ( θ, G ,t ) ) ; I y ) ≥ log( α C ∗ t ,t ) · ρ L -1 ( C ∗ t , { t } ) . In Appendix I.3.2 we do so.

Lastly, recalling that a finite union of measure zero sets has measure zero as well establishes that Equations (8) and (9) jointly hold for almost all assignments of weights.

I.3.1 Weights and Template Vectors Assignment for Graph Prediction (Proof of Equation (8))

We construct weights θ and template vectors satisfying log(rank q B ( f ( θ, G ) ) ; I y ) ≥ log( α C ∗ ) · ρ L -1 ( C ∗ , V ) , where C ∗ ∈ argmax C∈S ( I ) log( α C ) · ρ L -1 ( C , V ) .

If ρ L -1 ( C ∗ , V ) = 0 , then the claim is trivial since there exist weights and template vectors for which q B ( f ( θ, G ) ) ; I y is not the zero matrix ( e.g. taking all weight matrices to be zero-padded identity matrices and choosing a single template vector holding one in its first entry and zeros elsewhere).

glyph[negationslash]

Now, assuming that ρ L -1 ( C ∗ , V ) > 0 , which in particular implies that I glyph[negationslash] = ∅ , I glyph[negationslash] = V , and C ∗ = ∅ , we begin with the case of GNN depth L = 1 , after which we treat the more general L ≥ 2 case.

Case of L = 1 : Consider the weights θ = ( W (1) , W ( o ) ) given by W (1) := I ∈ R D h × D x and W ( o ) := (1 , . . . , 1) ∈ R 1 × D h , where I is a zero padded identity matrix, i.e. it holds ones on its diagonal and zeros elsewhere. We choose template vectors v (1) , . . . , v ( D ) ∈ R D x such that v ( m ) holds the m 'th standard basis vector of R D in its first D coordinates and zeros in the remaining entries, for m ∈ [ D ] (recall D := min { D x , D h } ). Namely, denote by e (1) , . . . , e ( D ) ∈ R D the standard basis vectors of R D , i.e. e ( m ) d = 1 if d = m and e ( m ) d = 0 otherwise for all m,d ∈ [ D ] . We let v ( m ) : D := e ( m ) and v ( m ) D +1: := 0 for all m ∈ [ D ] .

We prove that for this choice of weights and template vectors, for all d 1 , . . . , d |V| ∈ [ D ] :

To see it is so, notice that:

with h (1 ,i ) = glyph[circledot] j ∈N ( i ) ( W (1) v ( d j ) ) = glyph[circledot] j ∈N ( i ) ( Iv ( d j ) ) for all i ∈ V . Since v ( d j ) : D = e ( d j ) for all j ∈ N ( i ) and I is a zero-padded D × D identity matrix, it holds that:

Due to the existence of self-loops ( i.e. i ∈ N ( i ) for all i ∈ V ), for every d ∈ [ D ] the product ∏ i ∈V ,j ∈N ( i ) e ( d j ) d includes each of e ( d 1 ) d , . . . , e ( d |V| ) d at least once. Consequently, ∏ i ∈V ,j ∈N ( i ) e ( d j ) d = 1 if d 1 = · · · = d |V| = d and ∏ i ∈V ,j ∈N ( i ) e ( d j ) d = 0 otherwise. This implies that f ( θ, G ) ( v ( d 1 ) , . . . , v ( d |V| ) ) = 1 if d 1 = · · · = d |V| and f ( θ, G ) ( v ( d 1 ) , . . . , v ( d |V| ) ) = 0 otherwise, for all d 1 , . . . , d |V| ∈ [ D ] .

Equation (27) implies that q B ( f ( θ, G ) ) ; I y has exactly D non-zero entries, each in a different row and column. Thus, rank q B ( f ( θ, G ) ) ; I y = D . Recalling that α C ∗ := D 1 /ρ 0 ( C ∗ , V ) for L = 1 , we conclude:

Case of L ≥ 2 : Let M := ( ( D ρ L -1 ( C ∗ , V ) ) ) = ( D + ρ L -1 ( C ∗ , V ) -1 ρ L -1 ( C ∗ , V ) ) be the multiset coefficient of D and ρ L -1 ( C ∗ , V ) (recall D := min { D x , D h } ). By Lemma 7, there exists Z ∈ R M × D > 0 for which

with glyph[circledot] ρ L -1 ( C ∗ , V ) ( ZZ glyph[latticetop] ) standing for the ρ L -1 ( C ∗ , V ) 'th Hadamard power of ZZ glyph[latticetop] . For this Z , by Lemma 4 below we know that there exist weights θ and template vectors such that q B ( f ( θ, G ) ); I y has an M × M sub-matrix of the form S ( glyph[circledot] ρ L -1 ( C ∗ , V ) ( ZZ glyph[latticetop] )) Q , where S , Q ∈ R M × M are full-rank diagonal matrices. Since the rank of a matrix is at least the rank of any of its sub-matrices:

where the second transition stems from S and Q being full-rank. Applying Lemma 8 to lower bound the multiset coefficient, we have that:

Taking the log of both sides while recalling that α C ∗ := ( D -1) · ρ L -1 ( C ∗ , V ) -1 +1 , we conclude that:

Lemma 4. Suppose that the GNN inducing f ( θ, G ) is of depth L ≥ 2 and that ρ L -1 ( C ∗ , V ) > 0 . For any M ∈ N and matrix with positive entries Z ∈ R M × D > 0 , there exist weights θ and M + 1 template vectors v (1) , . . . , v ( M +1) ∈ R D x such that q B ( f ( θ, G ) ); I y has an M × M sub-matrix S ( glyph[circledot] ρ L -1 ( C ∗ , V ) ( ZZ glyph[latticetop] )) Q , where S , Q ∈ R M × M are full-rank diagonal matrices and glyph[circledot] ρ L -1 ( C ∗ , V ) ( ZZ glyph[latticetop] ) is the ρ L -1 ( C ∗ , V ) 'th Hadamard power of ZZ glyph[latticetop] .

Proof. Consider the weights θ = ( W (1) , . . . , W ( L ) , W ( o ) ) given by:

where I is a zero padded identity matrix, i.e. it holds ones on its diagonal and zeros elsewhere. We define the templates v (1) , . . . , v ( M ) ∈ R D x to be the vectors holding the respective rows of Z in their first D coordinates and zeros in the remaining entries (recall D := min { D x , D h } ). That is, denoting the rows of Z by z (1) , . . . , z ( M ) ∈ R D > 0 , we let v ( m ) : D := z ( m ) and v ( m ) D +1: := 0 for all m ∈ [ M ] . We set all entries of the last template vector to one, i.e. v ( M +1) := (1 , . . . , 1) ∈ R D x .

glyph[negationslash]

Since C ∗ ∈ S ( I ) , i.e. it is an admissible subset of C I (Definition 4), there exist I ′ ⊆ I , J ′ ⊆ I c with no repeating shared neighbors (Definition 3) such that C ∗ = N ( I ′ ) ∩ N ( J ′ ) . Notice that I ′ and J ′ are non-empty as C ∗ = ∅ (this is implied by ρ L -1 ( C ∗ , V ) > 0 ). We focus on the M × M sub-matrix of q B ( f ( θ, G ) ); I y that includes only rows and columns corresponding to evaluations of f ( θ, G ) where all variables indexed by I ′ are assigned the same template vector from v (1) , . . . , v ( M ) , all variables indexed by J ′ are assigned the same template vector from v (1) , . . . , v ( M ) , and all remaining variables are assigned the all-ones template vector v ( M +1) . Denoting this sub-matrix by U ∈ R M × M , it therefore upholds:

for all m,n ∈ [ M ] , where we use ( x ( i ) ← v ( m ) ) i ∈I ′ to denote that input variables indexed by I ′ are assigned the value v ( m ) . To show that U obeys the form S ( glyph[circledot] ρ L -1 ( C ∗ , V ) ( ZZ glyph[latticetop] )) Q for full-rank diagonal S , Q ∈ R M × M , we prove there exist φ, ψ : R D x → R > 0 such that U m,n = φ ( v ( m ) ) 〈 z ( m ) , z ( n ) 〉 ρ L -1 ( C ∗ , V ) ψ ( v ( n ) ) for all m,n ∈ [ M ] . Indeed, defining S to hold φ ( v (1) ) , . . . , φ ( v ( M ) ) on its diagonal and Q to hold ψ ( v (1) ) , . . . , ψ ( v ( M ) ) on its diagonal, we have that U = S ( glyph[circledot] ρ L -1 ( C ∗ , V ) ( ZZ glyph[latticetop] )) Q . Since S and Q are clearly full-rank (diagonal matrices with non-zero entries on their diagonal), the proof concludes.

For m,n ∈ [ M ] , let h ( l,i ) ∈ R D h be the hidden embedding for i ∈ V at layer l ∈ [ L ] of the GNN inducing f ( θ, G ) , over the following assignment to its input variables ( i.e. vertex features):

Invoking Lemma 10 with v ( m ) , v ( n ) , I ′ , and J ′ , for all i ∈ V it holds that:

for some φ ( L,i ) , ψ ( L,i ) : R D x → R > 0 . Since

and W ( o ) = (1 , 0 , . . . , 0) , this implies that:

Rearranging the last term leads to:

Let φ : v ↦→ ∏ i ∈V φ ( L,i ) ( v ) and ψ : v ↦→ ∏ i ∈V ψ ( L,i ) ( v ) . Noticing that their range is indeed R > 0 and that ∑ i ∈V ρ L -1 ( C ∗ , { i } ) = ρ L -1 ( C ∗ , V ) yields the sought-after expression for U m,n :

I.3.2 Weights and Template Vectors Assignment for Vertex Prediction (Proof of Equation (9))

This part of the proof follows a line similar to that of Appendix I.3.1, with differences stemming from the distinction between the operation of a GNN over graph and vertex prediction. Namely, we construct weights θ and template vectors satisfying log(rank q B ( f ( θ, G ,t ) ) ; I y ) ≥ log( α C ∗ t ,t ) · ρ L -1 ( C ∗ t , { t } ) , where C ∗ t ∈ argmax C∈S ( I ) log( α C ,t ) · ρ L -1 ( C , { t } ) .

If ρ L -1 ( C ∗ t , { t } ) = 0 , then the claim is trivial since there exist weights and template vectors for which q B ( f ( θ, G ,t ) ) ; I y is not the zero matrix ( e.g. taking all weight matrices to be zero-padded identity matrices and choosing a single template vector holding one in its first entry and zeros elsewhere).

glyph[negationslash]

Now, assuming that ρ L -1 ( C ∗ t , { t } ) > 0 , which in particular implies that I glyph[negationslash] = ∅ , I glyph[negationslash] = V , and C ∗ t = ∅ , we begin with the case of GNN depth L = 1 , after which we treat the more general L ≥ 2 case.

Case of L = 1 : Consider the weights θ = ( W (1) , W ( o ) ) given by W (1) := I ∈ R D h × D x and W ( o ) := (1 , . . . , 1) ∈ R 1 × D h , where I is a zero padded identity matrix, i.e. it holds ones on its diagonal and zeros elsewhere. We choose template vectors v (1) , . . . , v ( D ) ∈ R D x such that v ( m ) holds the m 'th standard basis vector of R D in its first D coordinates and zeros in the remaining entries, for m ∈ [ D ] (recall D := min { D x , D h } ). Namely, denote by e (1) , . . . , e ( D ) ∈ R D the standard basis vectors of R D , i.e. e ( m ) d = 1 if d = m and e ( m ) d = 0 otherwise for all m,d ∈ [ D ] . We let v ( m ) : D := e ( m ) and v ( m ) D +1: := 0 for all m ∈ [ D ] .

We prove that for this choice of weights and template vectors, for all d 1 , . . . , d |V| ∈ [ D ] :

To see it is so, notice that:

with h (1 ,t ) = glyph[circledot] j ∈N ( t ) ( W (1) v ( d j ) ) = glyph[circledot] j ∈N ( t ) ( Iv ( d j ) ) . Since v ( d j ) : D = e ( d j ) for all j ∈ N ( t ) and I is a zero-padded D × D identity matrix, it holds that:

For every d ∈ [ D ] we have that ∏ j ∈N ( t ) e ( d j ) d = 1 if d j = d for all j ∈ N ( t ) and ∏ j ∈N ( t ) e ( d j ) d = 0 otherwise. This implies that f ( θ, G ,t ) ( v ( d 1 ) , . . . , v ( d |V| ) ) = 1 if d j = d j ′ for all j, j ′ ∈ N ( t ) and f ( θ, G ,t ) ( v ( d 1 ) , . . . , v ( d |V| ) ) = 0 otherwise, for all d 1 , . . . , d |V| ∈ [ D ] .

Equation (28) implies that q B ( f ( θ, G ,t ) ) ; I y has a sub-matrix of rank D . Specifically, such a sub-matrix can be obtained by examining all rows and columns of q B ( f ( θ, G ,t ) ) ; I y corresponding to some fixed indices ( d i ∈ [ D ]) i ∈V\N ( t ) for the vertices that are not neighbors of t . Thus, rank q B ( f ( θ, G ,t ) ) ; I y ≥ D . Notice that necessarily ρ 0 ( C ∗ t , { t } ) = 1 , as it is not zero and there can only be one length zero walk to t (the trivial walk that starts and ends at t ). Recalling that α C ∗ t ,t := D for L = 1 , we therefore conclude:

Case of L ≥ 2 : Let M := ( ( D ρ L -1 ( C ∗ t , { t } ) ) ) = ( D + ρ L -1 ( C ∗ t , { t } ) -1 ρ L -1 ( C ∗ t , { t } ) ) be the multiset coefficient of D and ρ L -1 ( C ∗ t , { t } ) (recall D := min { D x , D h } ). By Lemma 7, there exists Z ∈ R M × D > 0 for which

with glyph[circledot] ρ L -1 ( C ∗ t , { t } ) ( ZZ glyph[latticetop] ) standing for the ρ L -1 ( C ∗ t , { t } ) 'th Hadamard power of ZZ glyph[latticetop] . For this Z , by Lemma 5 below we know that there exist weights θ and template vectors such that q B ( f ( θ, G ,t ) ); I y has an M × M sub-matrix of the form S ( glyph[circledot] ρ L -1 ( C ∗ t , { t } ) ( ZZ glyph[latticetop] )) Q , where S , Q ∈ R M × M are full-rank diagonal matrices. Since the rank of a matrix is at least the rank of any of its sub-matrices:

where the second transition is due to S and Q being full-rank. Applying Lemma 8 to lower bound the multiset coefficient, we have that:

Taking the log of both sides while recalling that α C ∗ t ,t := ( D -1) · ρ L -1 ( C ∗ t , { t } ) -1 +1 , we conclude that:

Lemma 5. Suppose that the GNN inducing f ( θ, G ,t ) is of depth L ≥ 2 and that ρ L -1 ( C ∗ t , { t } ) > 0 . For any M ∈ N and matrix with positive entries Z ∈ R M × D > 0 , there exist weights θ and M + 1 template vectors v (1) , . . . , v ( M +1) ∈ R D x such that q B ( f ( θ, G ,t ) ); I y has an M × M sub-matrix S ( glyph[circledot] ρ L -1 ( C ∗ t , { t } ) ( ZZ glyph[latticetop] )) Q , where S , Q ∈ R M × M are full-rank diagonal matrices and glyph[circledot] ρ L -1 ( C ∗ t , { t } ) ( ZZ glyph[latticetop] ) is the ρ L -1 ( C ∗ t , { t } ) 'th Hadamard power of ZZ glyph[latticetop] .

Proof. Consider the weights θ = ( W (1) , . . . , W ( L ) , W ( o ) ) defined by:

where I is a zero padded identity matrix, i.e. it holds ones on its diagonal and zeros elsewhere. We let the templates v (1) , . . . , v ( M ) ∈ R D x be the vectors holding the respective rows of Z in their first D coordinates and zeros in the remaining entries (recall D := min { D x , D h } ). That is, denoting the rows of Z by z (1) , . . . , z ( M ) ∈ R D > 0 , we let v ( m ) : D := z ( m ) and v ( m ) D +1: := 0 for all m ∈ [ M ] . We set all entries of the last template vector to one, i.e. v ( M +1) := (1 , . . . , 1) ∈ R D x .

glyph[negationslash]

Since C ∗ t ∈ S ( I ) , i.e. it is an admissible subset of C I (Definition 4), there exist I ′ ⊆ I , J ′ ⊆ I c with no repeating shared neighbors (Definition 3) such that C ∗ t = N ( I ′ ) ∩ N ( J ′ ) . Notice that I ′ and J ′ are non-empty as C ∗ t = ∅ (this is implied by ρ L -1 ( C ∗ t , { t } ) > 0 ). We focus on the M × M

sub-matrix of q B ( f ( θ, G ,t ) ); I y that includes only rows and columns corresponding to evaluations of f ( θ, G ,t ) where all variables indexed by I ′ are assigned the same template vector from v (1) , . . . , v ( M ) , all variables indexed by J ′ are assigned the same template vector from v (1) , . . . , v ( M ) , and all remaining variables are assigned the all-ones template vector v ( M +1) . Denoting this sub-matrix by U ∈ R M × M , it therefore upholds:

for all m,n ∈ [ M ] , where we use ( x ( i ) ← v ( m ) ) i ∈I ′ to denote that input variables indexed by I ′ are assigned the value v ( m ) . To show that U obeys the form S ( glyph[circledot] ρ L -1 ( C ∗ t , { t } ) ( ZZ glyph[latticetop] )) Q for full-rank diagonal S , Q ∈ R M × M , we prove there exist φ, ψ : R D x → R > 0 such that U m,n = φ ( v ( m ) ) 〈 z ( m ) , z ( n ) 〉 ρ L -1 ( C ∗ t , { t } ) ψ ( v ( n ) ) for all m,n ∈ [ M ] . Indeed, defining S to hold φ ( v (1) ) , . . . , φ ( v ( M ) ) on its diagonal and Q to hold ψ ( v (1) ) , . . . , ψ ( v ( M ) ) on its diagonal, we have that U = S ( glyph[circledot] ρ L -1 ( C ∗ t , { t } ) ( ZZ glyph[latticetop] )) Q . Since S and Q are clearly full-rank (diagonal matrices with non-zero entries on their diagonal), the proof concludes.

For m,n ∈ [ M ] , let h ( l,i ) ∈ R D h be the hidden embedding for i ∈ V at layer l ∈ [ L ] of the GNN inducing f ( θ, G ,t ) , over the following assignment to its input variables ( i.e. vertex features):

Invoking Lemma 10 with v ( m ) , v ( n ) , I ′ , and J ′ , it holds that:

for some φ ( L,t ) , ψ ( L,t ) : R D x → R > 0 . Since

and W ( o ) = (1 , 0 , . . . , 0) , this implies that:

Defining φ := φ ( L,t ) and ψ := ψ ( L,t ) leads to the sought-after expression for U m,n :

I.3.3 Technical Lemmas

For completeness, we include the vector rearrangement inequality from [61], which we employ for proving the subsequent Lemma 7.

Lemma 6 (Lemma 1 from [61]) . Let a (1) , . . . , a ( M ) ∈ R D ≥ 0 be M ∈ N different vectors with nonnegative entries. Then, for any permutation σ : [ M ] → [ M ] besides the identity permutation it holds that:

Taking the P 'th Hadamard power of a rank at most D matrix results in a matrix whose rank is at most the multiset coefficient ( ( D P ) ) := ( D + P -1 P ) (see, e.g. , Theorem 1 in [2]). Lemma 7, adapted from Appendix B.2 in [64], guarantees that we can always find a ( ( D P ) ) × D matrix Z with positive entries such that rank( glyph[circledot] P ( ZZ glyph[latticetop] ) ) is maximal, i.e. equal to ( ( D P ) ) .

Lemma 7 (adapted from Appendix B.2 in [64]) . For any D ∈ N and P ∈ N ≥ 0 , there exists a matrix with positive entries Z ∈ R ( ( D P ) ) × D > 0 for which:

where glyph[circledot] P ( ZZ glyph[latticetop] ) is the P 'th Hadamard power of ZZ glyph[latticetop] .

Proof. We let M := ( ( D P ) ) for notational convenience. Denote by z (1) , . . . , z ( M ) ∈ R D the row vectors of Z ∈ R M × D > 0 . Observing the ( m,n ) 'th entry of glyph[circledot] P ( ZZ glyph[latticetop] ) :

by expanding the power using the multinomial identity we have that:

where in the last equality we separated terms depending on m from those depending on n .

Let ( a ( q 1 ,...,q D ) ∈ R M ) q 1 ,...,q D ∈ N ≥ 0 s.t ∑ D d =1 q d = P be M vectors defined by a ( q 1 ,...,q D ) m = ∏ D d =1 ( z ( m ) d ) q d for all q 1 , . . . , q D ∈ N ≥ 0 satisfying ∑ D d =1 q d = P and m ∈ [ M ] . As can be seen from Equation (29), we can write:

where A ∈ R M × M is the matrix whose columns are ( a ( q 1 ,...,q D ) ) q 1 ,...,q D ∈ N ≥ 0 s.t ∑ D d =1 q d = P and S ∈ R M × M is the diagonal matrix holding ( P q 1 ,...,q D ) for every q 1 , . . . , q D ∈ N ≥ 0 satisfying ∑ D d =1 q d = P on its diagonal. Since all entries on the diagonal of S are positive, it is of full-rank, i.e. rank( S ) = M . Thus, to prove that there exists Z ∈ R M × D > 0 for which rank( glyph[circledot] P ( ZZ glyph[latticetop] )) = M , it suffices to show that we can choose z (1) , . . . , z ( M ) with positive entries inducing rank( A ) = M , for A as defined above. Below, we complete the proof by constructing such z (1) , . . . , z ( M ) .

We associate each of z (1) , . . . , z ( M ) with a different configuration from the set:

where note that this set contains M = ( ( D P ) ) elements. For m ∈ [ M ] , denote by q ( m ) the configuration associated with z ( m ) . For a variable γ ∈ R , to be determined later on, and every m ∈ [ M ] and d ∈ [ D ] , we set:

Given these z (1) , . . . , z ( M ) , the entries of A have the following form:

glyph[negationslash]

for all m,n ∈ [ M ] . Thus, det( A ) = ∑ permutation σ :[ M ] → [ M ] sign( σ ) · γ ∑ M m =1 〈 q ( m ) , q ( σ ( m )) 〉 is polynomial in γ . By Lemma 6, ∑ M m =1 〈 q ( m ) , q ( σ ( m )) 〉 < ∑ M m =1 ‖ q ( m ) ‖ 2 for all σ which is not the identity permutation. This implies that ∑ M m =1 ‖ q ( m ) ‖ 2 is the maximal degree of a monomial in det( A ) , and it is attained by a single element in ∑ permutation σ :[ M ] → [ M ] sign( σ ) · γ ∑ M m =1 〈 q ( m ) , q ( σ ( m )) 〉 -that corresponding to the identity permutation. Consequently, det( A ) cannot be the zero polynomial with respect to γ , and so it vanishes only on a finite set of values for γ . In particular, there exists γ > 0 such that det( A ) = 0 , meaning rank( A ) = M . The proof concludes by noticing that for a positive γ the entries of the chosen z (1) , . . . , z ( M ) are positive as well.

Additionally, we make use of the following lemmas.

Lemma 8. For any D,P ∈ N , let ( ( D P ) ) := ( D + P -1 P ) be the multiset coefficient. Then:

Proof. For any N ≥ K ∈ N , a known lower bound on the binomial coefficient is ( N K ) ≥ ( N K ) K . Hence:

Lemma 9. For D 1 , D 2 , K ∈ N , consider a polynomial function mapping variables θ ∈ R K to matrices A ( θ ) ∈ R D 1 × D 2 , i.e. the entries of A ( θ ) are polynomial in θ . If there exists a point θ ∗ ∈ R K such that rank( A ( θ ∗ )) ≥ R , for R ∈ [min { D 1 , D 2 } ] , then the set { θ ∈ R K : rank( A ( θ )) < R } has Lebesgue measure zero.

Proof. A matrix is of rank at least R if and only if it has a R × R sub-matrix whose determinant is non-zero. The determinant of any sub-matrix of A ( θ ) is polynomial in the entries of A ( θ ) , and so it is polynomial in θ as well. Since the zero set of a polynomial is either the entire space or a set of Lebesgue measure zero [19], the fact that rank( A ( θ ∗ )) ≥ R implies that { θ ∈ R K : rank( A ( θ )) < R } has Lebesgue measure zero.

Lemma 10. Let v , v ′ ∈ R D x ≥ 0 whose first D := min { D x , D h } entries are positive, and disjoint I ′ , J ′ ⊆ V with no repeating shared neighbors (Definition 3). Denote by h ( l,i ) ∈ R D h the hidden embedding for i ∈ V at layer l ∈ [ L ] of a GNN with depth L ≥ 2 and product aggregation (Equations (2) and (5) ), given the following assignment to its input variables (i.e. vertex features):

where 1 ∈ R D x is the vector holding one in all entries. Suppose that the weights W (1) , . . . , W ( L ) of the GNN are given by:

where I is a zero padded identity matrix, i.e. it holds ones on its diagonal and zeros elsewhere. Then, for all l ∈ { 2 , . . . , L } and i ∈ V , there exist φ ( l,i ) , ψ ( l,i ) : R D x → R > 0 such that:

Proof. The proof is by induction over the layer l ∈ { 2 , . . . , L } . For l = 2 , fix i ∈ V . By the update rule of a GNN with product aggregation:

Plugging in the value of W (2) we get:

Let ¯ v , ¯ v ′ ∈ R D h be the vectors holding v : D and v ′ : D in their first D coordinates and zero in the remaining entries, respectively. Similarly, we use ¯ 1 ∈ R D h to denote the vector whose first D entries are one and the remaining are zero. Examining h (1 ,j ) for j ∈ N ( i ) , by the assignment of input variables and the fact that W (1) is a zero padded identity matrix we have that:

Since the first D entries of ¯ v and ¯ v ′ are positive while the rest are zero, the same holds for h (1 ,j ) . Additionally, recall that I ′ and J ′ have no repeating shared neighbors. Thus, if j ∈ N ( I ′ ) ∩N ( J ′ ) = C , then j has a single neighbor in I ′ and a single neighbor in J ′ , implying h (1 ,j ) = ¯ v glyph[circledot] ¯ v ′ . Otherwise, if j / ∈ C , then N ( j ) ∩ I ′ = ∅ or N ( j ) ∩ J ′ = ∅ must hold. In the former h (1 ,j ) does not depend on v , whereas in the latter h (1 ,j ) does not depend on v ′ .

Going back to Equation (30), while noticing that |N ( i ) ∩ C| = ρ 1 ( C , { i } ) , we arrive at:

As discussed above, for each j ∈ N ( i ) \ C the hidden embedding h (1 ,j ) does not depend on v or it does not depend on v ′ . Furthermore, ∑ D h d =1 h (1 ,j ) d > 0 for all j ∈ N ( i ) . Hence, there exist φ (2 ,i ) , ψ (2 ,i ) : R D x → R > 0 such that:

completing the base case.

Now, assuming that the inductive claim holds for l -1 ≥ 2 , we prove that it holds for l . Let i ∈ V . By the update rule of a GNN with product aggregation h ( l,i ) = glyph[circledot] j ∈N ( i ) ( W ( l ) h ( l -1 ,j ) ) . Plugging in the value of W ( l ) we get:

By the inductive assumption h ( l -1 ,j ) 1 = φ ( l -1 ,j ) ( v ) 〈 v : D , v ′ : D 〉 ρ l -2 ( C , { j } ) ψ ( l -1 ,j ) ( v ′ ) for all j ∈ N ( i ) , where φ ( l -1 ,j ) , ψ ( l -1 ,j ) : R D x → R > 0 . Thus:

Define φ ( l,i ) : v ↦→ ∏ j ∈N ( i ) φ ( l -1 ,j ) ( v ) and ψ ( l,i ) : v ′ ↦→ ∏ j ∈N ( i ) ψ ( l -1 ,j ) ( v ′ ) . Since the range of φ ( l -1 ,j ) and ψ ( l -1 ,j ) is R > 0 for all j ∈ N ( i ) , so is the range of φ ( l,i ) and ψ ( l,i ) . The desired result thus readily follows by noticing that ∑ j ∈N ( i ) ρ l -2 ( C , { j } ) = ρ l -1 ( C , { i } ) :

I.4 Proof of Theorem 4

The proof follows a line identical to that of Theorem 2 (Appendix I.2), requiring only slight adjustments. We outline the necessary changes.

Extending the tensor network representations of GNNs with product aggregation to directed graphs and multiple edge types is straightforward. Nodes, legs, and leg weights are as described in Appendix E for undirected graphs with a single edge type, except that:

Legs connecting δ -tensors with weight matrices in the same layer are adapted such that only incoming neighbors are considered. Formally, in Equations (15) to (20), N ( i ) is replaced by N in ( i ) in the leg definitions, for i ∈ V .

Weight matrices ( W ( l,q ) ) l ∈ [ L ] ,q ∈ [ Q ] are assigned to nodes in accordance with edge types. Namely, if at layer l ∈ [ L ] a δ -tensor associated with i ∈ V is connected to a weight matrix associated with j ∈ N in ( i ) , then W ( l,τ ( j,i )) is assigned to the weight matrix node, as opposed to W ( l ) in the single edge type setting. Formally, let W ( l,j,γ ) be a node at layer l ∈ [ L ] connected to δ ( l,i,γ ′ ) , for i ∈ V , j ∈ N in ( i ) , and some γ, γ ′ ∈ N . Then, W ( l,j,γ ) stands for a copy of W ( l,τ ( j,i )) .

For X = ( x (1) , . . . , x ( |V| ) ) ∈ R D x ×|V| , let T ( X ) and T ( t ) ( X ) be the tensor networks corresponding to f ( θ, G ) ( X ) and f ( θ, G ,t ) ( X ) , respectively, whose construction is outlined above. Then, Lemma 1 (from Appendix I.2.1) and its proof apply as stated. Meaning, sep( f ( θ, G ) ; I ) and sep( f ( θ, G ,t ) ; I ) are upper bounded by the minimal modified multiplicative cut weights in T ( X ) and T ( t ) ( X ) , respectively, among cuts separating leaves associated with vertices of the input graph in I from leaves associated with vertices of the input graph in I c . Therefore, to establish Equations (11) and (12), it suffices to find cuts in the respective tensor networks with sufficiently low modified multiplicative weights. As is the case for undirected graphs with a single edge type (see Appendices I.2.2 and I.2.3), the cuts separating nodes corresponding to vertices in I from all other nodes yield the desired upper bounds.

I.5 Proof of Theorem 5

The proof follows a line identical to that of Theorem 3 (Appendix I.3), requiring only slight adjustments. We outline the necessary changes.

In the context of graph prediction, let C ∗ ∈ argmax C∈S → ( I ) log( α C ) · ρ L -1 ( C , V ) . By Lemma 3 (from Appendix I.3), to prove that Equation (13) holds for weights θ , it suffices to find template vectors for which log(rank q B ( f ( θ, G ) ) ; I y ) ≥ log( α C ∗ ) · ρ L -1 ( C ∗ , V ) . Notice that, since the outputs of f ( θ, G ) vary polynomially with θ , so do the entries of q B ( f ( θ, G ) ) ; I y for any choice of template vectors. Thus, according to Lemma 9 (from Appendix I.3.3), by constructing weights θ and template vectors satisfying log(rank q B ( f ( θ, G ) ) ; I y ) ≥ log( α C ∗ ) · ρ L -1 ( C ∗ , V ) , we may conclude that this is the case for almost all assignments of weights, meaning Equation (13) holds for almost all assignments of weights. For undirected graphs with a single edge type, Appendix I.3.1 provides such weights W (1) , . . . , W ( L ) , W ( o ) and template vectors. The proof in the case of directed graphs with multiple edge types is analogous, requiring only a couple adaptations: (i) weight matrices of all edge types at layer l ∈ [ L ] are set to the W ( l ) chosen in Appendix I.3.1; and (ii) C I and S ( I ) are replaced with their directed counterparts C I → and S → ( I ) , respectively.

In the context of vertex prediction, let C ∗ t ∈ argmax C∈S → ( I ) log( α C ,t ) · ρ L -1 ( C , { t } ) . Due to arguments similar to those above, to prove that Equation (14) holds for almost all assignments of weights, we need only find weights θ and template vectors satisfying log(rank q B ( f ( θ, G ,t ) ) ; I y ) ≥ log( α C ∗ t ,t ) · ρ L -1 ( C ∗ t , { t } ) . For undirected graphs with a single edge type, Appendix I.3.2 provides such weights and template vectors. The adaptations necessary to extend Appendix I.3.2 to directed graphs with multiple edge types are identical to those specified above for extending Appendix I.3.1 in the context of graph prediction.

Lastly, recalling that a finite union of measure zero sets has measure zero as well establishes that Equations (13) and (14) jointly hold for almost all assignments of weights.

I.6 Proof of Proposition 1

We first prove that the contractions described by T ( X ) produce f ( θ, G ) ( X ) . Through an induction over the layer l ∈ [ L ] , for all i ∈ V and γ ∈ [ ρ L -l ( { i } , V )] we show that contracting the sub-tree whose root is δ ( l,i,γ ) yields h ( l,i ) -the hidden embedding for i at layer l of the GNN inducing f ( θ, G ) , given vertex features x (1) , . . . , x ( |V| ) .

For l = 1 , fix some i ∈ V and γ ∈ [ ρ L -1 ( { i } , V )] . The sub-tree whose root is δ (1 ,i,γ ) comprises |N ( i ) | copies of W (1) , each associated with some j ∈ N ( i ) and contracted in its second mode with a copy of x ( j ) . Additionally, δ (1 ,i,γ ) , which is a copy of δ ( |N ( i ) | +1) , is contracted with the copies of W (1) in their first mode. Overall, the execution of all contractions in the sub-tree can be written as δ ( |N ( i ) | +1) ∗ j ∈ [ |N ( i ) | ] ( W (1) x ( N ( i ) j ) ) , where N ( i ) j , for j ∈ [ |N ( i ) | ] , denotes the j 'th neighbor of i according to an ascending order (recall vertices are represented by indices from 1 to |V| ). The base case concludes by Lemma 11:

Assuming that the inductive claim holds for l -1 ≥ 1 , we prove that it holds for l . Let i ∈ V and γ ∈ [ ρ L -l ( { i } , V )] . The children of δ ( l,i,γ ) in the tensor network are of the form W ( l, N ( i ) j ,φ l,i,j ( γ )) , for j ∈ [ |N ( i ) | ] , and each W ( l, N ( i ) j ,φ l,i,j ( γ )) is connected in its other mode to δ ( l -1 , N ( i ) j ,φ l,i,j ( γ )) . By the inductive assumption for l -1 , we know that performing all contractions in the sub-tree whose root is δ ( l -1 , N ( i ) j ,φ l,i,j ( γ )) produces h ( l -1 , N ( i ) j ) , for all j ∈ [ |N ( i ) | ] . Since δ ( l,i,γ ) is a copy of δ ( |N ( i ) | +1) , and each W ( l, N ( i ) j ,φ l,i,j ( γ )) is a copy of W ( l ) , the remaining contractions in the sub-tree of δ ( l,i,γ ) thus give:

which according to Lemma 11 amounts to:

establishing the induction step.

With the inductive claim at hand, we show that contracting T ( X ) produces f ( θ, G ) ( X ) . Applying the inductive claim for l = L , we have that h ( L, 1) , . . . , h ( L, |V| ) are the vectors produced by executing all contractions in the sub-trees whose roots are δ ( L, 1 , 1) , . . . , δ ( L, |V| , 1) , respectively. Performing the remaining contractions, defined by the legs of δ ( |V| +1) , therefore yields W ( o ) ( δ ( |V| +1) ∗ i ∈ [ |V| ] h ( L,i ) ) . By Lemma 11:

Hence, W ( o ) ( δ ( |V| +1) ∗ i ∈ [ |V| ] h ( L,i ) ) = W ( o ) ( glyph[circledot] i ∈ [ |V| ] h ( L,i ) ) = f ( θ, G ) ( X ) , meaning contracting T ( X ) results in f ( θ, G ) ( X ) .

An analogous proof establishes that the contractions described by T ( t ) ( X ) yield f ( θ, G ,t ) ( X ) . Specifically, the inductive claim and its proof are the same, up to γ taking values in [ ρ L -l ( { i } , { t } )] instead of [ ρ L -l ( { i } , V )] , for l ∈ [ L ] . This implies that h ( L,t ) is the vector produced by contracting the sub-tree whose root is δ ( L,t, 1) . Performing the only remaining contraction, defined by the leg connecting δ ( L,t, 1) with W ( o ) , thus results in W ( o ) h ( L,t ) = f ( θ, G ,t ) ( X ) .

I.6.1 Technical Lemma

Lemma 11. Let δ ( N +1) ∈ R D ×···× D be an order N + 1 ∈ N tensor that has ones on its hyperdiagonal and zeros elsewhere, i.e. δ ( N +1) d 1 ,...,d N +1 = 1 if d 1 = · · · = d N +1 and δ ( N +1) d 1 ,...,d N +1 = 0 otherwise, for all d 1 , . . . , d N +1 ∈ [ D ] . Then, for any x (1) , . . . , x ( N ) ∈ R D it holds that δ ( N +1) ∗ i ∈ [ N ] x ( i ) = glyph[circledot] i ∈ [ N ] x ( i ) ∈ R D .

Proof. By the definition of tensor contraction (Definition 7), for all d ∈ [ D ] we have that:

Peer reviews:
comments: SUMMARY:
The paper studies how to characterize the ability of Graph Neural Networks (GNNs) to model the interaction given a partition of vertices. They quanlify the interaction strength with separation rank, and prove that it is governed by walk index (number of walks starting from the partition boundary). This relation is empirically demonstrated in their experiments, where a GNN with higher walk index beats the one with lower Walk Index. Lastly, Walk Index Sparsification is proposed to perform edge deletion based on Walk Index, and it markedly outperforms other baselines in terms of preserving prediction accuracy.

STRENGTHS:
- An overall interesting theoretical work that supports the intuition: the interaction between two parts in a graph depends on how much interconnected their boundary is.  
- The orginization is good, the theoretical analysis looks sound and the experiments are solid. The application to edge sparsification is impressive.

WEAKNESSES:
- Walk Index Sparsification is unable to scale up to large graphs with deep GNNs, given its $O(|E||V|^3\log(L))$ complexity. Correspondingly, the experiments only covered 2-WIS and 1-WIS. To further improve, the authors could discuss some potentially efficient implementations.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
As stated in Weaknesses, the computational overhead is a concern.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new way to analyze the capacity of GNNs based on the complexity of interactions they can model across a partitioning of the nodes. In particular, this complexity is quantified via *separation rank*, and, for a certain kind of GNN with product aggregation, this rank is proved to scale with *walk index*, the number of random walks originating from a vertex on the partition's boundary. Experiments indicate that this finding generalizes beyond the particular GNN in the proofs. First, three different GNNs are shown to perform better at an image classification task when the data are converted to a graph so as to maximize walk index across the relevant partition. Second, the paper proposes WIS, a way to reduce the number of edges in the graph while maintaining GNN performance, based on maintaining high walk indices. WIS is shown to outperform prior methods on several real-world networks.\n\nSTRENGTHS:\n- Conceptual and theoretical explanations on the capacity of GNNs are an important topic, and this paper takes a fresh approach as compared to much of the WL-isomorphism-based work.\n- The paper is well-organized, and the writing is grammatical and easy-to-read.\n- Experiments show the superior efficacy of the proposed WIS method for graph sparsification relative to prior methods.\n\nWEAKNESSES:\n- While the machinery of the proof appears to be quite complex, I would appreciate some description of the proof concept in the main paper, even if it is at a very high level.\n- Related to the prior point on the proof concept, I am not totally convinced that walk index is the fundamental quantity here. Particularly the results in Figure 3 for Cora, where 2-WIS's performance is essentially matched by 1-WIS (which simply relates to node degrees), suggest that there may be a simpler explanation. To convince the reader, perhaps there could be more conceptual description of walk index and separation rank, and why the two are fundamentally linked. Or perhaps there could be more experiments showing that walk index is more predictive of performance as compared to other notions of connectivity.\n\nQUESTIONS:\n- Do you see this work as totally orthogonal to WL-isomorphism-based limitation work? As I see it, those results are all-or-nothing, in the sense that they show GNNs either can or cannot distinguish two kinds of graphs. The bound here instead is somehow softer and more nuanced, showing that the complexity at which the GNN can model some interaction (as quantified by separation rank) scales with walk index. Is there some sense in which, or some specific cases for which, the WL results are subsumed by your new results?\n\n- Related to the second 'weakness' above, do you see walk index as the fundamental notion here? If so, why?\n\nLIMITATIONS:\nSome limitations are noted throughout the work -- for example, it is noted that the theoretical results only formally apply to GNNs with product aggregation. A more collected discussion of limitations of the current analysis/understanding, and possible future directions, would be helpful.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to analyze the expressivity of graph neural networks (GNNs) by their ability to model interactions. The authors consider GNNs with product aggregation scheme, and analyze their ability to model interactions for a given partition of the graph through the notion of separation rank. The authors show quantitatively that these GNNs can model interactions better for partitions with higher walk index (Theorem 1). Motivated from their theoretical analysis, the authors propose an edge sparsification algorithm --- Walk Index Sparsification (WIS, Algorithm 1, 2) --- to drop edges that have least amount of impact on GNN's ability to model interations for a given partition in a given graph. The utility of WIS is demonstrated on both graph-level prediction and vertex-level prediction tasks.\n\nSTRENGTHS:\nStrengths:\n\n1. The proposal to study the expressivity of GNNs via their ability to model vertex interactions is interesting. \n\n2. The theoretical analysis is novel: by focusing on GNNs with product aggregation, the authors connect them to tensor networks, of which the notion of separation rank can be used to study the network's ability to model interactions. \n\n3. The paper is overall well-written and easy to follow, where important technical details and illustrative examples are explained clearly in the supplementary materials.\n\nWEAKNESSES:\nWeaknesses:\n\n1. The theoretical analysis focus on a special kind of GNNs with product aggregation, whereas the empirical evaluations are all done using standard GNNs with mean aggregation with ReLU nonlinearity. Although the authors meant to show that theory results can carry over to standard GNNs, they do not explain explicitly how such connections can be drawn. If the main contribution of the paper is...\n(a) to show that using separation rank to understand GNNs is relevant, then the authors should outline how the analysis carries forward to standard GNNs\n(b) to show that walk index of partitions in a graph determines GNN performance, then the authors should examine the case where the original graph is replaced with a fully-connected graph, and discuss GNNs with Graph Transformers\n(c) to propose an edge sparsification algorithm, then comparisons with stronger baselines are needed, such as DropEdge [1] and curvature-based graph rewriting methods [2].\n\n\n2. The conclusion that GNNs can model interactions better on partitions that have high walk index (i.e. more interconnected) seems to crucially depend on the choice of aggregation function is tied to the graph topology. However, it seems possible to use different aggregation/pooling function that delineates from the graph topology to capture interactions among partitions with low walk index. More explanations on how to interpret the results are needed.\n\n\nReferences:\n\n[1] Rong, Yu, et al. \"Dropedge: Towards deep graph convolutional networks on node classification.\" arXiv preprint arXiv:1907.10903 (2019).\n\n[2] Topping, Jake, et al. \"Understanding over-squashing and bottlenecks on graphs via curvature.\" arXiv preprint arXiv:2111.14522 (2021).\n\nQUESTIONS:\n1. The analysis focus on GNNs with product aggregation, i.e. multiplication of the neighborhood node's embedding vectors $\\mathbf{h}^{(l, i)}=\\odot_{j \\in \\mathcal{N}(i)}\\left(\\mathbf{W}^{(l)} \\mathbf{h}^{(l-1, j)}\\right)$, where $\\mathcal{N}(i)$ depends on the graph topology. In this set-up, the authors show that such GNNs can model interactions among two partitions in a graph better if they are more inter-connected. Yet, earlier work by Cohen and Shashuwa [3] show that for the case of convolutional neural networks (CNNs), their ability to model interactions depend on the choice of pooling functions. In light of this result, it seems plausible for GNNs to model partitions that are less-connected (e.g., Fig 2 -left) if one delineates the aggregation function from the graph topology, which is also commonly used in the literature of graph rewriting (e.g. [2]). Can the authors provide more explicit interpretations of their results and comparison with related work using separation rank for CNNs/RNNs?\n\n2. As a thought example: for GNNs with product aggregation, do they achieve maximally separation rank for a fully-connected graph? How would the WIS algorithm compare to a simple baseline: replacing the original graph with a fully-connected graph, followed by randomly dropping edges (as opposed to (i) edges are removed randomly from the original graph)? This baseline is motivated from [4], where the authors show that replacing the original graph with a fully-connected graph at the output layer can boost GNN performance. \n\n3. In line 264-266, the authors explain that the edge sparsification algorithm should first choose the partitions of interest, whereas in Algorithm 1 and 2, such choice is default to partitions induced by singletons and the rest of the nodes. It seems possible that the optimal partition choice depends on the downstream tasks (e.g., graph-level clustering versus node-level prediction). Can the authors provide more details and recommendations?\n\nReferences:\n\n[3] Cohen, Nadav, and Amnon Shashua. \"Inductive bias of deep convolutional networks through pooling geometry.\" arXiv preprint arXiv:1605.06743 (2016).\n\n[4] Alon, Uri, and Eran Yahav. \"On the bottleneck of graph neural networks and its practical implications.\" arXiv preprint arXiv:2006.05205 (2020).\n\nLIMITATIONS:\nThe limitations were discussed, but can be explained more thoroughly through (1) detailed interpretations of the theoretical results (assumptions and applicability); (2) comparison with prior works that make use of separation rank to analyze expressivity of deep learning architectures.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-22"}