SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Definition 3. We say that  $\mathcal{I},\mathcal{J}\subseteq \mathcal{V}$  have no repeating shared neighbors if every  $k\in \mathcal{N}(\mathcal{I})\cap \mathcal{N}(\mathcal{J})$  has only a single neighbor in each of  $\mathcal{I}$  and  $\mathcal{J}$ , i.e.  $|\mathcal{N}(k)\cap \mathcal{I}| = |\mathcal{N}(k)\cap \mathcal{J}| = 1$ .
Definition 4. For  $\mathcal{I} \subseteq \mathcal{V}$ , we refer to  $\mathcal{C} \subseteq \mathcal{C}_{\mathcal{I}}$  as an admissible subset of  $\mathcal{C}_{\mathcal{I}}$  if there exist  $\mathcal{I}' \subseteq \mathcal{I}, \mathcal{J}' \subseteq \mathcal{I}^c$  with no repeating shared neighbors such that  $\mathcal{C} = \mathcal{N}(\mathcal{I}') \cap \mathcal{N}(\mathcal{J}')$ . We use  $\mathcal{S}(\mathcal{I})$  to denote the set comprising all admissible subsets of  $\mathcal{C}_{\mathcal{I}}$ :
$$
\mathcal {S} (\mathcal {I}) := \left\{\mathcal {C} \subseteq \mathcal {C} _ {\mathcal {I}}: \mathcal {C} \text {i s a n a d m i s s i b l e s u b s e t o f} \mathcal {C} _ {\mathcal {I}} \right\}.
$$
Theorem 3 below establishes that almost all possible values for the network's weights lead the upper bounds in Theorem 2 to be tight, up to logarithmic terms and to the number of walks from  $\mathcal{C}_{\mathcal{I}}$  being replaced with the number of walks from any single  $\mathcal{C} \in S(\mathcal{I})$ . The extent to which  $\mathcal{C}_{\mathcal{I}}$  can be covered by an admissible subset thus determines how tight the upper bounds are. Trivially, at least the shared neighbors of any  $i \in \mathcal{I}, j \in \mathcal{I}^c$  can be covered, since  $\mathcal{N}(i) \cap \mathcal{N}(j) \in S(\mathcal{I})$ . Appendix C shows that for various canonical graphs all of  $\mathcal{C}_{\mathcal{I}}$ , or a large part of it, can be covered by an admissible subset.
Theorem 3. Consider the setting and notation of Theorem 2. Given  $\mathcal{I} \subseteq \mathcal{V}$ , for almost all assignments of weights  $\theta$ , i.e. for all but a set of Lebesgue measure zero, it holds that:
$$
\left(\operatorname {g r a p h} \text {p r e d i c t i o n}\right) \quad \log \left(\operatorname {s e p} \left(f ^ {(\theta , \mathcal {G})}; \mathcal {I}\right)\right) \geq \max  _ {\mathcal {C} \in \mathcal {S} (\mathcal {I})} \log \left(\alpha_ {\mathcal {C}}\right) \cdot \rho_ {L - 1} (\mathcal {C}, \mathcal {V}), \tag {8}
$$
$$
\left(\text {v e r t e x p r e d i c t i o n}\right) \quad \log \left(\operatorname {s e p} \left(f ^ {\left(\theta , \mathcal {G}, t\right)}; \mathcal {I}\right)\right) \geq \max  _ {\mathcal {C} \in \mathcal {S} (\mathcal {I})} \log \left(\alpha_ {\mathcal {C}, t}\right) \cdot \rho_ {L - 1} (\mathcal {C}, \{t \}), \tag {9}
$$
$$
\alpha_ {\mathcal {C}} := \left\{ \begin{array}{l l} D ^ {1 / \rho_ {0} (\mathcal {C}, \mathcal {V})} & , i f L = 1 \\ (D - 1) \cdot \rho_ {L - 1} (\mathcal {C}, \mathcal {V}) ^ {- 1} + 1 & , i f L \geq 2 \end{array} \right.,
$$
$$
\alpha_ {\mathcal {C}, t} := \left\{ \begin{array}{l l} D & , i f L = 1 \\ (D - 1) \cdot \rho_ {L - 1} (\mathcal {C}, \{t \}) ^ {- 1} + 1 & , i f L \geq 2 \end{array} \right.,
$$
with  $D \coloneqq \min \{D_x, D_h\}$ . If  $\rho_{L-1}(\mathcal{C}, \mathcal{V}) = 0$  or  $\rho_{L-1}(\mathcal{C}, \{t\}) = 0$ , the respective lower bound (right hand side of Equation (8) or Equation (9)) is zero by convention.
Proof sketch (proof in Appendix I.3). Our proof follows a line similar to that used in [64, 100, 65] for lower bounding the separation rank of self-attention neural networks. The separation rank of any  $f:(\mathbb{R}^{D_x})^{|\mathcal{V}|}\to \mathbb{R}$  can be lower bounded by examining its outputs over a grid of inputs. Specifically, for  $M\in \mathbb{N}$  template vectors  $\mathbf{v}^{(1)},\ldots ,\mathbf{v}^{(M)}\in \mathbb{R}^{D_x}$ , we can create a grid tensor for  $f$  by evaluating it over each point in  $\{(\mathbf{v}^{(d_1)},\dots,\mathbf{v}^{(d_{|\mathcal{V}|})})\}_{d_1,\dots,d_{|\mathcal{V}|}=1}^M$  and storing the outcomes in a tensor with  $|\mathcal{V}|$  axes of dimension  $M$  each. Arranging the grid tensor as a matrix  $\mathbf{B}(f)$  where rows correspond to axes indexed by  $\mathcal{I}$  and columns correspond to the remaining axes, we show that  $\mathrm{rank}(\mathbf{B}(f))\leq \mathrm{sep}(f;\mathcal{I})$ . The proof proceeds by establishing that for almost every assignment of  $\theta$ , there exist template vectors with which  $\log (\mathrm{rank}(\mathbf{B}(f^{(\theta ,\mathcal{G})}))$  and  $\log (\mathrm{rank}(\mathbf{B}(f^{(\theta ,\mathcal{G},t)})))$  are greater than (or equal to) the right hand sides of Equations (8) and (9), respectively.
Directed edges and multiple edge types Appendix D generalizes Theorems 2 and 3 to the case of graphs with directed edges and an arbitrary number of edge types.
**Expressivity of GNNs** The expressivity of GNNs has been predominantly evaluated through ability to distinguish non-isomorphic graphs, as measured by correspondence to Weisfeiler-Leman (WL) graph isomorphism tests (see [75] for a recent survey). [103, 74] instigated this thread of research, establishing that message-passing GNNs are at most as powerful as the WL algorithm, and can match it under certain technical conditions. Subsequently, architectures surpassing WL were proposed, with expressivity measured via higher-order WL variants (see, e.g., [74, 72, 25, 41, 6, 15, 10, 42, 17, 80]). Another line of inquiry regards universality among continuous permutation invariant or equivariant functions [73, 55, 69, 3, 42]. [25] showed that distinguishing non-isomorphic graphs and universality are, in some sense, equivalent. Lastly, there exist analyses of expressivity focused on the frequency response of GNNs [77, 5] and their capacity to compute specific graph functions, e.g. moments, shortest paths, and substructure counting [35, 9, 39, 69, 26, 23, 17].
Although a primary purpose of GNNs is to model interactions between vertices, none of the past works formally characterize their ability to do so, as our theory does. $^{12}$  The current work thus provides a novel perspective on the expressive power of GNNs. Furthermore, a major limitation of existing approaches — in particular, proofs of equivalence to WL tests and universality — is that they often operate in asymptotic regimes of unbounded network width or depth. Consequently, they fall short of addressing which type of functions can be realized by GNNs of practical size. In contrast, we characterize how the modeled interactions depend on both the input graph structure and the neural network architecture (width and depth). As shown in Section 5, this facilitates designing an efficient and effective edge sparsification algorithm.
Measuring modeled interactions via separation rank Separation rank (Section 2.2) has been paramount to the study of interactions modeled by certain convolutional, recurrent, and self-attention neural networks. It enabled theoretically analyzing how different architectural parameters impact expressivity [32, 29, 30, 34, 7, 90, 62, 61, 56, 57, 64, 100, 65] and implicit regularization [83, 84, 85].<sup>13</sup> On the practical side, insights brought forth by separation rank led to tools for improving performance, including: guidelines for architecture design [30, 62, 64, 100], pretraining schemes [65], and regularizers for countering locality in convolutional neural networks [85]. We employ separation rank
In [21], the mutual information between the embedding of a vertex and the embeddings of its neighbors was proposed as a measure of interaction. However, this measure is inherently local and allows reasoning only about the impact of neighboring nodes on each other in a GNN layer. In contrast, separation rank formulates the strength of interaction the whole GNN models across any partition of an input graph's vertices.
<sup>13</sup>We note that, over a two-dimensional grid graph, a message-passing GNN can be viewed as a convolutional neural network with overlapping convolutional windows. Similarly, over a chain graph, it can be viewed as a bidirectional recurrent neural network. Thus, for these special cases, our separation rank bounds (delivered in Section 4) extend those of [30, 62, 56, 61], which consider convolutional neural networks with non-overlapping convolutional windows and unidirectional recurrent neural networks.
for studying the interactions GNNs model between vertices, and similarly provide both theoretical insights and a practical application — edge sparsification algorithm (Section 5).
Edge sparsification Computations over large-scale graphs can be prohibitively expensive in terms of runtime and memory. As a result, various methods were proposed for sparsifying graphs by removing edges while attempting to maintain structural properties, such as distances between vertices [11, 48], graph Laplacian spectrum [93, 86], and vertex degree distribution [98], or outcomes of graph analysis and clustering algorithms [87, 20]. Most relevant to our work are recent edge sparsification methods aiming to preserve the prediction accuracy of GNNs as the number of removed edges increases [67, 24]. These methods require training a GNN over the original (non-sparsified) graph, hence only inference costs are reduced. Guided by our theory, in Section 5 we propose Walk Index Sparsification (WIS) — an edge sparsification algorithm that preserves expressive power in terms of ability to model interactions. WIS improves efficiency for both training and inference. Moreover, comparisons with the spectral algorithm of [93] and a recent method from [24] demonstrate that WIS brings about higher prediction accuracies across edge sparsity levels.
C Tightness of Upper Bounds for Separation Rank
Theorem 2 upper bounds the separation rank with respect to  $\mathcal{I} \subseteq \mathcal{V}$  of a depth  $L$  GNN with product aggregation. According to it, under the setting of graph prediction, the separation rank is largely capped by the  $(L - 1)$ -walk index of  $\mathcal{I}$ , i.e. the number of length  $L - 1$  walks from  $\mathcal{C}_{\mathcal{I}}$  — the set of vertices with an edge crossing the partition  $(\mathcal{I}, \mathcal{I}^c)$ . Similarly, for prediction over  $t \in \mathcal{V}$ , separation rank is largely capped by the  $(L - 1, t)$ -walk index of  $\mathcal{I}$ , which takes into account only length  $L - 1$  walks from  $\mathcal{C}_{\mathcal{I}}$  ending at  $t$ . Theorem 3 provides matching lower bounds, up to logarithmic terms and to the number of walks from  $\mathcal{C}_{\mathcal{I}}$  being replaced with the number of walks from any single admissible subset  $\mathcal{C} \in S(\mathcal{I})$  (Definition 4). Hence, the match between the upper and lower bounds is determined by the portion of  $\mathcal{C}_{\mathcal{I}}$  that can be covered by an admissible subset.
In this appendix, to shed light on the tightness of the upper bounds, we present several concrete examples on which a significant portion of  $\mathcal{C}_{\mathcal{I}}$  can be covered by an admissible subset.
Complete graph Suppose that every two vertices are connected by an edge, i.e.  $\mathcal{E} = \{\{i,j\} : i,j \in \mathcal{V}\}$ . For any non-empty  $\mathcal{I} \subsetneq \mathcal{V}$ , clearly  $\mathcal{C}_{\mathcal{I}} = \mathcal{N}(\mathcal{I}) \cap \mathcal{N}(\mathcal{I}^c) = \mathcal{V}$ . In this case,  $\mathcal{C}_{\mathcal{I}} = \mathcal{V} \in S(\mathcal{I})$ , meaning  $\mathcal{C}_{\mathcal{I}}$  is an admissible subset of itself. To see it is so, notice that for any  $i \in \mathcal{I}, j \in \mathcal{I}^c$ , all vertices are neighbors of both  $\mathcal{I}' := \{i\}$  and  $\mathcal{J}' := \{j\}$ , which trivially have no repeating shared neighbors (Definition 3). Thus, up to a logarithmic factor, the upper and lower bounds from Theorems 2 and 3 coincide.
Chain graph Suppose that  $\mathcal{E} = \{\{i,i + 1\} : i \in [|\mathcal{V}| - 1]\} \cup \{\{i,i\} : i \in \mathcal{V}\}$ . For any non-empty  $\mathcal{I} \subsetneq \mathcal{V}$ , at least half of the vertices in  $\mathcal{C}_{\mathcal{I}}$  can be covered by an admissible subset. That is, there exists  $\mathcal{C} \in S(\mathcal{I})$  satisfying  $|\mathcal{C}| \geq 2^{-1} \cdot |\mathcal{C}_{\mathcal{I}}|$ . For example, such  $\mathcal{C}$  can be constructed algorithmically as follows. Let  $\mathcal{I}'$ ,  $\mathcal{J}' = \emptyset$ . Starting from  $k = 1$ , if  $\{k,k + 1\} \subseteq \mathcal{C}_{\mathcal{I}}$  and one of  $\{k,k + 1\}$  is in  $\mathcal{I}$  while the other is in  $\mathcal{I}^c$ , then assign  $\mathcal{I}' \gets \mathcal{I}' \cup (\{k,k + 1\} \cap \mathcal{I})$ ,  $\mathcal{J}' \gets \mathcal{J}' \cup (\{k,k + 1\} \cap \mathcal{I}^c)$ , and  $k \gets k + 3$ . That is, add each of  $\{k,k + 1\}$  to either  $\mathcal{I}'$  if it is in  $\mathcal{I}$  or  $\mathcal{J}'$  if it is in  $\mathcal{I}^c$ , and skip vertex  $k + 2$ . Otherwise, set  $k \gets k + 1$ . The process terminates once  $k > |\mathcal{V}| - 1$ . By construction,  $\mathcal{I}' \subseteq \mathcal{I}$  and  $\mathcal{J}' \subseteq \mathcal{I}^c$ , implying that  $\mathcal{N}(\mathcal{I}') \cap \mathcal{N}(\mathcal{J}') \subseteq \mathcal{C}_{\mathcal{I}}$ . Due to the chain graph structure,  $\mathcal{I}' \cup \mathcal{J}' \subseteq \mathcal{N}(\mathcal{I}') \cap \mathcal{N}(\mathcal{J}')$  and  $\mathcal{I}'$  and  $\mathcal{J}'$  have no repeating shared neighbors (Definition 3). Furthermore, for every pair of vertices from  $\mathcal{C}_{\mathcal{I}}$  added to  $\mathcal{I}'$  and  $\mathcal{J}'$ , we can miss at most two other vertices from  $\mathcal{C}_{\mathcal{I}}$ . Thus,  $\mathcal{C} := \mathcal{N}(\mathcal{I}') \cap \mathcal{N}(\mathcal{J}')$  is an admissible subset of  $\mathcal{C}_{\mathcal{I}}$  satisfying  $|\mathcal{C}| \geq 2^{-1} \cdot |\mathcal{C}_{\mathcal{I}}|$ .
General graph For an arbitrary graph and non-empty  $\mathcal{I} \subsetneq \mathcal{V}$ , an admissible subset of  $\mathcal{C}_{\mathcal{I}}$  can be obtained by taking any sequence of pairs  $(i_1, j_1), \ldots, (i_M, j_M) \in \mathcal{I} \times \mathcal{I}^c$  with no shared neighbors, in the sense that  $[\mathcal{N}(i_m) \cup \mathcal{N}(j_m)] \cap [\mathcal{N}(i_{m'}) \cup \mathcal{N}(j_{m'})] = \emptyset$  for all  $m \neq m' \in [M]$ . Defining  $\mathcal{I}' := \{i_1, \ldots, i_M\}$  and  $\mathcal{J}' := \{j_1, \ldots, j_M\}$ , by construction they do not have repeating shared neighbors (Definition 3), and so  $\mathcal{N}(\mathcal{I}') \cap \mathcal{N}(\mathcal{J}') \in S(\mathcal{I})$ . In particular, the shared neighbors of each pair are covered by  $\mathcal{N}(\mathcal{I}') \cap \mathcal{N}(\mathcal{J}')$ , i.e.  $\cup_{m=1}^{M} \mathcal{N}(i_m) \cap \mathcal{N}(j_m) \subseteq \mathcal{N}(\mathcal{I}') \cap \mathcal{N}(\mathcal{J}')$ .
D Extension of Analysis to Directed Graphs With Multiple Edge Types
In this appendix, we generalize the separation rank bounds from Theorems 2 and 3 to directed graphs with multiple edge types.
Let  $\mathcal{G} = (\mathcal{V},\mathcal{E},\tau)$  be a directed graph with vertices  $\mathcal{V} = [|\mathcal{V}|]$ , edges  $\mathcal{E} \subseteq \{(i,j):i,j \in \mathcal{V}\}$ , and a map  $\tau: \mathcal{E} \to [Q]$  from edges to one of  $Q \in \mathbb{N}$  edge types. For  $i \in \mathcal{V}$ , let  $\mathcal{N}_{in}(i) := \{j \in \mathcal{V}: (j,i) \in \mathcal{E}\}$  be its incoming neighbors and  $\mathcal{N}_{out}(i) := \{j \in \mathcal{V}: (i,j) \in \mathcal{E}\}$  be its outgoing neighbors. For  $\mathcal{I} \subseteq \mathcal{V}$ , we denote  $\mathcal{N}_{in}(\mathcal{I}) := \cup_{i \in \mathcal{I}} \mathcal{N}_{in}(i)$  and  $\mathcal{N}_{out}(\mathcal{I}) := \cup_{i \in \mathcal{I}} \mathcal{N}_{out}(i)$ . As customary in the context of GNNs, we assume the existence of all self-loops (cf. Section 2.1).
Message-passing GNNs (Section 3) operate identically over directed and undirected graphs, except that in directed graphs the hidden embedding of a vertex is updated only according to its incoming neighbors. For handling multiple edge types, common practice is to use different weight matrices per type in the GNN's update rule (cf. [49, 88]). Hence, we consider the following update rule for directed graphs with multiple edge types, replacing that from Equation (2):
$$
\mathbf {h} ^ {(l, i)} = \text {A G G R E G A T E} \left(\left\{\left\{\mathbf {W} ^ {(l, \tau (j, i))} \mathbf {h} ^ {(l - 1, j)}: j \in \mathcal {N} _ {i n} (i) \right\} \right\}\right), \tag {10}
$$
where  $(\mathbf{W}^{(1,q)}\in \mathbb{R}^{D_h\times D_x})_{q\in [Q]}$  and  $(\mathbf{W}^{(l,q)}\in \mathbb{R}^{D_h\times D_h})_{l\in \{2,\dots ,L\} ,q\in [Q]}$  are learnable weight matrices.
In our analysis for undirected graphs (Appendix A), a central concept is  $\mathcal{C}_{\mathcal{I}}$  — the set of vertices with an edge crossing the partition induced by  $\mathcal{I} \subseteq \mathcal{V}$ . Due to the existence of self-loops it is equal to the shared neighbors of  $\mathcal{I}$  and  $\mathcal{I}^c$ , i.e.  $\mathcal{C}_{\mathcal{I}} = \mathcal{N}(\mathcal{I}) \cap \mathcal{N}(\mathcal{I}^c)$ . We generalize this concept to directed graphs, defining  $\mathcal{C}_{\mathcal{I}}^{\rightarrow}$  to be the set of vertices with an incoming edge from the other side of the partition induced by  $\mathcal{I}$ , i.e.  $\mathcal{C}_{\mathcal{I}}^{\rightarrow} := \{i \in \mathcal{I} : \mathcal{N}_{in}(i) \cap \mathcal{I}^c \neq \emptyset\} \cup \{j \in \mathcal{I}^c : \mathcal{N}_{in}(j) \cap \mathcal{I} \neq \emptyset\}$ . Due to the existence of self-loops it is given by  $\mathcal{C}_{\mathcal{I}}^{\rightarrow} = \mathcal{N}_{out}(\mathcal{I}) \cap \mathcal{N}_{out}(\mathcal{I}^c)$ . Indeed, for undirected graphs  $\mathcal{C}_{\mathcal{I}}^{\rightarrow} = \mathcal{C}_{\mathcal{I}}$ .
With the definition of  $\mathcal{C}_{\mathcal{I}}^{\rightarrow}$  in place, Theorem 4 upper bounds the separation ranks a GNN can achieve over directed graphs with multiple edge types. A technical subtlety is that the bounds depend on walks of lengths  $l = L - 1, L - 2, \ldots, 0$ , while those in Theorem 2 for undirected graphs depend only on walks of length  $L - 1$ . As shown in the proof of Theorem 2, this dependence exists in undirected graphs as well. Though, in undirected graphs with self-loops, the number of length  $l \in \mathbb{N}$  walks from  $\mathcal{C}_{\mathcal{I}}$  decays exponentially as  $l$  decreases. One can therefore replace the sum over walk lengths with walks of length  $L - 1$  (up to a multiplicative constant). By contrast, in directed graphs this is not true in general, e.g., when  $\mathcal{C}_{\mathcal{I}}^{\rightarrow}$  contains only vertices with no outgoing edges (besides self-loops).
Theorem 4. For a directed graph with multiple edge types  $\mathcal{G}$  and  $t\in \mathcal{V}$ , let  $f^{(\theta ,\mathcal{G})}$  and  $f^{(\theta ,\mathcal{G},t)}$  be the functions realized by depth  $L$  graph and vertex prediction GNNs, respectively, with width  $D_{h}$ , learnable weights  $\theta$ , and product aggregation (Equations (3) to (5) and (10)). Then, for any  $\mathcal{I}\subseteq \mathcal{V}$  and assignment of weights  $\theta$  it holds that:
$$
\left(\operatorname {g r a p h} \text {p r e d i c t i o n}\right) \quad \log \left(\operatorname {s e p} \left(f ^ {\left(\theta , \mathcal {G}\right)}; \mathcal {I}\right)\right) \leq \log \left(D _ {h}\right) \cdot \left(\sum_ {l = 1} ^ {L} \rho_ {L - l} \left(\mathcal {C} _ {\mathcal {I}} ^ {\rightarrow}, \mathcal {V}\right) + 1\right), \tag {11}
$$
$$
\text {(v e r t e x p r e d i c t i o n)} \quad \log \left(\operatorname {s e p} \left(f ^ {(\theta , \mathcal {G}, t)}; \mathcal {I}\right)\right) \leq \log \left(D _ {h}\right) \cdot \sum_ {l = 1} ^ {L} \rho_ {L - l} \left(\mathcal {C} _ {\mathcal {I}} ^ {\rightarrow}, \{t \}\right). \tag {12}
$$
Proof sketch (proof in Appendix I.4). The proof follows a line identical to that of Theorem 2, only requiring adjusting definitions from undirected graphs to directed graphs with multiple edge types.  $\square$
Towards lower bounding separation ranks, we generalize the definitions of vertex subsets with no repeating shared neighbors (Definition 3) and admissible subsets of  $\mathcal{C}_{\mathcal{I}}$  (Definition 4) to directed graphs.
Definition 5. We say that  $\mathcal{I},\mathcal{J}\subseteq \mathcal{V}$  have no outgoing repeating shared neighbors if every  $k\in$ $\mathcal{N}_{out}(\mathcal{I})\cap \mathcal{N}_{out}(\mathcal{J})$  has only a single incoming neighbor in each of  $\mathcal{I}$  and  $\mathcal{J}$ , i.e.  $|\mathcal{N}_{in}(k)\cap \mathcal{I}| = |\mathcal{N}_{in}(k)\cap \mathcal{J}| = 1$
Definition 6. For  $\mathcal{I} \subseteq \mathcal{V}$ , we refer to  $\mathcal{C} \subseteq \mathcal{C}_{\mathcal{I}}^{\rightarrow}$  as an admissible subset of  $\mathcal{C}_{\mathcal{I}}^{\rightarrow}$  if there exist  $\mathcal{I}' \subseteq \mathcal{I}, \mathcal{J}' \subseteq \mathcal{I}^c$  with no outgoing repeating shared neighbors such that  $\mathcal{C} = \mathcal{N}_{out}(\mathcal{I}') \cap \mathcal{N}_{out}(\mathcal{J}')$ . We use  $S^{\rightarrow}(\mathcal{I})$  to denote the set comprising all admissible subsets of  $\mathcal{C}_{\mathcal{I}}^{\rightarrow}$ :
$$
\mathcal {S} ^ {\rightarrow} (\mathcal {I}) := \left\{\mathcal {C} \subseteq \mathcal {C} _ {\mathcal {I}} ^ {\rightarrow}: \mathcal {C} \text {i s a n a d m i s s i b l e s u b s e t o f} \mathcal {C} _ {\mathcal {I}} ^ {\rightarrow} \right\}.
$$
Theorem 5 generalizes the lower bounds from Theorem 3 to directed graphs with multiple edge types.
Theorem 5. Consider the setting and notation of Theorem 4. Given  $I \subseteq \mathcal{V}$ , for almost all assignments of weights  $\theta$ , i.e. for all but a set of Lebesgue measure zero, it holds that:
$$
\left(\operatorname {g r a p h} \text {p r e d i c t i o n}\right) \quad \log \left(\operatorname {s e p} \left(f ^ {\left(\theta , \mathcal {G}\right)}; \mathcal {I}\right)\right) \geq \max  _ {\mathcal {C} \in \mathcal {S} ^ {\rightarrow} (\mathcal {I})} \log \left(\alpha_ {\mathcal {C}}\right) \cdot \rho_ {L - 1} (\mathcal {C}, \mathcal {V}), \tag {13}
$$
$$
\left(\text {v e r t e x p r e d i c t i o n}\right) \quad \log \left(\operatorname {s e p} \left(f ^ {\left(\theta , \mathcal {G}, t\right)}; \mathcal {I}\right)\right) \geq \max  _ {\mathcal {C} \in \mathcal {S} ^ {\rightarrow} (\mathcal {I})} \log \left(\alpha_ {\mathcal {C}, t}\right) \cdot \rho_ {L - 1} (\mathcal {C}, \{t \}), \tag {14}
$$
with  $D \coloneqq \min \{D_x, D_h\}$ . If  $\rho_{L-1}(\mathcal{C}, \mathcal{V}) = 0$  or  $\rho_{L-1}(\mathcal{C}, \{t\}) = 0$ , the respective lower bound (right hand side of Equation (13) or Equation (14)) is zero by convention.
Proof sketch (proof in Appendix I.5). The proof follows a line identical to that of Theorem 3, only requiring adjusting definitions from undirected graphs to directed graphs with multiple edge types.  $\square$
E Representing Graph Neural Networks With Product Aggregation as Tensor Networks
In this appendix, we prove that GNNs with product aggregation (Section 3) can be represented through tensor networks — a graphical language for expressing tensor contractions, widely used in quantum mechanics literature for modeling quantum states (cf. [97]). This representation facilitates upper bounding the separation ranks of a GNN with product aggregation (proofs for Theorem 2 and its extension in Appendix D), and is delivered in Appendix E.3. We note that analogous tensor network representations were shown for variants of recurrent and convolutional neural networks [61, 62]. For the convenience of the reader, we lay out basic concepts from the field of tensor analysis in Appendix E.1 and provide a self-contained introduction to tensor networks in Appendix E.2 (see [79] for a more in-depth treatment).
E.1 Primer on Tensor Analysis