SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbf {h} _ {1} ^ {(2, i)} = \prod_ {j \in \mathcal {N} (i) \cap \mathcal {C}} \left(\sum_ {d = 1} ^ {D _ {h}} \mathbf {h} _ {d} ^ {(1, j)}\right) \cdot \prod_ {j \in \mathcal {N} (i) \backslash \mathcal {C}} \left(\sum_ {d = 1} ^ {D _ {h}} \mathbf {h} _ {d} ^ {(1, j)}\right) \\ = \prod_ {j \in \mathcal {N} (i) \cap \mathcal {C}} \left(\sum_ {d = 1} ^ {D _ {h}} \left[ \bar {\mathbf {v}} \odot \bar {\mathbf {v}} ^ {\prime} \right] _ {d}\right) \cdot \prod_ {j \in \mathcal {N} (i) \backslash \mathcal {C}} \left(\sum_ {d = 1} ^ {D _ {h}} \mathbf {h} _ {d} ^ {(1, j)}\right) \\ = \left\langle \mathbf {v} _ {: D}, \mathbf {v} _ {: D} ^ {\prime} \right\rangle^ {\rho_ {1} (\mathcal {C}, \{i \})} \cdot \prod_ {j \in \mathcal {N} (i) \backslash \mathcal {C}} \left(\sum_ {d = 1} ^ {D _ {h}} \mathbf {h} _ {d} ^ {(1, j)}\right). \\ \end{array}
$$
As discussed above, for each  $j \in \mathcal{N}(i) \setminus \mathcal{C}$  the hidden embedding  $\mathbf{h}^{(1,j)}$  does not depend on  $\mathbf{v}$  or it does not depend on  $\mathbf{v}'$ . Furthermore,  $\sum_{d=1}^{D_h} \mathbf{h}_d^{(1,j)} > 0$  for all  $j \in \mathcal{N}(i)$ . Hence, there exist  $\phi^{(2,i)}, \psi^{(2,i)} : \mathbb{R}^{D_x} \to \mathbb{R}_{>0}$  such that:
$$
\mathbf {h} _ {1} ^ {(2, i)} = \phi^ {(2, i)} (\mathbf {v}) \left\langle \mathbf {v}: _ {D}, \mathbf {v} _ {: D} ^ {\prime} \right\rangle^ {\rho_ {1} (\mathcal {C}, \{i \})} \psi^ {(2, i)} (\mathbf {v} ^ {\prime}),
$$
completing the base case.
Now, assuming that the inductive claim holds for  $l - 1 \geq 2$ , we prove that it holds for  $l$ . Let  $i \in \mathcal{V}$ . By the update rule of a GNN with product aggregation  $\mathbf{h}^{(l,i)} = \odot_{j \in \mathcal{N}(i)} (\mathbf{W}^{(l)} \mathbf{h}^{(l-1,j)})$ . Plugging in the value of  $\mathbf{W}^{(l)}$  we get:
$$
\mathbf {h} _ {1} ^ {(l, i)} = \prod_ {j \in \mathcal {N} (i)} \mathbf {h} _ {1} ^ {(l - 1, j)}, \quad \forall d \in \{2, \dots , D _ {h} \}: \mathbf {h} _ {d} ^ {(l, i)} = 0.
$$
By the inductive assumption  $\mathbf{h}_1^{(l - 1,j)} = \phi^{(l - 1,j)}(\mathbf{v})\langle \mathbf{v}_{:D},\mathbf{v}_{:D}^{\prime}\rangle^{\rho_{l - 2}(\mathcal{C},\{j\})}\psi^{(l - 1,j)}(\mathbf{v}^{\prime})$  for all  $j\in \mathcal{N}(i)$ , where  $\phi^{(l - 1,j)},\psi^{(l - 1,j)}:\mathbb{R}^{D_x}\to \mathbb{R}_{>0}$ . Thus:
$$
\begin{array}{l} \mathbf {h} _ {1} ^ {(l, i)} = \prod_ {j \in \mathcal {N} (i)} \mathbf {h} _ {1} ^ {(l - 1, j)} \\ = \prod_ {j \in \mathcal {N} (i)} \phi^ {(l - 1, j)} (\mathbf {v}) \left\langle \mathbf {v}: _ {D}, \mathbf {v} _ {: D} ^ {\prime} \right\rangle^ {\rho_ {l - 2} (\mathcal {C}, \{j \})} \psi^ {(l - 1, j)} (\mathbf {v} ^ {\prime}) \\ = \left(\prod_ {j \in \mathcal {N} (i)} \phi^ {(l - 1, j)} (\mathbf {v})\right) \cdot \left\langle \mathbf {v}; D, \mathbf {v} _ {: D} ^ {\prime} \right\rangle^ {\sum_ {j \in \mathcal {N} (i)} \rho_ {l - 2} (\mathcal {C}, \{j \})} \cdot \left(\prod_ {j \in \mathcal {N} (i)} \psi^ {(l - 1, j)} (\mathbf {v} ^ {\prime})\right). \\ \end{array}
$$
Define  $\phi^{(l,i)}:\mathbf{v}\mapsto \prod_{j\in \mathcal{N}(i)}\phi^{(l - 1,j)}(\mathbf{v})$  and  $\psi^{(l,i)}:\mathbf{v}^{\prime}\mapsto \prod_{j\in \mathcal{N}(i)}\psi^{(l - 1,j)}(\mathbf{v}^{\prime})$ . Since the range of  $\phi^{(l - 1,j)}$  and  $\psi^{(l - 1,j)}$  is  $\mathbb{R}_{>0}$  for all  $j\in \mathcal{N}(i)$ , so is the range of  $\phi^{(l,i)}$  and  $\psi^{(l,i)}$ . The desired result thus readily follows by noticing that  $\sum_{j\in \mathcal{N}(i)}\rho_{l - 2}(\mathcal{C},\{j\}) = \rho_{l - 1}(\mathcal{C},\{i\})$ :
$$
\mathbf {h} _ {1} ^ {(l, i)} = \phi^ {(l, i)} (\mathbf {v}) \left\langle \mathbf {v}: _ {D}, \mathbf {v} _ {: D} ^ {\prime} \right\rangle^ {\rho_ {l - 1} (\mathcal {C}, \{i \})} \psi^ {(l, i)} \left(\mathbf {v} ^ {\prime}\right).
$$
I.4 Proof of Theorem 4
The proof follows a line identical to that of Theorem 2 (Appendix I.2), requiring only slight adjustments. We outline the necessary changes.
Extending the tensor network representations of GNNs with product aggregation to directed graphs and multiple edge types is straightforward. Nodes, legs, and leg weights are as described in Appendix E for undirected graphs with a single edge type, except that:
- Legs connecting  $\delta$ -tensors with weight matrices in the same layer are adapted such that only incoming neighbors are considered. Formally, in Equations (15) to (20),  $\mathcal{N}(i)$  is replaced by  $\mathcal{N}_{in}(i)$  in the leg definitions, for  $i \in \mathcal{V}$ .
- Weight matrices  $(\mathbf{W}^{(l,q)})_{l\in [L],q\in [Q]}$  are assigned to nodes in accordance with edge types. Namely, if at layer  $l\in [L]$  a  $\delta$ -tensor associated with  $i\in \mathcal{V}$  is connected to a weight matrix associated with  $j\in \mathcal{N}_{in}(i)$ , then  $\mathbf{W}^{(l,\tau(j,i))}$  is assigned to the weight matrix node, as opposed to  $\mathbf{W}^{(l)}$  in the single edge type setting. Formally, let  $\mathbf{W}^{(l,j,\gamma)}$  be a node at layer  $l\in [L]$  connected to  $\delta^{(l,i,\gamma')}$ , for  $i\in \mathcal{V}, j\in \mathcal{N}_{in}(i)$ , and some  $\gamma, \gamma' \in \mathbb{N}$ . Then,  $\mathbf{W}^{(l,j,\gamma)}$  stands for a copy of  $\mathbf{W}^{(l,\tau(j,i))}$ .
For  $\mathbf{X} = (\mathbf{x}^{(1)},\dots,\mathbf{x}^{(|\mathcal{V}|)})\in \mathbb{R}^{D_x\times |\mathcal{V}|}$ , let  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}^{(t)}(\mathbf{X})$  be the tensor networks corresponding to  $f^{(\theta ,\mathcal{G})}(\mathbf{X})$  and  $f^{(\theta ,\mathcal{G},t)}(\mathbf{X})$ , respectively, whose construction is outlined above. Then, Lemma 1 (from Appendix I.2.1) and its proof apply as stated. Meaning,  $\mathrm{sep}(f^{(\theta ,\mathcal{G})};\mathcal{I})$  and  $\mathrm{sep}(f^{(\theta ,\mathcal{G},t)};\mathcal{I})$  are upper bounded by the minimal modified multiplicative cut weights in  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}^{(t)}(\mathbf{X})$ , respectively, among cuts separating leaves associated with vertices of the input graph in  $\mathcal{I}$  from leaves associated with vertices of the input graph in  $\mathcal{I}^c$ . Therefore, to establish Equations (11) and (12), it suffices to find cuts in the respective tensor networks with sufficiently low modified multiplicative weights. As is the case for undirected graphs with a single edge type (see Appendices I.2.2 and I.2.3), the cuts separating nodes corresponding to vertices in  $\mathcal{I}$  from all other nodes yield the desired upper bounds.
I.5 Proof of Theorem 5
The proof follows a line identical to that of Theorem 3 (Appendix I.3), requiring only slight adjustments. We outline the necessary changes.
In the context of graph prediction, let  $\mathcal{C}^* \in \operatorname{argmax}_{\mathcal{C} \in \mathcal{S} \to (\mathcal{I})} \log(\alpha_{\mathcal{C}}) \cdot \rho_{L-1}(\mathcal{C}, \mathcal{V})$ . By Lemma 3 (from Appendix I.3), to prove that Equation (13) holds for weights  $\theta$ , it suffices to find template vectors for which  $\log(\operatorname{rank}[[\mathcal{B}(f^{(\theta, \mathcal{G})}); \mathcal{I}]]) \geq \log(\alpha_{\mathcal{C}^*}) \cdot \rho_{L-1}(\mathcal{C}^*, \mathcal{V})$ . Notice that, since the outputs of  $f^{(\theta, \mathcal{G})}$  vary polynomially with  $\theta$ , so do the entries of  $[[[\mathcal{B}(f^{(\theta, \mathcal{G})}); \mathcal{I}]]$  for any choice of template vectors. Thus, according to Lemma 9 (from Appendix I.3.3), by constructing weights  $\theta$  and template vectors satisfying  $\log(\operatorname{rank}[[\mathcal{B}(f^{(\theta, \mathcal{G})}); \mathcal{I}]]) \geq \log(\alpha_{\mathcal{C}^*}) \cdot \rho_{L-1}(\mathcal{C}^*, \mathcal{V})$ , we may conclude that this is the case for almost all assignments of weights, meaning Equation (13) holds for almost all assignments of weights. For undirected graphs with a single edge type, Appendix I.3.1 provides such weights  $\mathbf{W}^{(1)}, \ldots, \mathbf{W}^{(L)}, \mathbf{W}^{(o)}$  and template vectors. The proof in the case of directed graphs with multiple edge types is analogous, requiring only a couple of adaptations: (i) weight matrices of all edge types at layer  $l \in [L]$  are set to the  $\mathbf{W}^{(l)}$  chosen in Appendix I.3.1; and (ii)  $\mathcal{C}_{\mathcal{I}}$  and  $\mathcal{S}(\mathcal{I})$  are replaced with their directed counterparts  $\mathcal{C}_{\mathcal{I}}^{\rightarrow}$  and  $\mathcal{S}^{\rightarrow}(\mathcal{I})$ , respectively.
In the context of vertex prediction, let  $\mathcal{C}_t^* \in \mathrm{argmax}_{\mathcal{C} \in \mathcal{S} \to (\mathcal{I})} \log(\alpha_{\mathcal{C}, t}) \cdot \rho_{L-1}(\mathcal{C}, \{t\})$ . Due to arguments similar to those above, to prove that Equation (14) holds for almost all assignments of weights, we need only find weights  $\theta$  and template vectors satisfying  $\log(\mathrm{rank}[[\mathbf{B}(f^{(\theta, \mathcal{G}, t)}); \mathcal{I}]]) \geq \log(\alpha_{\mathcal{C}_t^*, t}) \cdot \rho_{L-1}(\mathcal{C}_t^*, \{t\})$ . For undirected graphs with a single edge type, Appendix I.3.2 provides such weights and template vectors. The adaptations necessary to extend Appendix I.3.2 to directed graphs with multiple edge types are identical to those specified above for extending Appendix I.3.1 in the context of graph prediction.
Lastly, recalling that a finite union of measure zero sets has measure zero as well establishes that Equations (13) and (14) jointly hold for almost all assignments of weights.
I.6 Proof of Proposition 1
We first prove that the contractions described by  $\mathcal{T}(\mathbf{X})$  produce  $f^{(\theta ,\mathcal{G})}(\mathbf{X})$ . Through an induction over the layer  $l\in [L]$ , for all  $i\in \mathcal{V}$  and  $\gamma \in [\rho_{L - l}(\{i\},\mathcal{V})]$  we show that contracting the sub-tree whose root is  $\delta^{(l,i,\gamma)}$  yields  $\mathbf{h}^{(l,i)}$  — the hidden embedding for  $i$  at layer  $l$  of the GNN inducing  $f^{(\theta ,\mathcal{G})}$ , given vertex features  $\mathbf{x}^{(1)},\ldots ,\mathbf{x}^{(|\mathcal{V}|)}$ .
For  $l = 1$ , fix some  $i \in \mathcal{V}$  and  $\gamma \in [\rho_{L-1}(\{i\}, \mathcal{V})]$ . The sub-tree whose root is  $\delta^{(1,i,\gamma)}$  comprises  $|\mathcal{N}(i)|$  copies of  $\mathbf{W}^{(1)}$ , each associated with some  $j \in \mathcal{N}(i)$  and contracted in its second mode with a copy of  $\mathbf{x}^{(j)}$ . Additionally,  $\delta^{(1,i,\gamma)}$ , which is a copy of  $\delta^{(|\mathcal{N}(i)| + 1)}$ , is contracted with the copies of  $\mathbf{W}^{(1)}$  in their first mode. Overall, the execution of all contractions in the sub-tree can be written as  $\delta^{(|\mathcal{N}(i)| + 1)} *_{j \in [|\mathcal{N}(i)|]} (\mathbf{W}^{(1)} \mathbf{x}^{(\mathcal{N}(i)_j)})$ , where  $\mathcal{N}(i)_j$ , for  $j \in [|\mathcal{N}(i)|]$ , denotes the  $j$ 'th neighbor of  $i$  according to an ascending order (recall vertices are represented by indices from 1 to  $|\mathcal{V}|$ ). The base case concludes by Lemma 11:
$$
\boldsymbol {\delta} ^ {(| \mathcal {N} (i) | + 1)} * _ {j \in [ | \mathcal {N} (i) | ]} \left(\mathbf {W} ^ {(1)} \mathbf {x} ^ {(\mathcal {N} (i) _ {j})}\right) = \odot_ {j \in [ | \mathcal {N} (i) | ]} \left(\mathbf {W} ^ {(1)} \mathbf {x} ^ {(\mathcal {N} (i) _ {j})}\right) = \mathbf {h} ^ {(1, i)}.
$$
Assuming that the inductive claim holds for  $l - 1 \geq 1$ , we prove that it holds for  $l$ . Let  $i \in \mathcal{V}$  and  $\gamma \in [\rho_{L - l}(\{i\}, \mathcal{V})]$ . The children of  $\delta^{(l,i,\gamma)}$  in the tensor network are of the form  $\mathbf{W}^{(l,\mathcal{N}(i)_j, \phi_{l,i,j}(\gamma))}$ , for  $j \in [| \mathcal{N}(i)|]$ , and each  $\mathbf{W}^{(l,\mathcal{N}(i)_j, \phi_{l,i,j}(\gamma))}$  is connected in its other mode to  $\delta^{(l - 1,\mathcal{N}(i)_j, \phi_{l,i,j}(\gamma))}$ . By the inductive assumption for  $l - 1$ , we know that performing all contractions in the sub-tree whose root is  $\delta^{(l - 1,\mathcal{N}(i)_j, \phi_{l,i,j}(\gamma))}$  produces  $\mathbf{h}^{(l - 1,\mathcal{N}(i)_j)}$ , for all  $j \in [| \mathcal{N}(i)|]$ . Since  $\delta^{(l,i,\gamma)}$  is a copy of  $\delta^{(|\mathcal{N}(i)| + 1)}$ , and each  $\mathbf{W}^{(l,\mathcal{N}(i)_j, \phi_{l,i,j}(\gamma))}$  is a copy of  $\mathbf{W}^{(l)}$ , the remaining contractions in the sub-tree of  $\delta^{(l,i,\gamma)}$  thus give:
$$
\boldsymbol {\delta} ^ {(| \mathcal {N} (i) | + 1)} * _ {j \in [ | \mathcal {N} (i) | ]} \left(\mathbf {W} ^ {(l)} \mathbf {h} ^ {(l - 1, \mathcal {N} (i) _ {j})}\right),
$$
which according to Lemma 11 amounts to:
$$
\boldsymbol {\delta} ^ {(| \mathcal {N} (i) | + 1)} * _ {j \in [ | \mathcal {N} (i) | ]} \left(\mathbf {W} ^ {(l)} \mathbf {h} ^ {(l - 1, \mathcal {N} (i) _ {j})}\right) = \odot_ {j \in [ | \mathcal {N} (i) | ]} \left(\mathbf {W} ^ {(l)} \mathbf {h} ^ {(l - 1, \mathcal {N} (i) _ {j})}\right) = \mathbf {h} ^ {(l, i)},
$$
establishing the induction step.
With the inductive claim at hand, we show that contracting  $\mathcal{T}(\mathbf{X})$  produces  $f^{(\theta ,\mathcal{G})}(\mathbf{X})$ . Applying the inductive claim for  $l = L$ , we have that  $\mathbf{h}^{(L,1)},\ldots ,\mathbf{h}^{(L,|\mathcal{V}|)}$  are the vectors produced by executing all contractions in the sub-trees whose roots are  $\delta^{(L,1,1)},\ldots ,\delta^{(L,|\mathcal{V}|,1)}$ , respectively. Performing the remaining contractions, defined by the legs of  $\delta^{(|\mathcal{V}| + 1)}$ , therefore yields  $\mathbf{W}^{(o)}\big(\pmb{\delta}^{(|\mathcal{V}| + 1)}*_{i\in [|\mathcal{V}|]}\mathbf{h}^{(L,i)}\big)$ . By Lemma 11:
$$
\boldsymbol {\delta} ^ {(| \mathcal {V} | + 1)} * _ {i \in [ | \mathcal {V} | ]} \mathbf {h} ^ {(L, i)} = \odot_ {i \in [ | \mathcal {V} | ]} \mathbf {h} ^ {(L, i)}.
$$
Hence,  $\mathbf{W}^{(o)}\big(\delta^{(|\mathcal{V}| + 1)}*_{i\in [|\mathcal{V}|]}\mathbf{h}^{(L,i)}\big) = \mathbf{W}^{(o)}(\odot_{i\in [|\mathcal{V}|]}\mathbf{h}^{(L,i)}) = f^{(\theta ,\mathcal{G})}(\mathbf{X}),$  meaning contracting  $\mathcal{T}(\mathbf{X})$  results in  $f^{(\theta ,\mathcal{G})}(\mathbf{X})$
An analogous proof establishes that the contractions described by  $\mathcal{T}^{(t)}(\mathbf{X})$  yield  $f^{(\theta ,\mathcal{G},t)}(\mathbf{X})$ . Specifically, the inductive claim and its proof are the same, up to  $\gamma$  taking values in  $[\rho_{L - l}(\{i\},\{t\})]$  instead of  $[\rho_{L - l}(\{i\},\mathcal{V})]$ , for  $l\in [L]$ . This implies that  $\mathbf{h}^{(L,t)}$  is the vector produced by contracting the sub-tree whose root is  $\pmb{\delta}^{(L,t,1)}$ . Performing the only remaining contraction, defined by the leg connecting  $\pmb{\delta}^{(L,t,1)}$  with  $\mathbf{W}^{(o)}$ , thus results in  $\mathbf{W}^{(o)}\mathbf{h}^{(L,t)} = f^{(\theta ,\mathcal{G},t)}(\mathbf{X})$ .
I.6.1 Technical Lemma
Lemma 11. Let  $\delta^{(N + 1)}\in \mathbb{R}^{D\times \dots \times D}$  be an order  $N + 1\in \mathbb{N}$  tensor that has ones on its hyperdiagonal and zeros elsewhere, i.e.  $\delta_{d_1,\ldots ,d_{N + 1}}^{(N + 1)} = 1$  if  $d_{1} = \dots = d_{N + 1}$  and  $\delta_{d_1,\ldots ,d_{N + 1}}^{(N + 1)} = 0$  otherwise, for all  $d_1,\ldots ,d_{N + 1}\in [D]$ . Then, for any  $\mathbf{x}^{(1)},\ldots ,\mathbf{x}^{(N)}\in \mathbb{R}^D$  it holds that  $\delta^{(N + 1)}*_{i\in [N]}\mathbf{x}^{(i)} = \odot_{i\in [N]}\mathbf{x}^{(i)}\in \mathbb{R}^D$ .
Proof. By the definition of tensor contraction (Definition 7), for all  $d \in [D]$  we have that:
$$
\left(\pmb {\delta} ^ {(N + 1)} * _ {i \in [ N ]} \mathbf {x} ^ {(i)}\right) _ {d} = \sum_ {d _ {1}, \ldots , d _ {N} = 1} ^ {D} \pmb {\delta} _ {d _ {1}, \ldots , d _ {N}, d} ^ {(N + 1)} \cdot \prod_ {i \in [ N ]} \mathbf {x} _ {d _ {i}} ^ {(i)} = \prod_ {i \in [ N ]} \mathbf {x} _ {d} ^ {(i)} = \left(\odot_ {i \in [ N ]} \mathbf {x} ^ {(i)}\right) _ {d}.
$$