SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For our purposes, a tensor is simply a multi-dimensional array. The order of a tensor is its number of axes, which are typically called modes (e.g. a vector is an order one tensor and a matrix is an order two tensor). The dimension of a mode refers to its length, i.e. the number of values it can be indexed with. For an order  $N \in \mathbb{N}$  tensor  $\mathcal{A} \in \mathbb{R}^{D_1 \times \dots \times D_N}$  with modes of dimensions  $D_1, \ldots, D_N \in \mathbb{N}$ , we will denote by  $\mathcal{A}_{d_1, \ldots, d_N}$  its  $(d_1, \ldots, d_N)$ 'th entry, where  $(d_1, \ldots, d_N) \in [D_1] \times \dots \times [D_N]$ .
It is possible to rearrange tensors into matrices — a process known as matricization. The matricization of  $\mathcal{A}$  with respect to  $\mathcal{I} \subseteq [N]$ , denoted  $\llbracket \mathcal{A}; \mathcal{I} \rrbracket \in \mathbb{R}^{\prod_{i \in \mathcal{I}} D_i \times \prod_{j \in \mathcal{I}^c} D_j}$  is its arrangement as a matrix where rows correspond to modes indexed by  $\mathcal{I}$  and columns correspond to the remaining modes. Specifically, denoting the elements in  $\mathcal{I}$  by  $i_1 < \dots < i_{|\mathcal{I}|}$  and those in  $\mathcal{I}^c$  by  $j_1 < \dots < j_{|\mathcal{I}^c|}$ , the matricization  $\llbracket \mathcal{A}; \mathcal{I} \rrbracket$  holds the entries of  $\mathcal{A}$  such that  $\mathcal{A}_{d_1, \ldots, d_N}$  is placed in row index  $1 + \sum_{l=1}^{|\mathcal{I}|} (d_{i_l} - 1) \prod_{l'=l+1}^{|\mathcal{I}|} D_{i_{l'}}$  and column index  $1 + \sum_{l=1}^{|\mathcal{I}^c|} (d_{j_l} - 1) \prod_{l'=l+1}^{|\mathcal{I}^c|} D_{j_{l'}}$ .
Tensors with modes of the same dimension can be combined via contraction — a generalization of matrix multiplication. It will suffice to consider contractions where one of the modes being contracted is the last mode of its tensor.
Definition 7. Let  $\mathcal{A} \in \mathbb{R}^{D_1 \times \dots \times D_N}$ ,  $\mathcal{B} \in \mathbb{R}^{D_1' \times \dots \times D_{N'}'}$  for orders  $N, N' \in \mathbb{N}$  and mode dimensions  $D_1, \ldots, D_N, D_1', \ldots, D_{N'}' \in \mathbb{N}$  satisfying  $D_n = D_{N'}'$  for some  $n \in [N]$ . The mode- $n$  contraction of  $\mathcal{A}$  with  $\mathcal{B}$ , denoted  $\mathcal{A}_{*n}$ ,  $\mathcal{B} \in \mathbb{R}^{D_1 \times \dots \times D_{n-1} \times D_1' \times \dots \times D_{N'-1}' \times D_{n+1} \times \dots \times D_N}$ , is given element-wise by:
$$
\left(\boldsymbol {\mathcal {A}} * _ {n} \boldsymbol {\mathcal {B}}\right) _ {d _ {1}, \dots , d _ {n - 1}, d _ {1} ^ {\prime}, \dots , d _ {N ^ {\prime} - 1} ^ {\prime}, d _ {n + 1}, \dots , d _ {N}} = \sum_ {d _ {n} = 1} ^ {D _ {n}} \boldsymbol {\mathcal {A}} _ {d _ {1}, \dots , d _ {N}} \cdot \boldsymbol {\mathcal {B}} _ {d _ {1} ^ {\prime}, \dots , d _ {N ^ {\prime} - 1} ^ {\prime}, d _ {n}},
$$
Order  $N$  Tensor
Vector-Matrix Multiplication
Figure 4: Tensor network diagrams of (from left to right): a vector  $\mathbf{v} \in \mathbb{R}^D$ , matrix  $\mathbf{M} \in \mathbb{R}^{D_1 \times D_2}$ , order  $N \in \mathbb{N}$  tensor  $\pmb{\tau} \in \mathbb{R}^{D_1 \times \dots \times D_N}$ , and vector-matrix multiplication  $\mathbf{M}\mathbf{v} \in \mathbb{R}^{D_1}$ . The mode index associated with a leg's end point is specified in gray, and the weight of the leg, specified in black, determines the mode dimension.
for all  $d_1 \in [D_1], \ldots, d_{n-1} \in [D_{n-1}], d_1' \in [D_1'], \ldots, d_{N'-1}' \in [D_{N'-1}], d_{n+1} \in [D_{n+1}], \ldots, d_N \in [D_N]$ .
For example, the mode-2 contraction of  $\mathbf{A} \in \mathbb{R}^{D_1 \times D_2}$  with  $\mathbf{B} \in \mathbb{R}^{D_1' \times D_2}$  boils down to multiplying  $\mathbf{A}$  with  $\mathbf{B}^\top$  from the right, i.e.  $\mathbf{A} *_2\mathbf{B} = \mathbf{A}\mathbf{B}^\top$ . It is oftentimes convenient to jointly contract multiple tensors. Given an order  $N$  tensor  $\mathcal{A}$  and  $M \in \mathbb{N}_{\leq N}$  tensors  $\mathcal{B}^{(1)}, \ldots, \mathcal{B}^{(M)}$ , we use  $\mathcal{A} *_{i \in [M]} \mathcal{B}^{(i)}$  to denote the contraction of  $\mathcal{A}$  with  $\mathcal{B}^{(1)}, \ldots, \mathcal{B}^{(M)}$  in modes  $1, \ldots, M$ , respectively (assuming mode dimensions are such that the contractions are well-defined).
E.2 Tensor Networks
A tensor network is an undirected weighted graph  $\mathcal{T} = (\mathcal{V}_{\mathcal{T}},\mathcal{E}_{\mathcal{T}},w_{\mathcal{T}})$  that describes a sequence of tensor contractions (Definition 7), with vertices  $\nu_{\mathcal{T}}$  , edges  $\varepsilon_{\mathcal{T}}$  , and a function mapping edges to natural weights  $w_{\mathcal{T}}:\mathcal{E}_{\mathcal{T}}\to \mathbb{N}$  . We will only consider tensor networks that are connected. To avoid confusion with vertices and edges of a GNN's input graph, and in accordance with tensor network terminology, we refer by nodes and legs to the vertices and edges of a tensor network, respectively.
Every node in a tensor network is associated with a tensor, whose order is equal to the number of legs emanating from the node. Each end point of a leg is associated with a mode index, and the leg's weight determines the dimension of the corresponding tensor mode. That is, an end point of  $e \in \mathcal{E}_{\mathcal{T}}$  is a pair  $(\mathcal{A}, n) \in \mathcal{V}_{\mathcal{T}} \times \mathbb{N}$ , with  $n$  ranging from one to the order of  $\mathcal{A}$ , and  $w_{\mathcal{T}}(e)$  is the dimension of  $\mathcal{A}$  in mode  $n$ . A leg can either connect two nodes or be connected to a node on one end and be loose on the other end. If two nodes are connected by a leg, their associated tensors are contracted together in the modes specified by the leg. Legs with a loose end are called open legs. The number of open legs is exactly the order of the tensor produced by executing all contractions in the tensor network, i.e. by contracting the tensor network. Figure 4 presents exemplar tensor network diagrams of a vector, matrix, order  $N \in \mathbb{N}$  tensor, and vector-matrix multiplication.
E.3 Tensor Networks Corresponding to Graph Neural Networks With Product Aggregation
Fix some undirected graph  $\mathcal{G}$  and learnable weights  $\theta = (\mathbf{W}^{(1)},\dots,\mathbf{W}^{(L)},\mathbf{W}^{(o)})$ . Let  $f^{(\theta ,\mathcal{G},t)}$  and  $f^{(\theta ,\mathcal{G},t)}$ , for  $t\in \mathcal{V}$ , be the functions realized by depth  $L$  graph and vertex prediction GNNs, respectively, with width  $D_{h}$  and product aggregation (Equations (2) to (5)). For  $\mathbf{X} = (\mathbf{x}^{(1)},\ldots ,\mathbf{x}^{(|\mathcal{V}|)})\in \mathbb{R}^{D_x\times |\mathcal{V}|}$ , we construct tensor networks  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}^{(t)}(\mathbf{X})$  whose contraction yields  $f^{(\theta ,\mathcal{G})}(\mathbf{X})$  and  $f^{(\theta ,\mathcal{G},t)}(\mathbf{X})$ , respectively. Both  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}^{(t)}(\mathbf{X})$  adhere to a tree structure, where each leaf node is associated with a vertex feature vector, i.e. one of  $\mathbf{x}^{(1)},\ldots ,\mathbf{x}^{(|\mathcal{V}|)}$ , and each interior node is associated with a weight matrix from  $\mathbf{W}^{(1)},\ldots ,\mathbf{W}^{(L)},\mathbf{W}^{(o)}$  or a  $\delta$ -tensor with modes of dimension  $D_{h}$ , holding ones on its hyper-diagonal and zeros elsewhere. We denote an order  $N\in \mathbb{N}$  tensor of the latter type by  $\pmb{\delta}^{(N)}\in \mathbb{R}^{D_h\times \dots \times D_h}$ , i.e.  $\pmb{\delta}_{d_1,\dots,d_N}^{(N)} = 1$  if  $d_{1} = \dots = d_{N}$  and  $\pmb{\delta}_{d_1,\dots,d_N}^{(N)} = 0$  otherwise for all  $d_{1},\ldots ,d_{N}\in [D_{h}]$ .
(b)
Figure 5: Tensor network diagrams of the operations performed by GNNs with product aggregation (Section 3). (a) Hidden embedding update (cf. Equations (2) and (5)):  $\mathbf{h}^{(l,i)} = (\mathbf{W}^{(l)}\mathbf{h}^{(l - 1,j_1)})\odot \dots \odot (\mathbf{W}^{(l)}\mathbf{h}^{(l - 1,j_M)})$  where  $\mathcal{N}(i) = \{j_1,\ldots ,j_M\}$ , for  $l\in [L],i\in \mathcal{V}$ . (b) Output layer for graph prediction (cf. Equations (3) and (5)):  $f^{(\theta ,\mathcal{G})}(\mathbf{X}) = \mathbf{W}^{(o)}(\mathbf{h}^{(L,1)}\odot \dots \odot \mathbf{h}^{(L,|\mathcal{V}|)})$ . (c) Output layer for vertex prediction over  $t\in \mathcal{V}$  (cf. Equation (4)):  $f^{(\theta ,\mathcal{G},t)}(\mathbf{X}) = \mathbf{W}^{(o)}\mathbf{h}^{(L,t)}$ . We draw nodes associated with  $\delta$ -tensors as rectangles to signify their special (hyper-diagonal) structure, and omit leg weights to avoid clutter (legs connected to  $\mathbf{h}^{(0,i)} = \mathbf{x}^{(i)}$  for  $i\in \mathcal{V}$ , have weight  $D_{x}$  while all other legs have weight  $D_{h}$ ).
Intuitively,  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}^{(t)}(\mathbf{X})$  embody unrolled computation trees, describing the operations performed by the respective GNNs through tensor contractions. Let  $\mathbf{h}^{(l,i)} = \odot_{j\in \mathcal{N}(i)}(\mathbf{W}^{(l)}\mathbf{h}^{(l - 1,j)})$  be the hidden embedding of  $i\in \mathcal{V}$  at layer  $l\in [L]$  (recall  $\mathbf{h}^{(0,j)} = \mathbf{x}^{(j)}$  for  $j\in \mathcal{V}$ ), and denote  $\mathcal{N}(i) = \{j_1,\ldots ,j_M\}$ . We can describe  $\mathbf{h}^{(l,i)}$  as the outcome of contracting each  $\mathbf{h}^{(l - 1,j_1)},\dots,\mathbf{h}^{(l - 1,j_M)}$  with  $\mathbf{W}^{(l)}$ , i.e. computing  $\mathbf{W}^{(l)}\mathbf{h}^{(l - 1,j_1)},\dots,\mathbf{W}^{(l)}\mathbf{h}^{(l - 1,j_M)}$ , followed by contracting the resulting vectors with  $\delta^{(|\mathcal{N}(i)| + 1)}$ , which induces product aggregation (see Figure 5(a)). Furthermore, in graph prediction, the output layer producing  $f^{(\theta ,\mathcal{G})}(\mathbf{X}) = \mathbf{W}^{(o)}(\odot_{i\in \mathcal{V}}\mathbf{h}^{(L,i)})$  amounts to contracting  $\mathbf{h}^{(L,1)},\ldots ,\mathbf{h}^{(L,|\mathcal{V}|)}$  with  $\delta^{(|\mathcal{V}| + 1)}$ , and subsequently contracting the resulting vector with  $\mathbf{W}^{(o)}$  (see Figure 5(b)); while for vertex prediction,  $f^{(\theta ,\mathcal{G},t)}(\mathbf{X}) = \mathbf{W}^{(o)}\mathbf{h}^{(L,t)}$  is a contraction of  $\mathbf{h}^{(L,t)}$  with  $\mathbf{W}^{(o)}$  (see Figure 5(c)).
Overall, every layer in a GNN with product aggregation admits a tensor network formulation given the outputs of the previous layer. Thus, we can construct a tree tensor network for the whole GNN by starting from the output layer — Figure 5(b) for graph prediction or Figure 5(c) for vertex prediction — and recursively expanding nodes associated with  $\mathbf{h}^{(l,i)}$  according to Figure 5(a), for  $l = L,\dots ,1$  and  $i\in \mathcal{V}$ . A technical subtlety is that each  $\mathbf{h}^{(l,i)}$  can appear multiple times during this procedure. In the language of tensor networks this translates to duplication of nodes. Namely, there are multiple copies of the sub-tree representing  $\mathbf{h}^{(l,i)}$  in the tensor network — one copy per appearance when unraveling the recursion. Figure 6 displays examples for tensor network diagrams of  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}^{(t)}(\mathbf{X})$ .
We note that, due to the node duplication mentioned above, the explicit definitions of  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}^{(t)}(\mathbf{X})$  entail cumbersome notation. Nevertheless, we provide them in Appendix E.3.1 for the interested reader.
E.3.1 Explicit Tensor Network Definitions
The tree tensor network representing  $f^{(\theta, \mathcal{G})}(\mathbf{X})$  consists of an initial input level — the leaves of the tree — comprising  $\rho_L(\{i\}, \mathcal{V})$  copies of  $\mathbf{x}^{(i)}$  for each  $i \in \mathcal{V}$ . We will use  $\mathbf{x}^{(i, \gamma)}$  to denote the copies of  $\mathbf{x}^{(i)}$  for  $i \in \mathcal{V}$  and  $\gamma \in [\rho_L(\{i\}, \mathcal{V})]$ . In accordance with the GNN inducing  $f^{(\theta, \mathcal{G})}$ , following the initial input level are  $L + 1$  layers. Each layer  $l \in [L]$  includes two levels: one comprising  $\rho_{L - l + 1}(\mathcal{V}, \mathcal{V})$  nodes standing for copies of  $\mathbf{W}^{(l)}$ , and another containing  $\delta$ -tensors —  $\rho_{L - l}(\{i\}, \mathcal{V})$  copies of  $\delta^{(|\mathcal{N}(i)| + 1)}$  per  $i \in \mathcal{V}$ . We associate each node in these layers with its layer index and a vertex of the input graph  $i \in \mathcal{V}$ . Specifically, we will use  $\mathbf{W}^{(l,i,\gamma)}$  to denote copies of  $\mathbf{W}^{(l)}$  and  $\delta^{(l,i,\gamma)}$  to denote copies of  $\delta^{(|\mathcal{N}(i)| + 1)}$ , for  $l \in [L], i \in \mathcal{V}$ , and  $\gamma \in \mathbb{N}$ . In terms of connectivity, every leaf  $\mathbf{x}^{(i,\gamma)}$  has a leg to  $\mathbf{W}^{(1,i,\gamma)}$ . The rest of the connections between nodes are such that each sub-tree whose root is  $\delta^{(l,i,\gamma)}$  represents  $\mathbf{h}^{(l,i)}$ , i.e. contracting the sub-tree results in the hidden
$$
\mathcal {G} = \begin{array}{c c c} 1 & - - - 2 & 3 \end{array} \quad \mathcal {V} = \{1, 2, 3 \} \quad \mathcal {E} = \left\{\{1, 2 \}, \{1, 1 \}, \{2, 2 \}, \{3, 3 \} \right\}
$$
$\pmb{T}(\mathbf{X})$
$\mathcal{T}^{(1)}(\mathbf{X})$
Figure 6: Tensor network diagrams of  $\mathcal{T}(\mathbf{X})$  (left) and  $\mathcal{T}^{(t)}(\mathbf{X})$  (right) representing  $f^{(\theta, \mathcal{G})}(\mathbf{X})$  and  $f^{(\theta, \mathcal{G}, t)}(\mathbf{X})$ , respectively, for  $t = 1 \in \mathcal{V}$ , vertex features  $\mathbf{X} = (\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(|\mathcal{V}|)})$ , and depth  $L = 2$  GNNs with product aggregation (Section 3). The underlying input graph  $\mathcal{G}$ , over which the GNNs operate, is depicted at the top. We draw nodes associated with  $\delta$ -tensors as rectangles to signify their special (hyper-diagonal) structure, and omit leg weights to avoid clutter (legs connected to  $\mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \mathbf{x}^{(3)}$  have weight  $D_x$  while all other legs have weight  $D_h$ ). See Appendix E.3 for further details on the construction of  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}^{(t)}(\mathbf{X})$ , and Appendix E.3.1 for explicit formulations.
$$
\mathcal {G} = \begin{array}{c c c c} 1 & - & 2 \\ & 3 \end{array} \quad \mathcal {V} = \{1, 2, 3 \} \quad \mathcal {E} = \left\{\{1, 2 \}, \{1, 1 \}, \{2, 2 \}, \{3, 3 \} \right\}
$$
$\pmb{\mathcal{T}}(\mathbf{X})$
Figure 7: Tensor network diagrams (with explicit node duplication notation) of  $\mathcal{T}(\mathbf{X})$  (left) and  $\mathcal{T}^{(t)}(\mathbf{X})$  (right) representing  $f^{(\theta, \mathcal{G})}(\mathbf{X})$  and  $f^{(\theta, \mathcal{G}, t)}(\mathbf{X})$ , respectively, for  $t = 1 \in \mathcal{V}$ , vertex features  $\mathbf{X} = (\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(|\mathcal{V}|)})$  and depth  $L = 2$  GNNs with product aggregation (Section 3). This figure is identical to Figure 6, except that it uses the explicit notation for node duplication detailed in Appendix E.3.1. Specifically, each feature vector, weight matrix, and  $\delta$ -tensor is attached with an index specifying which copy it is (rightmost index in the superscript). Additionally, weight matrices and  $\delta$ -tensors are associated with a layer index and vertex in  $\mathcal{V}$  (except for the output layer  $\delta$ -tensor in  $\mathcal{T}(\mathbf{X})$  and  $\mathbf{W}^{(o)})$ ). See Equations (15) to (20) for the explicit definitions of these tensor networks.
$\pmb{\mathcal{T}}^{(1)}(\mathbf{X})$
embedding for  $i\in \mathcal{V}$  at layer  $l\in [L]$  of the GNN inducing  $f^{(\theta ,\mathcal{G})}$ . Last, is an output layer consisting of two connected nodes: a  $\delta^{(|\mathcal{V}| + 1)}$  node, which has a leg to every  $\delta$ -tensor from layer  $L$ , and a  $\mathbf{W}^{(o)}$  node. See Figure 7 (left) for an example of a tensor network diagram representing  $f^{(\theta ,\mathcal{G})}(\mathbf{X})$  with this notation.
The tensor network construction for  $f^{(\theta, \mathcal{G}, t)}(\mathbf{X})$  is analogous to that for  $f^{(\theta, \mathcal{G})}(\mathbf{X})$ , comprising an initial input level followed by  $L + 1$  layers. Its input level and first  $L$  layers are structured the same, up to differences in the number of copies for each node. Specifically, the number of copies of  $\mathbf{x}^{(i)}$  is  $\rho_L(\{i\}, \{t\})$  instead of  $\rho_L(\{i\}, \mathcal{V})$ , the number of copies of  $\mathbf{W}^{(l)}$  is  $\rho_{L - l + 1}(\mathcal{V}, \{t\})$  instead of  $\rho_{L - l + 1}(\mathcal{V}, \mathcal{V})$ , and the number of copies of  $\delta^{(|\mathcal{N}(i)| + 1)}$  is  $\rho_{L - l}(\{i\}, \{t\})$  instead of  $\rho_{L - l}(\{i\}, \mathcal{V})$  for  $i \in \mathcal{V}$  and  $l \in [L]$ . The output layer consists only of a  $\mathbf{W}^{(o)}$  node, which is connected to the  $\delta$ -tensor in layer  $L$  corresponding to vertex  $t$ . See Figure 7 (right) for an example of a tensor network diagram representing  $f^{(\theta, \mathcal{G}, t)}(\mathbf{X})$  with this notation.
Formally, the tensor network producing  $f^{(\theta ,\mathcal{G})}(\mathbf{X})$ , denoted  $\mathcal{T}(\mathbf{X}) = (\mathcal{V}_{\mathcal{T}(\mathbf{X})},\mathcal{E}_{\mathcal{T}(\mathbf{X})},w_{\mathcal{T}(\mathbf{X})})$ , is defined by:
$$
\begin{array}{l} \mathcal {V} _ {\mathcal {T} (\mathbf {X})} := \left\{\mathbf {x} ^ {(i, \gamma)}: i \in \mathcal {V}, \gamma \in [ \rho_ {L} (\{i \}, \mathcal {V}) ] \right\} \cup \\ \left\{\mathbf {W} ^ {(l, i, \gamma)}: l \in [ L ], i \in \mathcal {V}, \gamma \in \left[ \rho_ {L - l + 1} (\{i \}, \mathcal {V}) \right] \right\} \cup \tag {15} \\ \left\{\boldsymbol {\delta} ^ {(l, i, \gamma)}: l \in [ L ], i \in \mathcal {V}, \gamma \in [ \rho_ {L - l} (\{i \}, \mathcal {V}) ] \right\} \cup \\ \left\{\delta^ {(| \mathcal {V} | + 1)}, \mathbf {W} ^ {(o)} \right\}, \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {E} _ {\mathcal {T} (\mathbf {X})} := \left\{\left\{(\mathbf {x} ^ {(i, \gamma)}, 1), (\mathbf {W} ^ {(1, i, \gamma)}, 2) \right\}: i \in \mathcal {V}, \gamma \in [ \rho_ {L} (\{i \}, \mathcal {V}) ] \right\} \cup \\ \left\{\left\{(\boldsymbol {\delta} ^ {(l, i, \gamma)}, j), (\mathbf {W} ^ {(l, \mathcal {N} (i) _ {j}, \phi_ {l, i, j} (\gamma))}, 1) \right\}: l \in [ L ], i \in \mathcal {V}, j \in [ | \mathcal {N} (i) | ], \gamma \in [ \rho_ {L - l} (\{i \}, \mathcal {V}) ] \right\} \cup \\ \left\{\left\{(\boldsymbol {\delta} ^ {(l, i, \gamma)}, | \mathcal {N} (i) | + 1), (\mathbf {W} ^ {(l + 1, i, \gamma)}, 2) \right\}: l \in [ L - 1 ], i \in \mathcal {V}, \gamma \in [ \rho_ {L - l} (\{i \}, \mathcal {V}) ] \right\} \cup \\ \left. \left\{\left\{\left(\boldsymbol {\delta} ^ {(| \mathcal {V} | + 1)}, i\right), \left(\boldsymbol {\delta} ^ {(L, i, 1)}, | \mathcal {N} (i) | + 1\right) \right\}: i \in \mathcal {V} \right\} \cup \left\{\left\{\left(\boldsymbol {\delta} ^ {(| \mathcal {V} | + 1)}, | \mathcal {V} | + 1\right), \left(\mathbf {W} ^ {(o)}, 2\right) \right\} \right\}, \right. \tag {16} \\ \end{array}
$$
$$
w _ {\mathcal {T} (\mathbf {X})} (e) := \left\{ \begin{array}{l l} D _ {x} & , \text {i f} (\mathbf {x} ^ {(i, \gamma)}, 1) \text {i s a n e d p o i n t o f} e \in \mathcal {E} _ {\mathcal {T}} \text {f o r s o m e} i \in \mathcal {V}, \gamma \in [ \rho_ {L} (\{i \}, \mathcal {V}) ] \\ D _ {h} & , \text {o t h e r w i s e} \end{array} \right., \tag {17}
$$
where  $\phi_{l,i,j}(\gamma) := \gamma + \sum_{k < i \text{ s.t. } k \in \mathcal{N}(j)} \rho_{L - l}(\{k\}, \mathcal{V})$ , for  $l \in [L], i \in \mathcal{V}$ , and  $\gamma \in [\rho_{L - l}(\{i\}, \mathcal{V})]$ , is used to map a  $\delta$ -tensor copy corresponding to  $i$  in layer  $l$  to a  $\mathbf{W}^{(l)}$  copy, and  $\mathcal{N}(i)_j$ , for  $i \in \mathcal{V}$  and  $j \in [| \mathcal{N}(i)|]$ , denotes the  $j$ 'th neighbor of  $i$  according to an ascending order (recall vertices are represented by indices from 1 to  $|\mathcal{V}|$ ).
Similarly, the tensor network  $\mathcal{T}^{(t)}(\mathbf{X}) = (\mathcal{V}_{\mathcal{T}^{(t)}(\mathbf{X})},\mathcal{E}_{\mathcal{T}^{(t)}(\mathbf{X})},w_{\mathcal{T}^{(t)}(\mathbf{X})})$ , producing  $f^{(\theta ,\mathcal{G},t)}(\mathbf{X})$ , is defined by:
$$
\begin{array}{l} \mathcal {V} _ {\mathcal {T} ^ {(t)} (\mathbf {X})} := \left\{\mathbf {x} ^ {(i, \gamma)}: i \in \mathcal {V}, \gamma \in [ \rho_ {L} (\{i \}, \{t \}) ] \right\} \cup \\ \left. \left\{\mathbf {W} ^ {(l, i, \gamma)}: l \in [ L ], i \in \mathcal {V}, \gamma \in \left[ \rho_ {L - l + 1} (\{i \}, \{t \}) \right] \right\} \cup \right. \tag {18} \\ \left\{\boldsymbol {\delta} ^ {(l, i, \gamma)}: l \in [ L ], i \in \mathcal {V}, \gamma \in [ \rho_ {L - l} (\{i \}, \{t \}) ] \right\} \cup \\ \left\{\mathbf {W} ^ {(o)} \right\}, \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {E} _ {\mathcal {T} ^ {(t)} (\mathbf {X})} := \left\{\left(\left(\mathbf {x} ^ {(i, \gamma)}, 1\right), \left(\mathbf {W} ^ {(1, i, \gamma)}, 2\right) \right\}: i \in \mathcal {V}, \gamma \in \left[ \rho_ {L} (\{i \}, \{t \}) \right] \right\} \cup \\ \left\{\left\{(\boldsymbol {\delta} ^ {(l, i, \gamma)}, j), (\mathbf {W} ^ {(l, \mathcal {N} (i) _ {j}, \phi_ {l, i, j} ^ {(t)} (\gamma))}, 1) \right\}: l \in [ L ], i \in \mathcal {V}, j \in [ | \mathcal {N} (i) | ], \gamma \in [ \rho_ {L - l} (\{i \}, \{t \}) ] \right\} \cup \\ \left\{\left(\boldsymbol {\delta} ^ {(l, i, \gamma)}, | \mathcal {N} (i) | + 1), (\mathbf {W} ^ {(l + 1, i, \gamma)}, 2) \right\}: l \in [ L - 1 ], i \in \mathcal {V}, \gamma \in [ \rho_ {L - l} (\{i \}, \{t \}) ] \right\} \cup \\ \left\{\left\{(\boldsymbol {\delta} ^ {(L, t, 1)}, | \mathcal {N} (t) | + 1), (\mathbf {W} ^ {(o)}, 2) \right\} \right\}, \tag {19} \\ \end{array}
$$
$$
w _ {\mathcal {T} ^ {(t)} (\mathbf {X})} (e) := \left\{ \begin{array}{l l} D _ {x} & , \text {i f} (\mathbf {x} ^ {(i, \gamma)}, 1) \text {i s a n e d p o i n t o f} e \in \mathcal {E} _ {\mathcal {T}} \text {f o r s o m e} i \in \mathcal {V}, \gamma \in [ \rho_ {L} (\{i \}, \{t \}) ] \\ D _ {h} & , \text {o t h e r w i s e} \end{array} \right. \tag {20}
$$
where  $\phi_{l,i,j}^{(t)}(\gamma) \coloneqq \gamma + \sum_{k < i \text{ s.t. } k \in \mathcal{N}(j)} \rho_{L-l}(\{k\}, \{t\})$ , for  $l \in [L]$ ,  $i \in \mathcal{V}$ , and  $\gamma \in [\rho_{L-l}(\{i\}, \{t\})]$ , is used to map a  $\delta$ -tensor copy corresponding to  $i$  in layer  $l$  to a  $\mathbf{W}^{(l)}$  copy.
Proposition 1 verifies that contracting  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}^{(t)}(\mathbf{X})$  yields  $f^{(\theta, \mathcal{G})}(\mathbf{X})$  and  $f^{(\theta, \mathcal{G}, t)}(\mathbf{X})$ , respectively.
Proposition 1. For an undirected graph  $\mathcal{G}$  and  $t\in \mathcal{V}$ , let  $f^{(\theta ,\mathcal{G})}$  and  $f^{(\theta ,\mathcal{G},t)}$  be the functions realized by depth  $L$  graph and vertex prediction GNNs, respectively, with width  $D_{h}$ , learnable weights  $\theta$ , and product aggregation (Equations (2) to (5)). For vertex features  $\mathbf{X} = (\mathbf{x}^{(1)},\ldots ,\mathbf{x}^{(|\mathcal{V}|)})\in \mathbb{R}^{D_x\times |\mathcal{V}|}$ , let the tensor networks  $\mathcal{T}(\mathbf{X}) = (\mathcal{V}_{\mathcal{T}(\mathbf{X})},\mathcal{E}_{\mathcal{T}(\mathbf{X})},w_{\mathcal{T}(\mathbf{X})})$  and  $\mathcal{T}^{(t)}(\mathbf{X}) = (\mathcal{V}_{\mathcal{T}^{(t)}(\mathbf{X})},\mathcal{E}_{\mathcal{T}^{(t)}(\mathbf{X})},w_{\mathcal{T}^{(t)}(\mathbf{X})})$  be as defined in Equations (15) to (20), respectively. Then, performing the contractions described by  $\mathcal{T}(\mathbf{X})$  produces  $f^{(\theta ,\mathcal{G})}(\mathbf{X})$ , and performing the contractions described by  $\mathcal{T}^{(t)}(\mathbf{X})$  produces  $f^{(\theta ,\mathcal{G},t)}(\mathbf{X})$ .