SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Similarly, denote by  $\mathcal{V}_{\mathcal{T}(\mathbf{X})}[\mathcal{I}^c] \subseteq \mathcal{V}_{\mathcal{T}(\mathbf{X})}$  and  $\mathcal{V}_{\mathcal{T}^{(t)}(\mathbf{X})}[\mathcal{I}^c] \subseteq \mathcal{V}_{\mathcal{T}^{(t)}(\mathbf{X})}$  the sets of leaf nodes in  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}^{(t)}(\mathbf{X})$ , respectively, associated with vertices in  $\mathcal{I}^c$ . Then, the following hold:
GRAPH prediction)  $\operatorname{sep}\big(f^{(\theta, \mathcal{G})}; \mathcal{I}\big) \leq \min_{\mathcal{J}_{\mathcal{T}(\mathbf{X})} \subseteq \mathcal{V}_{\mathcal{T}(\mathbf{X})}} \widetilde{w}_{\mathcal{T}(\mathbf{X})}^{\Pi}(\mathcal{J}_{\mathcal{T}(\mathbf{X})}),$  (21)
$$
s. t. \mathcal {V} _ {\mathcal {T} (\mathbf {X})} [ \mathcal {I} ] \subseteq \mathcal {I} _ {\mathcal {T} (\mathbf {X})} a n d \mathcal {V} _ {\mathcal {T} (\mathbf {X})} [ \mathcal {I} ^ {c} ] \subseteq \mathcal {I} _ {\mathcal {T} (\mathbf {X})} ^ {c}
$$
(  $vertex~pred$  iction)  $\begin{array}{r}\sup \bigl (f^{(\theta ,\mathcal{G},t)};\mathcal{I}\bigr)\leq \end{array}$
$$
s. t. \mathcal {V} _ {\mathcal {T} ^ {(t)} (\mathbf {X})} [ \mathcal {I} ] \subseteq \mathcal {I} _ {\mathcal {T} ^ {(t)} (\mathbf {X})} a n d \mathcal {V} _ {\mathcal {T} ^ {(t)} (\mathbf {X})} [ \mathcal {I} ^ {c} ] \subseteq \mathcal {I} _ {\mathcal {T} ^ {(t)} (\mathbf {X})} ^ {c} \tag {22}
$$
where  $\widetilde{w}_{\mathcal{T}(\mathbf{X})}^{\Pi}(\mathcal{I}_{\mathcal{T}(\mathbf{X})})$  is the modified multiplicative cut weight of  $\mathcal{I}_{\mathcal{T}(\mathbf{X})}$  in  $\mathcal{T}(\mathbf{X})$  and  $\widetilde{w}_{\mathcal{T}^{(t)}(\mathbf{X})}^{\Pi}(\mathcal{I}_{\mathcal{T}^{(t)}(\mathbf{X})})$  is the modified multiplicative cut weight of  $\mathcal{I}_{\mathcal{T}^{(t)}(\mathbf{X})}$  in  $\mathcal{T}^{(t)}(\mathbf{X})$  (Definition 8).
Proof. We first prove Equation (21). Examining  $\mathcal{T}(\mathbf{X})$ , notice that: (i) by Proposition 1 its contraction yields  $f^{(\theta, \mathcal{G})}(\mathbf{X})$ ; (ii) it has a tree structure; and (iii) each of its leaves is associated with a vertex feature vector, i.e. one of  $\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(|\mathcal{V}|)}$ , whereas all other aspects of the tensor network do not depend on  $\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(|\mathcal{V}|)}$ . Specifically, for any  $\mathbf{X}$  and  $\mathbf{X}'$  the nodes, legs, and leg weights of  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}(\mathbf{X}')$  are identical, up to the assignment of features in the leaf nodes. Let  $\mathcal{F} \in \mathbb{R}^{D_x \times \dots \times D_x}$  be the order  $\rho_L(\mathcal{V}, \mathcal{V})$  tensor obtained by contracting all interior nodes in  $\mathcal{T}(\mathbf{X})$ . The above implies that we may write  $f^{(\theta, \mathcal{G})}(\mathbf{X})$  as a contraction of  $\mathcal{F}$  with  $\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(|\mathcal{V}|)}$ . Specifically, it holds that:
$$
f ^ {(\theta , \mathcal {G})} (\mathbf {X}) = \mathcal {F} * _ {n \in [ \rho_ {L} (\mathcal {V}, \mathcal {V}) ]} \mathbf {x} ^ {(\mu (n))}, \tag {23}
$$
for any  $\mathbf{X} = (\mathbf{x}^{(1)},\dots ,\mathbf{x}^{(|\mathcal{V}|)})\in \mathbb{R}^{D_x\times |\mathcal{V}|}$ , where  $\mu :[\rho_L(\mathcal{V},\mathcal{V})]\to \mathcal{V}$  maps a mode index of  $\mathcal{F}$  to the appropriate vertex of  $\mathcal{G}$  according to  $\mathcal{T}(\mathbf{X})$ . Let  $\mu^{-1}(\mathcal{I})\coloneqq \{n\in [\rho_L(\mathcal{V},\mathcal{V})]:\mu (n)\in \mathcal{I}\}$  be the mode indices of  $\mathcal{F}$  corresponding to vertices in  $\mathcal{I}$ . Invoking Lemma 2 leads to the following matricized form of Equation (23):
$$
f ^ {(\theta , \mathcal {G})} (\mathbf {X}) = \left(\circ_ {n \in \mu^ {- 1} (\mathcal {I})} \mathbf {x} ^ {(\mu (n))}\right) ^ {\top} \llbracket \mathcal {F}; \mu^ {- 1} (\mathcal {I}) \rrbracket \left(\circ_ {n \in \mu^ {- 1} (\mathcal {I} ^ {c})} \mathbf {x} ^ {(\mu (n))}\right),
$$
where  $\circ$  denotes the Kronecker product.
We claim that  $\operatorname{sep}(f^{(\theta, \mathcal{G})}; \mathcal{I}) \leq \operatorname{rank}[[\mathcal{F}; \mu^{-1}(\mathcal{I})]]$ . To see it is so, denote  $R := \operatorname{rank}[[\mathcal{F}; \mu^{-1}(\mathcal{I})]]$  and let  $\mathbf{u}^{(1)}, \ldots, \mathbf{u}^{(R)} \in \mathbb{R}^{D_x^{\rho_L(\mathcal{I}, \nu)}}$  and  $\bar{\mathbf{u}}^{(1)}, \ldots, \bar{\mathbf{u}}^{(R)} \in \mathbb{R}^{D_x^{\rho_L(\mathcal{I}^c, \nu)}}$  be such that  $[[ \mathcal{F}; \mu^{-1}(\mathcal{I}) ]] = \sum_{r=1}^{R} \mathbf{u}^{(r)} (\bar{\mathbf{u}}^{(r)})^\top$ . Then, defining  $g^{(r)}: (\mathbb{R}^{D_x})^{|\mathcal{I}|} \to \mathbb{R}$  and  $\bar{g}^{(r)}: (\mathbb{R}^{D_x})^{|\mathcal{I}^c|} \to \mathbb{R}$ , for  $r \in [R]$ , as:
$$
g ^ {(r)} (\mathbf {X} _ {\mathcal {I}}) := \left\langle \circ_ {n \in \mu^ {- 1} (\mathcal {I})} \mathbf {x} ^ {(\mu (n))}, \mathbf {u} ^ {(r)} \right\rangle , \quad \bar {g} ^ {(r)} (\mathbf {X} _ {\mathcal {I} ^ {c}}) := \left\langle \circ_ {n \in \mu^ {- 1} (\mathcal {I} ^ {c})} \mathbf {x} ^ {(\mu (n))}, \bar {\mathbf {u}} ^ {(r)} \right\rangle ,
$$
where  $\mathbf{X}_{\mathcal{I}} := (\mathbf{x}^{(i)})_{i \in \mathcal{I}}$  and  $\mathbf{X}_{\mathcal{I}^c} := (\mathbf{x}^{(j)})_{j \in \mathcal{I}^c}$ , we have that:
$$
\begin{array}{l} f ^ {(\theta , \mathcal {G})} (\mathbf {X}) = \left(\circ_ {n \in \mu^ {- 1} (\mathcal {I})} \mathbf {x} ^ {(\mu (n))}\right) ^ {\top} \left(\sum_ {r = 1} ^ {R} \mathbf {u} ^ {(r)} \left(\bar {\mathbf {u}} ^ {(r)}\right) ^ {\top}\right) \left(\circ_ {n \in \mu^ {- 1} (\mathcal {I} ^ {\mathrm {c}})} \mathbf {x} ^ {(\mu (n))}\right) \\ = \sum_ {r = 1} ^ {R} \left\langle \circ_ {n \in \mu^ {- 1} (\mathcal {I})} \mathbf {x} ^ {(\mu (n))}, \mathbf {u} ^ {(r)} \right\rangle \cdot \left\langle \circ_ {n \in \mu^ {- 1} (\mathcal {I} ^ {c})} \mathbf {x} ^ {(\mu (n))}, \bar {\mathbf {u}} ^ {(r)} \right\rangle \\ = \sum_ {r = 1} ^ {R} g ^ {(r)} (\mathbf {X} _ {\mathcal {I}}) \cdot \bar {g} ^ {(r)} (\mathbf {X} _ {\mathcal {I} ^ {c}}). \\ \end{array}
$$
Since  $\operatorname{sep}(f^{(\theta, \mathcal{G})}; \mathcal{I})$  is the minimal number of summands in a representation of this form of  $f^{(\theta, \mathcal{G})}$ , indeed,  $\operatorname{sep}(f^{(\theta, \mathcal{G})}; \mathcal{I}) \leq R = \operatorname{rank}[[\mathcal{F}; \mu^{-1}(\mathcal{I})]]$ .
What remains is to apply Claim 7 from [62], which upper bounds the rank of a tensor's matricization with multiplicative cut weights in a tree tensor network. In particular, consider an order  $N \in \mathbb{N}$
tensor  $\mathcal{A}$  produced by contracting a tree tensor network  $\mathcal{T}$ . Then, for any  $\mathcal{K} \subseteq [N]$  we have that  $\mathrm{rank}[[\mathcal{A}; \mathcal{K}]]$  is at most the minimal modified multiplicative cut weight in  $\mathcal{T}$ , among cuts separating leaves corresponding to modes  $\mathcal{K}$  from leaves corresponding to modes  $\mathcal{K}^c$ . Thus, invoking Claim 7 from [62] establishes Equation (21):
$$
\begin{array}{l} \operatorname {s e p} \big (f ^ {(\theta , \mathcal {G})}; \mathcal {I} \big) \leq \operatorname {r a n k} \llbracket \boldsymbol {\mathcal {F}}; \mu^ {- 1} (\mathcal {I}) \rrbracket \leq \min  _ {\mathcal {J} _ {\mathcal {T} (\mathbf {X})} \subseteq \mathcal {V} _ {\mathcal {T} (\mathbf {X})}} \quad \tilde {w} _ {\mathcal {T} (\mathbf {X})} ^ {\Pi} (\mathcal {J} _ {\mathcal {T} (\mathbf {X})}). \\ s. t. \mathcal {V} _ {\mathcal {T} (\mathbf {X})} [ \mathcal {I} ] \subseteq \mathcal {J} _ {\mathcal {T} (\mathbf {X})} \text {a n d} \mathcal {V} _ {\mathcal {T} (\mathbf {X})} [ \mathcal {I} ^ {c} ] \subseteq \mathcal {J} _ {\mathcal {T} (\mathbf {X})} ^ {c} \\ \end{array}
$$
Equation (22) readily follows by steps analogous to those used above for proving Equation (21).
I.2.2 Cut in Tensor Network for Graph Prediction (Proof of Equation (6))
For  $\mathbf{X} = (\mathbf{x}^{(1)},\dots ,\mathbf{x}^{(|\mathcal{V}|)})\in \mathbb{R}^{D_x\times |\mathcal{V}|}$ , let  $\mathcal{T}(\mathbf{X}) = (\mathcal{V}_{\mathcal{T}(\mathbf{X})},\mathcal{E}_{\mathcal{T}(\mathbf{X})},w_{\mathcal{T}(\mathbf{X})})$  be the tensor network corresponding to  $f^{(\theta ,\mathcal{G})}(\mathbf{X})$  (detailed in Appendix E.3 and formally defined in Equations (15) to (17)). By Lemma 1, to prove that
$$
\sup  \left(f ^ {(\theta , \mathcal {G})}; \mathcal {I}\right) \leq D _ {h} ^ {4 \rho_ {L - 1} \left(\mathcal {C} _ {\mathcal {I}}, \mathcal {V}\right) + 1},
$$
it suffices to find  $\mathcal{I}_{\mathcal{T}(\mathbf{X})}\subseteq \mathcal{V}_{\mathcal{T}(\mathbf{X})}$  satisfying: (i) leaves of  $\mathcal{T}(\mathbf{X})$  associated with vertices in  $\mathcal{I}$  are in  $\mathcal{I}_{\mathcal{T}(\mathbf{X})}$ , whereas leaves associated with vertices in  $\mathcal{I}^c$  are not in  $\mathcal{I}_{\mathcal{T}(\mathbf{X})}$ ; and (ii)  $\widetilde{w}_{\mathcal{T}(\mathbf{X})}^{\Pi}(\mathcal{I}_{\mathcal{T}(\mathbf{X})})\leq D_h^{4\rho_{L - 1}(\mathcal{C}_\mathcal{I},\mathcal{V}) + 1}$ , where  $\widetilde{w}_{\mathcal{T}(\mathbf{X})}^{\Pi}(\mathcal{I}_{\mathcal{T}(\mathbf{X})})$  is the modified multiplicative cut weight of  $\mathcal{I}_{\mathcal{T}(\mathbf{X})}$  (Definition 8). To this end, define  $\mathcal{I}_{\mathcal{T}(\mathbf{X})}$  to hold all nodes in  $\mathcal{V}_{\mathcal{T}(\mathbf{X})}$  corresponding to vertices in  $\mathcal{I}$ . Formally:
$$
\begin{array}{l} \mathcal {J} _ {\mathcal {T} (\mathbf {X})} := \left\{\mathbf {x} ^ {(i, \gamma)}: i \in \mathcal {I}, \gamma \in [ \rho_ {L} (\{i \}, \mathcal {V}) ] \right\} \cup \\ \left\{\mathbf {W} ^ {(l, i, \gamma)}: l \in [ L ], i \in \mathcal {I}, \gamma \in [ \rho_ {L - l + 1} (\{i \}, \mathcal {V}) ] \right\} \cup \\ \left\{\boldsymbol {\delta} ^ {(l, i, \gamma)}: l \in [ L ], i \in \mathcal {I}, \gamma \in [ \rho_ {L - l} (\{i \}, \mathcal {V}) ] \right\}. \\ \end{array}
$$
Clearly,  $\mathcal{J}_{\mathcal{T}(\mathbf{X})}$  upholds  $(i)$ .
As for (ii), there are two types of legs crossing the cut induced by  $\mathcal{J}_{\mathcal{T}(\mathbf{X})}$  in  $\mathcal{T}(\mathbf{X})$ . First, are those connecting a  $\delta$ -tensor with a weight matrix in the same layer, where one is associated with a vertex in  $\mathcal{I}$  and the other with a vertex in  $\mathcal{I}^c$ . That is, legs connecting  $\delta^{(l,i,\gamma)}$  with  $\mathbf{W}^{(l,\mathcal{N}(i)_j,\phi_{l,i,j}(\gamma))}$ , where  $i\in \mathcal{V}$  and  $\mathcal{N}(i)_j\in \mathcal{V}$  are on different sides of the partition  $(\mathcal{I},\mathcal{I}^c)$  in the input graph, for  $j\in [|\mathcal{N}(i)|],l\in [L],\gamma \in [\rho_{L - l}(\{i\},\mathcal{V})]$ . The  $\delta$ -tensors participating in these legs are exactly those associated with some  $i\in \mathcal{C}_{\mathcal{I}}$  (recall  $\mathcal{C}_{\mathcal{I}}$  is the set of vertices with an edge crossing the partition  $(\mathcal{I},\mathcal{I}^c)$ ). So, for every  $l\in [L]$  and  $i\in \mathcal{C}_{\mathcal{I}}$  there are  $\rho_{L - l}(\{i\},\mathcal{V})$  such  $\delta$ -tensors. Second, are legs from  $\delta$ -tensors associated with  $i\in \mathcal{I}$  in the  $L$ th layer to the  $\delta$ -tensor in the output layer of  $\mathcal{T}(\mathbf{X})$ . That is, legs connecting  $\delta^{(L,i,1)}$  with  $\delta^{(|\mathcal{V}| + 1)}$ , for  $i\in \mathcal{I}$ . Legs incident to the same  $\delta$ -tensor only contribute once to  $\widetilde{w}_{\mathcal{T}(\mathbf{X})}^{\Pi}(\mathcal{J}_{\mathcal{T}(\mathbf{X})})$ . Thus, since the weights of all legs connected to  $\delta$ -tensors are equal to  $D_h$ , we have that:
$$
\widetilde {w} _ {\mathcal {T} (\mathbf {X})} ^ {\Pi} (\mathcal {J} _ {\mathcal {T} (\mathbf {X})}) \leq D _ {h} ^ {1 + \sum_ {l = 1} ^ {L} \sum_ {i \in \mathcal {C} _ {\mathcal {I}}} \rho_ {L - l} (\{i \}, \mathcal {V})} = D _ {h} ^ {1 + \sum_ {l = 1} ^ {L} \rho_ {L - l} (\mathcal {C} _ {\mathcal {I}}, \mathcal {V})}.
$$
Lastly, it remains to show that  $\sum_{l=1}^{L} \rho_{L-l}(\mathcal{C}_{\mathcal{I}}, \mathcal{V}) \leq 4 \rho_{L-1}(\mathcal{C}_{\mathcal{I}}, \mathcal{V})$ , since in that case Lemma 1 implies:
$$
\sup  \left(f ^ {(\theta , \mathcal {G})}; \mathcal {I}\right) \leq \tilde {w} _ {\mathcal {T} (\mathbf {X})} ^ {\Pi} \left(\mathcal {J} _ {\mathcal {T} (\mathbf {X})}\right) \leq D _ {h} ^ {4 \rho_ {L - 1} \left(\mathcal {C} _ {\mathcal {I}}, \mathcal {V}\right) + 1},
$$
which yields Equation (6) by taking the log of both sides.
The main idea is that, in an undirected graph with self-loops, the number of length  $l \in \mathbb{N}$  walks from vertices with at least one neighbor decays exponentially when  $l$  decreases. Observe that  $\rho_{l}(\mathcal{C}_{\mathcal{I}}, \mathcal{V}) \leq \rho_{l+1}(\mathcal{C}_{\mathcal{I}}, \mathcal{V})$  for all  $l \in \mathbb{N}$ . Hence:
$$
\sum_ {l = 1} ^ {L} \rho_ {L - l} \left(\mathcal {C} _ {\mathcal {I}}, \mathcal {V}\right) \leq 2 \sum_ {l \in \{1, 3, \dots , L - 1 \}} \rho_ {L - l} \left(\mathcal {C} _ {\mathcal {I}}, \mathcal {V}\right). \tag {24}
$$
Furthermore, any length  $l \in \mathbb{N}_{\geq 0}$  walk  $i_0, i_1, \ldots, i_l \in \mathcal{V}$  from  $\mathcal{C}_{\mathcal{I}}$  induces at least two walks of length  $l + 2$  from  $\mathcal{C}_{\mathcal{I}}$ , distinct from those induced by other length  $l$  walks — one which goes twice through the
self-loop of  $i_0$  and then proceeds according to the length  $l$  walk, i.e.  $i_0, i_0, i_0, i_1, \ldots, i_l$ , and another that goes to a neighboring vertex (exists since  $i_0 \in \mathcal{C}_{\mathcal{I}}$ ), returns to  $i_0$ , and then proceeds according to the length  $l$  walk. This means that  $\rho_{L - l}(\mathcal{C}_{\mathcal{I}}, \mathcal{V}) \leq 2^{-1} \cdot \rho_{L - l + 2}(\mathcal{C}_{\mathcal{I}}, \mathcal{V}) \leq \dots \leq 2^{-\lfloor l / 2 \rfloor} \cdot \rho_{L - 1}(\mathcal{C}_{\mathcal{I}}, \mathcal{V})$  for all  $l \in \{3, 5, \ldots, L - 1\}$ . Going back to Equation (24), this leads to:
$$
\begin{array}{l} \sum_ {l = 1} ^ {L} \rho_ {L - l} (\mathcal {C} _ {\mathcal {I}}, \mathcal {V}) \leq 2 \sum_ {l \in \{1, 3, \dots , L - 1 \}} 2 ^ {\lfloor l / 2 \rfloor} \cdot \rho_ {L - 1} (\mathcal {C} _ {\mathcal {I}}, \mathcal {V}) \\ \leq 2 \sum_ {l = 0} ^ {\infty} 2 ^ {- l} \cdot \rho_ {L - 1} (\mathcal {C} _ {\mathcal {I}}, \mathcal {V}) \\ = 4 \rho_ {L - 1} \left(\mathcal {C} _ {\mathcal {I}}, \mathcal {V}\right), \\ \end{array}
$$
completing the proof of Equation (6).
I.2.3 Cut in Tensor Network for Vertex Prediction (Proof of Equation (7))
This part of the proof follows a line similar to that of Appendix I.2.2, with differences stemming from the distinction between the operation of a GNN over graph and vertex prediction tasks.
For  $\mathbf{X} = (\mathbf{x}^{(1)},\dots ,\mathbf{x}^{(|\mathcal{V}|)})\in \mathbb{R}^{D_x\times |\mathcal{V}|}$ , let  $\mathcal{T}^{(t)}(\mathbf{X}) = (\mathcal{V}_{\mathcal{T}^{(t)}(\mathbf{X})},\mathcal{E}_{\mathcal{T}^{(t)}(\mathbf{X})},w_{\mathcal{T}^{(t)}(\mathbf{X})})$  be the tensor network corresponding to  $f^{(\theta ,\mathcal{G},t)}(\mathbf{X})$  (detailed in Appendix E.3 and formally defined in Equations (18) to (20)). By Lemma 1, to prove that
$$
\operatorname {s e p} \left(f ^ {\left(\theta , \mathcal {G}, t\right)}; \mathcal {I}\right) \leq D _ {h} ^ {4 \rho_ {L - 1} \left(\mathcal {C} _ {\mathcal {I}}, \{t \}\right)},
$$
it suffices to find  $\mathcal{J}_{\mathcal{T}^{(t)}(\mathbf{X})}\subseteq \mathcal{V}_{\mathcal{T}^{(t)}(\mathbf{X})}$  satisfying: (i) leaves of  $\mathcal{T}^{(t)}(\mathbf{X})$  associated with vertices in  $\mathcal{I}$  are in  $\mathcal{J}_{\mathcal{T}^{(t)}(\mathbf{X})}$ , whereas leaves associated with vertices in  $\mathcal{I}^c$  are not in  $\mathcal{J}_{\mathcal{T}^{(t)}(\mathbf{X})}$ ; and (ii)  $\widetilde{w}_{\mathcal{T}^{(t)}(\mathbf{X})}^{\Pi}(\mathcal{J}_{\mathcal{T}^{(t)}(\mathbf{X})})\leq D_h^{4\rho_{L - 1}(\mathcal{C}_{\mathcal{I}},\{t\})}$ , where  $\widetilde{w}_{\mathcal{T}^{(t)}(\mathbf{X})}^{\Pi}(\mathcal{J}_{\mathcal{T}^{(t)}(\mathbf{X})})$  is the modified multiplicative cut weight of  $\mathcal{J}_{\mathcal{T}^{(t)}(\mathbf{X})}$  (Definition 8). To this end, define  $\mathcal{J}_{\mathcal{T}^{(t)}(\mathbf{X})}$  to hold all nodes in  $\mathcal{V}_{\mathcal{T}^{(t)}(\mathbf{X})}$  corresponding to vertices in  $\mathcal{I}$ . Formally:
$$
\begin{array}{l} \mathcal {J} _ {\mathcal {T} ^ {(t)} (\mathbf {X})} := \left\{\mathbf {x} ^ {(i, \gamma)}: i \in \mathcal {I}, \gamma \in [ \rho_ {L} (\{i \}, \{t \}) ] \right\} \cup \\ \left\{\mathbf {W} ^ {(l, i, \gamma)}: l \in [ L ], i \in \mathcal {I}, \gamma \in [ \rho_ {L - l + 1} (\{i \}, \{t \}) ] \right\} \cup \\ \left\{\boldsymbol {\delta} ^ {(l, i, \gamma)}: l \in [ L ], i \in \mathcal {I}, \gamma \in [ \rho_ {L - l} (\{i \}, \{t \}) ] \right\} \cup \\ \mathcal {W} ^ {(o)}, \\ \end{array}
$$
where  $\mathcal{W}^{(o)}\coloneqq \{\mathbf{W}^{(o)}\}$  if  $t\in \mathcal{T}$  and  $\mathcal{W}^{(o)}\coloneqq \emptyset$  otherwise. Clearly,  $\mathcal{J}_{\mathcal{T}^{(t)}(\mathbf{X})}$  upholds  $(i)$ .
As for (ii), the legs crossing the cut induced by  $\mathcal{J}_{\mathcal{T}^{(t)}(\mathbf{X})}$  in  $\mathcal{T}^{(t)}(\mathbf{X})$  are those connecting a  $\delta$ -tensor with a weight matrix in the same layer, where one is associated with a vertex in  $\mathcal{I}$  and the other with a vertex in  $\mathcal{I}^c$ . That is, legs connecting  $\delta^{(l,i,\gamma)}$  with  $\mathbf{W}^{(l,\mathcal{N}(i)_j,\phi_{l,i,j}^{(t)}(\gamma))}$ , where  $i\in \mathcal{V}$  and  $\mathcal{N}(i)_j\in \mathcal{V}$  are on different sides of the partition  $(\mathcal{I},\mathcal{I}^c)$  in the input graph, for  $j\in [|\mathcal{N}(i)|],l\in [L],\gamma \in [\rho_{L - l}(\{i\},\{t\})]$ . The  $\delta$ -tensors participating in these legs are exactly those associated with some  $i\in \mathcal{C}_{\mathcal{I}}$  (recall  $\mathcal{C}_{\mathcal{I}}$  is the set of vertices with an edge crossing the partition  $(\mathcal{I},\mathcal{I}^c)$ ). Hence, for every  $l\in [L]$  and  $i\in \mathcal{C}_{\mathcal{I}}$  there are  $\rho_{L - l}(\{i\},\{t\})$  such  $\delta$ -tensors. Legs connected to the same  $\delta$ -tensor only contribute once to  $\tilde{w}_{\mathcal{T}^{(t)}(\mathbf{X})}^{\Pi}(\mathcal{J}_{\mathcal{T}^{(t)}(\mathbf{X})})$ . Thus, since the weights of all legs connected to  $\delta$ -tensors are equal to  $D_h$ , we have that:
$$
\widetilde {w} _ {\mathcal {T} ^ {(t)} (\mathbf {X})} ^ {\Pi} \big (\mathcal {I} _ {\mathcal {T} ^ {(t)} (\mathbf {X})} \big) = D _ {h} ^ {\sum_ {l = 1} ^ {L} \sum_ {i \in \mathcal {C} _ {\mathcal {I}}} \rho_ {L - l} \big (\{i \}, \{t \} \big)} = D _ {h} ^ {\sum_ {l = 1} ^ {L} \rho_ {L - l} \big (\mathcal {C} _ {\mathcal {I}}, \{t \} \big)}.
$$
Lastly, it remains to show that  $\sum_{l=1}^{L} \rho_{L-l}(\mathcal{C}_{\mathcal{I}}, \{t\}) \leq 4\rho_{L-1}(\mathcal{C}_{\mathcal{I}}, \{t\})$ , as in that case Lemma 1 implies:
$$
\sup  \left(f ^ {(\theta , \mathcal {G}, t)}; \mathcal {I}\right) \leq \tilde {w} _ {\mathcal {T} ^ {(t)} (\mathbf {X})} ^ {\Pi} \left(\mathcal {J} _ {\mathcal {T} ^ {(t)} (\mathbf {X})}\right) \leq D _ {h} ^ {4 \rho_ {L - 1} \left(\mathcal {C} _ {\mathcal {I}}, \{t \}\right)},
$$
which leads to Equation (7) by taking the log of both sides.
The main idea is that, in an undirected graph with self-loops, the number of length  $l \in \mathbb{N}$  walks ending at  $t$  that originate from vertices with at least one neighbor decays exponentially when  $l$  decreases. First, clearly  $\rho_{l}(\mathcal{C}_{\mathcal{I}},\{t\}) \leq \rho_{l + 1}(\mathcal{C}_{\mathcal{I}},\{t\})$  for all  $l \in \mathbb{N}$ . Therefore:
$$
\sum_ {l = 1} ^ {L} \rho_ {L - l} \left(\mathcal {C} _ {\mathcal {I}}, \{t \}\right) \leq 2 \sum_ {l \in \{1, 3, \dots , L - 1 \}} \rho_ {L - l} \left(\mathcal {C} _ {\mathcal {I}}, \{t \}\right). \tag {25}
$$
Furthermore, any length  $l \in \mathbb{N}_{\geq 0}$  walk  $i_0, i_1, \ldots, i_{l-1}, t \in \mathcal{V}$  from  $\mathcal{C}_{\mathcal{I}}$  to  $t$  induces at least two walks of length  $l+2$  from  $\mathcal{C}_{\mathcal{I}}$  to  $t$ , distinct from those induced by other length  $l$  walks — one which goes twice through the self-loop of  $i_0$  and then proceeds according to the length  $l$  walk, i.e.  $i_0, i_0, i_0, i_1, \ldots, i_{l-1}, t$ , and another that goes to a neighboring vertex (exists since  $i_0 \in \mathcal{C}_{\mathcal{I}}$ ), returns to  $i_0$ , and then proceeds according to the length  $l$  walk. This means that  $\rho_{L-l}(\mathcal{C}_{\mathcal{I}}, \{t\}) \leq 2^{-1} \cdot \rho_{L-l+2}(\mathcal{C}_{\mathcal{I}}, \{t\}) \leq \cdots \leq 2^{-\lfloor l/2 \rfloor} \cdot \rho_{L-1}(\mathcal{C}_{\mathcal{I}}, \{t\})$  for all  $l \in \{3,5,\ldots,L-1\}$ . Going back to Equation (25), we have that:
$$
\begin{array}{l} \sum_ {l = 1} ^ {L} \rho_ {L - l} (\mathcal {C} _ {\mathcal {I}}, \{t \}) \leq 2 \sum_ {l \in \{1, 3, \dots , L - 1 \}} 2 ^ {\lfloor l / 2 \rfloor} \cdot \rho_ {L - 1} (\mathcal {C} _ {\mathcal {I}}, \{t \}) \\ \leq 2 \sum_ {l = 0} ^ {\infty} 2 ^ {- l} \cdot \rho_ {L - 1} (\mathcal {C} _ {\mathcal {I}}, \{t \}) \\ = 4 \rho_ {L - 1} (\mathcal {C} _ {\mathcal {I}}, \{t \}), \\ \end{array}
$$
concluding the proof of Equation (7).
I.2.4 Technical Lemma
Lemma 2. For any order  $N \in \mathbb{N}$  tensor  $\mathcal{A} \in \mathbb{R}^{D \times \dots \times D}$ , vectors  $\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(N)} \in \mathbb{R}^D$ , and subset of mode indices  $\mathcal{I} \subseteq [N]$ , it holds that  $\mathcal{A} *_{i \in [N]} \mathbf{x}^{(i)} = (\circ_{i \in I} \mathbf{x}^{(i)})^\top[[\mathcal{A}; \mathcal{I}]](\circ_{j \in I^c} \mathbf{x}^{(j)}) \in \mathbb{R}$ .
Proof. The identity follows directly from the definitions of tensor contraction, matricization, and Kronecker product (Appendix I.1):
$$
\mathcal {A} * _ {i \in [ N ]} \mathbf {x} ^ {(i)} = \sum_ {d _ {1}, \dots , d _ {N} = 1} ^ {D} \mathcal {A} _ {d _ {1}, \dots , d _ {N}} \cdot \prod_ {i \in [ N ]} \mathbf {x} _ {d _ {i}} ^ {(i)} = \left(\circ_ {i \in I} \mathbf {x} ^ {(i)}\right) ^ {\top} [   [ \mathcal {A}; \mathcal {I} ]   ] \left(\circ_ {j \in \mathcal {I} ^ {c}} \mathbf {x} ^ {(j)}\right).
$$
I.3 Proof of Theorem 3
We assume familiarity with the basic concepts from tensor analysis introduced in Appendix E.1.
We begin by establishing a general technique for lower bounding the separation rank of a function through grid tensors, also used in [64, 100, 65, 85]. For any  $f: (\mathbb{R}^{D_x})^N \to \mathbb{R}$  and  $M \in \mathbb{N}$  template vectors  $\mathbf{v}^{(1)}, \ldots, \mathbf{v}^{(M)} \in \mathbb{R}^{D_x}$ , we can create a grid tensor of  $f$ , which is a form of function discretization, by evaluating it over each point in  $\{(\mathbf{v}^{(d_1)}, \ldots, \mathbf{v}^{(d_N)})\}_{d_1, \ldots, d_N = 1}^M$  and storing the outcomes in an order  $N$  tensor with modes of dimension  $M$ . That is, the grid tensor of  $f$  for templates  $\mathbf{v}^{(1)}, \ldots, \mathbf{v}^{(M)}$ , denoted  $\mathcal{B}(f) \in \mathbb{R}^{M \times \cdots \times M}$ , is defined by  $\mathcal{B}(f)_{d_1, \ldots, d_N} = f(\mathbf{v}^{(d_1)}, \ldots, \mathbf{v}^{(d_N)})$  for all  $d_1, \ldots, d_N \in [M]$ .<sup>16</sup> Lemma 3 shows that  $\mathrm{sep}(f; \mathcal{I})$  is lower bounded by the rank of  $\mathcal{B}(f)$ 's matricization with respect to  $\mathcal{I}$ .
Lemma 3. For  $f: (\mathbb{R}^{D_x})^N \to \mathbb{R}$  and  $M \in \mathbb{N}$  template vectors  $\mathbf{v}^{(1)}, \ldots, \mathbf{v}^{(M)} \in \mathbb{R}^{D_x}$ , let  $\mathcal{B}(f) \in \mathbb{R}^{M \times \dots \times M}$  be the corresponding order  $N$  grid tensor of  $f$ . Then, for any  $\mathcal{I} \subseteq [N]$ :
$$
\operatorname {r a n k} \llbracket \boldsymbol {\mathcal {B}} (f); \mathcal {I} \rrbracket \leq \operatorname {s e p} (f; \mathcal {I}).
$$
Proof. If  $\operatorname{sep}(f; \mathcal{I})$  is  $\infty$  or zero, i.e.  $f$  cannot be represented as a finite sum of separable functions (with respect to  $\mathcal{I}$ ) or is identically zero, then the claim is trivial. Otherwise, denote  $R := \operatorname{sep}(f; \mathcal{I})$ , and let  $g^{(1)}, \ldots, g^{(R)}: (\mathbb{R}^{D_x})^{|\mathcal{I}|} \to \mathbb{R}$  and  $\bar{g}^{(1)}, \ldots, \bar{g}^{(R)}: (\mathbb{R}^{D_x})^{|\mathcal{I}^c|} \to \mathbb{R}$  such that: