SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof sketch (proof in Appendix 1.6). For both  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}^{(t)}(\mathbf{X})$ , a straightforward induction over the layer  $l \in [L]$  establishes that contracting the sub-tree whose root is  $\delta^{(l,i,\gamma)}$  results in  $\mathbf{h}^{(l,i)}$  for all  $i \in \mathcal{V}$  and  $\gamma$ , where  $\mathbf{h}^{(l,i)}$  is the hidden embedding for  $i$  at layer  $l$  of the GNNs inducing  $f^{(\theta, \mathcal{G})}$  and  $f^{(\theta, \mathcal{G}, t)}$ , given vertex features  $\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(|\mathcal{V}|)}$ . The proof concludes by showing that the contractions in the output layer of  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}^{(t)}(\mathbf{X})$  reproduce the operations defining  $f^{(\theta, \mathcal{G})}(\mathbf{X})$  and  $f^{(\theta, \mathcal{G}, t)}(\mathbf{X})$  in Equations (3) and (4), respectively.
F General Walk Index Sparsification
Our edge sparsification algorithm — Walk Index Sparsification (WIS) — was obtained as an instance of the General Walk Index Sparsification (GWIS) scheme described in Section 5. Algorithm 3 formally outlines this general scheme.
G Efficient Implementation of 1-Walk Index Sparsification
Algorithm 2 (Section 5) provides an efficient implementation for 1-WIS, i.e. Algorithm 1 with  $L = 2$ . In this appendix, we formalize the equivalence between the two algorithms, meaning, we establish that Algorithm 2 indeed implements 1-WIS.
Examining some iteration  $n \in [N]$  of 1-WIS, let  $\mathbf{s} \in \mathbb{R}^{|\mathcal{V}|}$  be the tuple defined by  $\mathbf{s}_t = \mathrm{WI}_{1,t}(\{t\}) = \rho_1(\mathcal{C}_{\{t\}}, \{t\})$  for  $t \in \mathcal{V}$ . Recall that  $\mathcal{C}_{\{t\}}$  is the set of vertices with an edge crossing the partition induced by  $\{t\}$ . Thus, if  $t$  is not isolated, then  $\mathcal{C}_{\{t\}} = \mathcal{N}(t)$  and  $\mathbf{s}_t = \mathrm{WI}_{1,t}(\{t\}) = |\mathcal{N}(t)|$ . Otherwise, if  $t$  is isolated, then  $\mathcal{C}_{\{t\}} = \emptyset$  and  $\mathbf{s}_t = \mathrm{WI}_{1,t}(\{t\}) = 0$ . 1-WIS computes for each  $e \in \mathcal{E}$  (excluding self-loops) a tuple  $\mathbf{s}^{(e)} \in \mathbb{R}^{|\mathcal{V}|}$  holding in its  $t$ 'th entry what the value of  $\mathrm{WI}_{1,t}(\{t\})$  would be if  $e$  is to be removed, for all  $t \in \mathcal{V}$ . Notice that  $\mathbf{s}^{(e)}$  and  $\mathbf{s}$  agree on all entries except for  $i, j \in e$ , since removing  $e$  from the graph only affects the degrees of  $i$  and  $j$ . Specifically, for  $i \in e$ ,
Algorithm 3  $(L - 1)$ -General Walk Index Sparsification (GWIS)
Input:
-  $\mathcal{G}$  — graph
-  $L \in \mathbb{N}$  — GNN depth
-  $N \in \mathbb{N}$  — number of edges to remove
-  $\mathcal{I}_1, \ldots, \mathcal{I}_M \subseteq \mathcal{V}$  — vertex subsets specifying walk indices to maintain for graph prediction
-  $\mathcal{J}_1, \ldots, \mathcal{J}_{M'} \subseteq \mathcal{V}$  and  $t_1, \ldots, t_{M'} \in \mathcal{V}$  — vertex subsets specifying walk indices to maintain with respect to target vertices, for vertex prediction
-  $\operatorname{ARGMAX}$  — operator over tuples  $(\mathbf{s}^{(e)} \in \mathbb{R}^{M + M'})_{e \in \mathcal{E}}$  that returns the edge whose tuple is maximal according to some order
Result: Sparsified graph obtained by removing  $N$  edges from  $\mathcal{G}$
for  $n = 1,\dots ,N$  do # for every edge, compute walk indices of partitions after the edge's removal   
for  $e\in \mathcal{E}$  (excluding self-loops) do initialize  $\mathbf{s}^{(e)} = (0,\ldots ,0)\in \mathbb{R}^{M + M^{\prime}}$  remove  $e$  from  $\mathcal{G}$  (temporarily) for every  $m\in [M]$  , set  $\mathbf{s}_m^{(e)} = \mathrm{WI}_{L - 1}(\mathcal{I}_m)\# = \rho_{L - 1}(\mathcal{C}_{\mathcal{I}_m},\mathcal{V})$  for every  $m\in [M^{\prime}]$  , set  $\mathbf{s}_{M + m}^{(e)} = \mathrm{WI}_{L - 1,t_m}(\mathcal{J}_m)\# = \rho_{L - 1}(\mathcal{C}_{\mathcal{J}_m},\{t_m\})$  add  $e$  back to  $\mathcal{G}$    
end for # prune edge whose removal harms walk indices the least according to the ARGMAX operator let  $e^{\prime}\in \mathrm{ARGMAX}_{e\in \mathcal{E}}\mathbf{s}^{(e)}$  remove  $e^{\prime}$  from  $\mathcal{G}$  (permanently)   
end for
either  $\mathbf{s}_i^{(e)} = \mathbf{s}_i - 1 = |\mathcal{N}(i)| - 1$  if the removal of  $e$  did not isolate  $i$ , or  $\mathbf{s}_i^{(e)} = \mathbf{s}_i - 2 = 0$  if it did (due to self-loops, if a vertex has a single edge to another then  $|\mathcal{N}(i)| = 2$ , so removing that edge changes  $\mathrm{WI}_{1,i}(\{i\})$  from two to zero). As a result, for any  $e = \{i,j\}$ ,  $e' = \{i',j'\} \in \mathcal{E}$ , after sorting the entries of  $\mathbf{s}^{(e)}$  and  $\mathbf{s}^{(e')}$  in ascending order we have that  $\mathbf{s}^{(e')}$  is greater in lexicographic order than  $\mathbf{s}^{(e)}$  if and only if the pair  $(\min \{| \mathcal{N}(i')|, |\mathcal{N}(j')| \}, \max \{| \mathcal{N}(i')|, |\mathcal{N}(j')| \})$  is greater in lexicographic order than  $(\min \{| \mathcal{N}(i)|, |\mathcal{N}(j)| \}, \max \{| \mathcal{N}(i)|, |\mathcal{N}(j)| \})$ . Therefore, at every iteration  $n \in [N]$  Algorithm 2 and 1-WIS (Algorithm 1 with  $L = 2$ ) remove the same edge.
H Further Experiments and Implementation Details
H.1 Further Experiments
Figure 8 supplements Figure 3 from Section 5.2 by including experiments with additional: (i) GNN architectures — GIN and ResGCN; and (ii) datasets — Chameleon, Squirrel, and Amazon Computers. Overall, our evaluation includes six standard vertex prediction datasets in which we observed the graph structure to be crucial for accurate prediction, as measured by the difference between the test accuracy of a GCN trained and evaluated over the original graph and its test accuracy when trained and evaluated over the graph after all of the graph's edges were removed. We also considered, but excluded, the following datasets in which the accuracy difference was insignificant (less than five percentage points): CiteSeer [89], PubMed [76], Coauthor CS and Physics [92], and Amazon Photo [92].
H.2 Further Implementation Details
We provide implementation details omitted from our experimental reports (Section 4.2, Section 5, and Appendix H.1). Source code for reproducing our results and figures, based on the PyTorch [81] and PyTorch Geometric [38] frameworks, can be found at https://github.com/noamrazin/
Figure 8: Comparison of GNN accuracies following sparsification of input edges — WIS, the edge sparsification algorithm brought forth by our theory (Algorithm 1), markedly outperforms alternative methods. This figure supplements Figure 3 from Section 5.2 by including experiments with: (i) a depth  $L = 3$  GIN over the Cora, DBLP, and OGBN-ArXiv datasets; (ii) a depth  $L = 10$  ResGCN over the Cora, DBLP, and OGBN-ArXiv datasets; and (iii) a depth  $L = 3$  GCN over the Chameleon, Squirrel, and Amazon Computers datasets. Markers and error bars report means and standard deviations, respectively, taken over ten runs per configuration for GCN and GIN, and over five runs per configuration for ResGCN (we use fewer runs due to the larger size of ResGCN). For further details see caption of Figure 3 as well as Appendix H.2.
gnn_interactions. All experiments were run either on a single Nvidia RTX 2080 Ti GPU or a single Nvidia RTX A6000 GPU.
H.2.1 Empirical Demonstration of Theoretical Analysis (Table 1)
Models All models used, i.e. GCN, GAT, and GIN, had three layers of width 16 with ReLU nonlinearity. To ease optimization, we added layer normalization [4] after each one. Mean aggregation and a linear output layer were applied over the last hidden embeddings for prediction. As in the synthetic experiments of [1], each GAT layer consisted of four attention heads. Each GIN layer had its  $\epsilon$  parameter fixed to zero and contained a two-layer feed-forward network, whose layers comprised a linear layer, batch normalization [54], and ReLU non-linearity.
Data The datasets consisted of 10000 train and 2000 test graphs. For every graph, we drew uniformly at random a label from  $\{0,1\}$  and an image from Fashion-MNIST. Then, depending on the chosen label, another image was sampled either from the same class (for label 1) or from all other classes (for label 0). We extracted patches of pixels from each image by flattening it into a vector and splitting the vector to 16 equally sized segments.
Optimization The binary cross-entropy loss was minimized via the Adam optimizer [58] with default  $\beta_{1},\beta_{2}$  coefficients and full-batches (i.e. every batch contained the whole training set). Optimiza
tion proceeded until the train accuracy did not improve by at least 0.01 over 1000 consecutive epochs or 10000 epochs elapsed. The learning rates used for GCN, GAT, and GIN were  $5 \cdot 10^{-3}$ ,  $5 \cdot 10^{-3}$ , and  $10^{-2}$ , respectively.
Hyperparameter tuning For each model separately, to tune the learning rate we carried out five runs (differing in random seed) with every value in the range  $\{10^{-1}, 5 \cdot 10^{-2}, 10^{-2}, 5 \cdot 10^{-3}, 10^{-3}\}$  over the dataset whose essential partition has low walk index. Since our interest resides in expressivity, which manifests in ability to fit the training set, for every model we chose the learning rate that led to the highest mean train accuracy.
H.2.2 Edge Sparsification (Figures 3 and 8)
Adaptations to UGS [24] [24] proposed UGS as a framework for jointly pruning input graph edges and weights of a GNN. At a high-level, UGS trains two differentiable masks,  $m_{g}$  and  $m_{\theta}$ , that are multiplied with the graph adjacency matrix and the GNN's weights, respectively. Then, after a certain number of optimization steps, a predefined percentage  $p_{g}$  of graph edges are removed according to the magnitudes of entries in  $m_{g}$ , and similarly,  $p_{\theta}$  percent of the GNN's weights are fixed to zero according to the magnitudes of entries in  $m_{\theta}$ . This procedure continues in iterations, where each time the remaining GNN weights are rewinded to their initial values, until the desired sparsity levels are attained — see Algorithms 1 and 2 in [24]. To facilitate a fair comparison of our  $(L - 1)$ -WIS edge sparsification algorithm with UGS, we make the following adaptations to UGS.
- We adapt UGS to only remove edges, which is equivalent to fixing the entries in the weight mask  $m_{\theta}$  to one and setting  $p_{\theta} = 0$  in Algorithm 1 of [24].
- For comparing performance across a wider range of sparsity levels, the number of edges removed at each iteration is changed from  $5\%$  of the current number of edges to  $5\%$  of the original number of edges.
- Since our evaluation focuses on undirected graphs, we enforce the adjacency matrix mask  $m_{g}$  to be symmetric.
Spectral sparsification [93] For Cora and DBLP, we used a Python implementation of the spectral sparsification algorithm from [93], based on the PyGSP library implementation.14 To enable more efficient experimentation over the larger scale OGBN-ArXiv dataset, we used a Julia implementation based on that from the Laplacians library.15
Models The GCN and GIN models had three layers of width 64 with ReLU non-linearity. As in the experiments of Section 4.2, we added layer normalization [4] after each one. Every GIN layer had a trainable  $\epsilon$  parameter and contained a two-layer feed-forward network, whose layers comprised a linear layer, batch normalization [54], and ReLU non-linearity. For ResGCN, we used the implementation from [24] with ten layers of width 64. In all models, a linear output layer was applied over the last hidden embeddings for prediction.
Data All datasets in our evaluation are multi-class vertex prediction tasks, each consisting of a single graph. In Cora, DBLP, and OGBN-ArXiv, vertices represent scientific publications and edges stand for citation links. In Chameleon and Squirrel, vertices represent web pages on Wikipedia and edges stand for mutual links between pages. In Amazon Computers, vertices represent products and edges indicate that two products are frequently bought together. For simplicity, we treat all graphs as undirected. Table 2 reports the number of vertices and undirected edges in each dataset. For all datasets, except OGBN-ArXiv, we randomly split the labels of vertices into train, validation, and test sets comprising  $80\%$ ,  $10\%$ , and  $10\%$  of all labels, respectively. For OGBN-ArXiv, we used the default split from [52].
**Optimization** The cross-entropy loss was minimized via the Adam optimizer [58] with default  $\beta_{1}, \beta_{2}$  coefficients and full-batches (i.e. every batch contained the whole training set). Optimization proceeded until the validation accuracy did not improve by at least 0.01 over 1000 consecutive epochs or 10000 epochs elapsed. The test accuracies reported in Figure 3 are those achieved during the epochs with highest validation accuracies. Table 3 specifies additional optimization hyperparameters.
14See https://github.com/epfl-lts2/pygsp/.
15See https://github.com/danspielman/Laplacians.jl.
Table 2: Graph size of each dataset used for comparing edge sparsification algorithms in Figures 3 and 8.
# of Vertices	# of Undirected Edges
Cora	2,708	5,278
DBLP	17,716	52,867
OGBN-ArXiv	169,343	1,157,799
Chameleon	2,277	31,396
Squirrel	5,201	198,423
Amazon Computers	13,381	245,861
Table 3: Optimization hyperparameters used in the experiments of Figures 3 and 8 per model and dataset.
Learning Rate	Weight Decay	Edge Mask ℓ1 Regularization of UGS
GCN	Cora	5·10-4	10-3	10-2
DBLP	10-3	10-4	10-2
OGBN-ArXiv	10-3	0	10-2
Chameleon	10-3	10-4	10-2
Squirrel	5·10-4	0	10-4
Amazon Computers	10-3	10-4	10-2
GIN	Cora	10-3	10-3	10-2
DBLP	10-3	10-3	10-2
OGBN-ArXiv	10-4	0	10-2
ResGCN	Cora	5·10-4	10-3	10-4
DBLP	5·10-4	10-4	10-4
OGBN-ArXiv	10-3	0	10-2
Hyperparameter tuning For each combination of model and dataset separately, we tuned the learning rate, weight decay coefficient, and edge mask  $\ell_1$  regularization coefficient for UGS, and applied the chosen values for evaluating all methods without further tuning (note that the edge mask  $\ell_1$  regularization coefficient is relevant only for UGS). In particular, we carried out a grid search over learning rates  $\{10^{-3}, 5 \cdot 10^{-4}, 10^{-4}\}$ , weight decay coefficients  $\{10^{-3}, 10^{-4}, 0\}$ , and edge mask  $\ell_1$  regularization coefficients  $\{10^{-2}, 10^{-3}, 10^{-4}\}$ . Per hyperparameter configuration, we ran ten repetitions of UGS (differing in random seed), each until all of the input graph's edges were removed. At every edge sparsity level ( $0\%$ ,  $5\%$ ,  $10\%$ , ...,  $100\%$ ), in accordance with [24], we trained a new model with identical hyperparameters, but a fixed edge mask, over each of the ten graphs. We chose the hyperparameters that led to the highest mean validation accuracy, taken over the sparsity levels and ten runs.
Due to the size of the ResGCN model, tuning its hyperparameters entails significant computational costs. Thus, over the Cora and DBLP datasets, per hyperparameter configuration we ran five repetitions of UGS with ResGCN instead of ten. For the large-scale OGBN-ArXiv dataset, we adopted the same hyperparameters used for GCN.
Other To allow more efficient experimentation, we compute the edge removal order of 2-WIS (Algorithm 1) in batches of size 100. Specifically, at each iteration of 2-WIS, instead of removing the edge  $e'$  with maximal walk index tuple  $\mathbf{s}^{(e')}$ , the 100 edges with largest walk index tuples are removed. For randomized edge sparsification algorithms — random pruning, the spectral sparsification method of [93], and the adaptation of UGS [24] — the evaluation runs for a given dataset and percentage of removed edges were carried over sparsified graphs obtained using different random seeds.
I Deferred Proofs
I.1 Additional Notation
For vectors, matrices, or tensors, parenthesized superscripts denote elements in a collection, e.g.  $(\mathbf{a}^{(i)}\in \mathbb{R}^D)_{n = 1}^N$ , while subscripts refer to entries, e.g.  $\mathbf{A}_{d_1,d_2}\in \mathbb{R}$  is the  $(d_{1},d_{2})$ th entry of  $\mathbf{A}\in \mathbb{R}^{D_1\times D_2}$ . A colon is used to indicate a range of entries, e.g.  $\mathbf{a}_{:d}$  is the first  $d$  entries of
$\mathbf{a} \in \mathbb{R}^D$ . We use * to denote tensor contractions (Definition 7),  $\circ$  to denote the Kronecker product, and  $\odot$  to denote the Hadamard product. For  $P \in \mathbb{N}_{\geq 0}$ , the  $P$ 'th Hadamard power operator is denoted by  $\odot^P$ , i.e.  $[\odot^P \mathbf{A}]_{d_1,d_2} = \mathbf{A}_{d_1,d_2}^P$  for  $\mathbf{A} \in \mathbb{R}^{D_1 \times D_2}$ . Lastly, when enumerating over sets of indices an ascending order is assumed.
I.2 Proof of Theorem 2
We assume familiarity with the basic concepts from tensor analysis introduced in Appendix E.1, and rely on the tensor network representations established for GNNs with product aggregation in Appendix E. Specifically, we use the fact that for any  $\mathbf{X} = (\mathbf{x}^{(1)},\dots,\mathbf{x}^{(|\mathcal{V}|)})\in \mathbb{R}^{D_x\times |\mathcal{V}|}$  there exist tree tensor networks  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}^{(t)}(\mathbf{X})$  (described in Appendix E.3 and formally defined in Equations (15) to (20)) such that: (i) their contraction yields  $f^{(\theta ,\mathcal{G})}(\mathbf{X})$  and  $f^{(\theta ,\mathcal{G},t)}(\mathbf{X})$ , respectively (Proposition 1); and (ii) each of their leaves is associated with a vertex feature vector, i.e. one of  $\mathbf{x}^{(1)},\ldots ,\mathbf{x}^{(|\mathcal{V}|)}$ , whereas all other aspects of the tensor networks do not depend on  $\mathbf{x}^{(1)},\ldots ,\mathbf{x}^{(|\mathcal{V}|)}$ .
The proof proceeds as follows. In Appendix I.2.1, by importing machinery from tensor analysis literature (in particular, adapting Claim 7 from [62]), we show that the separation ranks of  $f^{(\theta ,\mathcal{G})}$  and  $f^{(\theta ,\mathcal{G},t)}$  can be upper bounded via cuts in their corresponding tensor networks. Namely,  $\mathrm{sep}(f^{(\theta ,\mathcal{G})};\mathcal{I})$  is at most the minimal multiplicative cut weight in  $\mathcal{T}(\mathbf{X})$ , among cuts separating leaves associated with vertices of the input graph in  $\mathcal{I}$  from leaves associated with vertices of the input graph in  $\mathcal{I}^c$ , where multiplicative cut weight refers to the product of weights belonging to legs crossing the cut. Similarly,  $\mathrm{sep}(f^{(\theta ,\mathcal{G},t)};\mathcal{I})$  is at most the minimal multiplicative cut weight in  $\mathcal{T}^{(t)}(\mathbf{X})$ , among cuts of the same form. We conclude in Appendices I.2.2 and I.2.3 by applying this technique for upper bounding  $\mathrm{sep}(f^{(\theta ,\mathcal{G})};\mathcal{I})$  and  $\mathrm{sep}(f^{(\theta ,\mathcal{G},t)};\mathcal{I})$ , respectively, i.e. by finding cuts in the respective tensor networks with sufficiently low multiplicative weights.
I.2.1 Upper Bounding Separation Rank via Multiplicative Cut Weight in Tensor Network
In a tensor network  $\mathcal{T} = (\mathcal{V}_{\mathcal{T}},\mathcal{E}_{\mathcal{T}},w_{\mathcal{T}})$ , every  $\mathcal{I}_{\mathcal{T}}\subseteq \mathcal{V}_{\mathcal{T}}$  induces a cut  $(\mathcal{I}_{\mathcal{T}},\mathcal{I}_{\mathcal{T}}^{c})$ , i.e. a partition of the nodes into two sets. We denote by  $\mathcal{E}_{\mathcal{T}}(\mathcal{I}_{\mathcal{T}}):=\{\{u,v\}\in\mathcal{E}_{\mathcal{T}}:u\in\mathcal{I}_{\mathcal{T}},v\in\mathcal{I}_{\mathcal{T}}^{c}\}$  the set of legs crossing the cut, and define the multiplicative cut weight of  $\mathcal{I}_{\mathcal{T}}$  to be the product of weights belonging to legs in  $\mathcal{E}_{\mathcal{T}}(\mathcal{I}_{\mathcal{T}})$ , i.e.:
$$
w _ {\mathcal {T}} ^ {\Pi} (\mathcal {I} _ {\mathcal {T}}) := \prod_ {e \in \mathcal {E} _ {\mathcal {T}} (\mathcal {I} _ {\mathcal {T}})} w _ {\mathcal {T}} (e).
$$
For  $\mathbf{X} = (\mathbf{x}^{(1)},\dots ,\mathbf{x}^{(|\mathcal{V}|)})\in \mathbb{R}^{D_x\times |\mathcal{V}|}$ , let  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}^{(t)}(\mathbf{X})$  be the tensor networks corresponding to  $f^{(\theta ,\mathcal{G})}(\mathbf{X})$  and  $f^{(\theta ,\mathcal{G},t)}(\mathbf{X})$  (detailed in Appendix E.3), respectively. Both  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}^{(t)}(\mathbf{X})$  adhere to a tree structure. Each leaf node is associated with a vertex feature vector (i.e. one of  $\mathbf{x}^{(1)},\ldots ,\mathbf{x}^{(|\mathcal{V}|)})$ , while interior nodes are associated with weight matrices or  $\delta$ -tensors. The latter are tensors with modes of equal dimension holding ones on their hyper-diagonal and zeros elsewhere. The restrictions imposed by  $\delta$ -tensors induce a modified notion of multiplicative cut weight, where legs incident to the same  $\delta$ -tensor only contribute once to the weight product (note that weights of legs connected to the same  $\delta$ -tensor are equal since they stand for mode dimensions).
Definition 8. For a tensor network  $\mathcal{T} = (\mathcal{V}_{\mathcal{T}},\mathcal{E}_{\mathcal{T}},w_{\mathcal{T}})$  and subset of nodes  $\mathcal{I}_{\mathcal{T}}\subseteq \mathcal{V}_{\mathcal{T}}$  , let  $\mathcal{E}_{\mathcal{T}}(\mathcal{I}_{\mathcal{T}})$  be the set of edges crossing the cut  $(\mathcal{J}_{\mathcal{T}},\mathcal{J}_{\mathcal{T}}^{c})$  . Denote by  $\widetilde{\mathcal{E}}_{\mathcal{T}}(\mathcal{I}_{\mathcal{T}})\subseteq \mathcal{E}_{\mathcal{T}}(\mathcal{I}_{\mathcal{T}})$  a subset of legs containing for each  $\delta$  -tensor in  $\nu_{T}$  only a single leg from  $\mathcal{E}_{\mathcal{T}}(\mathcal{I}_{\mathcal{T}})$  incident to it, along with all legs in  $\mathcal{E}_{\mathcal{T}}(\mathcal{I}_{\mathcal{T}})$  not connected to  $\delta$  -tensors. Then, the modified multiplicative cut weight of  $\mathcal{I}_{\mathcal{T}}$  is:
$$
\tilde {w} _ {\mathcal {T}} ^ {\Pi} (\mathcal {I} _ {\mathcal {T}}) := \prod_ {e \in \tilde {\mathcal {E}} _ {\mathcal {T}} (\mathcal {I} _ {\mathcal {T}})} w _ {\mathcal {T}} (e).
$$
Lemma 1 establishes that  $\mathrm{sep}(f^{(\theta, \mathcal{G})}; \mathcal{I})$  and  $\mathrm{sep}(f^{(\theta, \mathcal{G}, t)}; \mathcal{I})$  are upper bounded by the minimal modified multiplicative cut weights in  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}^{(t)}(\mathbf{X})$ , respectively, among cuts separating leaves associated with vertices in  $\mathcal{I}$  from leaves associated vertices in  $\mathcal{I}^c$ .
Lemma 1. For any  $\mathbf{X} = (\mathbf{x}^{(1)},\dots ,\mathbf{x}^{(|\mathcal{V}|)})\in \mathbb{R}^{D_x\times |\mathcal{V}|}$ , let  $\mathcal{T}(\mathbf{X}) = (\mathcal{V}_{\mathcal{T}(\mathbf{X})},\mathcal{E}_{\mathcal{T}(\mathbf{X})},w_{\mathcal{T}(\mathbf{X})})$  and  $\mathcal{T}^{(t)}(\mathbf{X}) = (\mathcal{V}_{\mathcal{T}^{(t)}(\mathbf{X})},\mathcal{E}_{\mathcal{T}^{(t)}(\mathbf{X})},w_{\mathcal{T}^{(t)}(\mathbf{X})})$  be the tensor network representations of  $f^{(\theta ,\mathcal{G})}(\mathbf{X})$  and  $f^{(\theta ,\mathcal{G},t)}(\mathbf{X})$  (described in Appendix E.3 and formally defined in Equations (15) to (20)), respectively.
Denote by  $\mathcal{V}_{\mathcal{T}(\mathbf{X})}[\mathcal{I}] \subseteq \mathcal{V}_{\mathcal{T}(\mathbf{X})}$  and  $\mathcal{V}_{\mathcal{T}^{(t)}(\mathbf{X})}[\mathcal{I}] \subseteq \mathcal{V}_{\mathcal{T}^{(t)}(\mathbf{X})}$  the sets of leaf nodes in  $\mathcal{T}(\mathbf{X})$  and  $\mathcal{T}^{(t)}(\mathbf{X})$ , respectively, associated with vertices in  $\mathcal{I}$  from the input graph  $\mathcal{G}$ . Formally:
$$
\mathcal {V} _ {\mathcal {T} (\mathbf {X})} [ \mathcal {I} ] := \left\{\mathbf {x} ^ {(i, \gamma)} \in \mathcal {V} _ {\mathcal {T} (\mathbf {X})}: i \in \mathcal {I}, \gamma \in [ \rho_ {L} (\{i \}, \mathcal {V}) ] \right\},
$$
$$
\mathcal {V} _ {\mathcal {T} ^ {(t)} (\mathbf {X})} [ \mathcal {I} ] := \left\{\mathbf {x} ^ {(i, \gamma)} \in \mathcal {V} _ {\mathcal {T} ^ {(t)} (\mathbf {X})}: i \in \mathcal {I}, \gamma \in [ \rho_ {L} (\{i \}, \{t \}) ] \right\}.
$$