SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- CNN: Same architecture used for the experiments of Figure 3 (see Appendix L.1.1).
- S4: Same architecture used for the experiments of Figure 3 (see Appendix L.1.1).
- Local-Attention: Same architecture used in the experiments of Figure 3, but with a depth 4 network (we reduced the depth to allow for more efficient experimentation).
Training and evaluation: The cross-entropy loss was minimized via the Adam optimizer [32] with default  $\beta_{1},\beta_{2}$  coefficients. Batch sizes were chosen to be the largest powers of two that fit in the GPU memory. The duration of optimization was 200, 200 and 450 epochs for the CNN, S4 and Local-Attention models, respectively (number of epochs was taken large enough such that the train loss plateaued). After the last training epoch, the model which performed best on the validation set was chosen, and its average test accuracy is reported. Additional optimization hyperparameters are provided in Table 9. We note that for S4 [26], in accordance with its official implementation, we use a cosine annealing learning rate scheduler.
Table 9: Optimization hyperparameters for the experiments of Table 1.
Model	Optimizer	Learning Rate	Weight Decay	Batch Size
CNN	Adam	0.001	0.0001	128
S4	AdamW	0.001	0.01	64
Local-Attention	Adam	0.0001	0	32
L.2.3 Tabular Datasets (Tables 2 and 5)
Datasets: The datasets "dna", "semeion" and "isolet" are all from the OpenML repository [60]. For each dataset we split the samples into three folds, which were used for evaluation according to a
standard three-way cross-validation protocol. That is, for each of the three folds, we used one third of the data as a test set and the remaining for train and validation. One third of the samples in the remaining folds (not used for testing) were allocated for the validation set.
- CNN: We used a ResNet adapted for tabular data. It consisted of residual blocks of the following form:
$$
\operatorname {B l o c k} (\mathbf {x}) = \operatorname {d r o p o u t} (\mathbf {x} + \operatorname {B N} (\operatorname {m a x p o o l} (\operatorname {R e L U} (\operatorname {c o n v} (\mathbf {x}))))).
$$
After applying a predetermined amount of residual blocks, a global average pooling and fully connected layers were used to output the prediction. The architectural hyperparameters are specified in Table 10.
- S4: Same architecture used for the experiments of Figure 3 (see Appendix L.1.1), but with a hidden dimension of 64.
- Local-Attention: Same architecture used for the experiments of Figure 3 (see Appendix L.1.1), but with 4 attention heads of dimension 32 and a local-attention window size of 25.
Table 10: Architectural hyperparameters for the convolutional neural network used in the experiments of Tables 2 and 4.
Hyperparameter	Value
Stride	3
Kernel size	3
Pooling window size	3
Number of blocks	8
Hidden dimension	32
Training and evaluation: The cross-entropy loss was minimized for 300 epochs via the Adam optimizer [32] with default  $\beta_{1},\beta_{2}$  coefficients (number of epochs was taken large enough such that the train loss plateaued). Batch sizes were chosen to be the largest powers of two that fit in the GPU memory. After the last training epoch, the model which performed best according to the validation sets was chosen, and test accuracy was measured on the test set. The reported accuracy is the average over the three folds. Additional optimization hyperparameters are specified in Table 11. We note that for S4 [26], in accordance with its official implementation, we use a cosine annealing learning rate scheduler.
Table 11: Optimization hyperparameters for the experiments of Table 2.
Model	Optimizer	Learning Rate	Weight Decay	Batch Size
CNN	Adam	0.001	0.0001	64
S4	AdamW	0.001	0.01	64
Local-Attention	Adam	0.00005	0	64
L.2.4 Randomly Permuted Image Datasets (Table 3)
Dataset: The data acquisition process followed the protocol described in Appendix L.1.2, except that the data was not converted into a binary one-vs-all classification dataset.
- CNN: Same architecture used for the experiments of Figure 8 (see Appendix L.1.2).
Training and evaluation: The cross-entropy loss was minimized for 500 epochs via the Adam optimizer [32] with default  $\beta_{1},\beta_{2}$  coefficients (number of epochs was taken large enough such that the train loss plateaued). Batch size was chosen to be the largest power of two that fit in the GPU memory. After the last training epoch, the model which performed best on the validation set was chosen, and its average test accuracy is reported. Additional optimization hyperparameters are provided in Table 12.
Table 12: Optimization hyperparameters for the experiments of Table 3.
Model	Optimizer	Learning Rate	Weight Decay	Batch Size
CNN	Adam	0.001 (multiplied by 0.1 after 300 epochs)	0.0001	128
L.2.5 Randomly Permuted Audio Datasets With a Large Number of Features (Table 4)
Dataset: The data acquisition process followed the protocol described in Appendix L.1.1, except that the data was not transformed into a binary one-vs-all classification dataset and the audio segments were upsampled from 16,000 to 50,000.
Edge Sparsification: To facilitate running the METIS graph partitioning algorithm over the minimum balanced cut problems encountered as part of Algorithm 2, we first removed edges from the graph using the spectral sparsification algorithm of [57]. Specifically, we used the official Julia implementation (https://github.com/danspielman/Laplacians.jl) with hyperparameter  $\epsilon = 0.15$ .
- CNN: Same architecture used for the experiments of Table 2 (see Appendix L.2.3).
- S4: Same architecture used for the experiments of Table 1 (see Appendix L.1.1), but with a hidden dimension of 32 (we reduced the hidden dimension due to GPU memory considerations).
Training and evaluation: The cross-entropy loss was minimized for 200 epochs via the Adam optimizer [32] with default  $\beta_{1},\beta_{2}$  coefficients (number of epochs was taken large enough such that the train loss plateaued). Batch sizes were chosen to be the largest powers of two that fit in the GPU memory. After the last training epoch, the model which performed best on the validation set was chosen, and its average test accuracy is reported. Additional optimization hyperparameters are provided in Table 13. We note that for S4 [26], in accordance with its official implementation, we use a cosine annealing learning rate scheduler.
Table 13: Optimization hyperparameters for the experiments of Table 4.
Model	Optimizer	Learning Rate	Weight Decay	Batch Size
CNN	Adam	0.001	0.0001	64
S4	AdamW	0.001	0.01	64
M Deferred Proofs
M.1 Useful Lemmas
Below we collect a few useful results, which we will use in our proofs.
Lemma 2. We denote the vector of singular values of a matrix  $\mathbf{X}$  (arranged in decreasing order) by  $S(\mathbf{X})$ . For any  $\mathbf{A}, \mathbf{B} \in \mathbb{R}^{D_1 \times D_2}$  it holds that:
$$
\| S (\mathbf {A}) - S (\mathbf {B}) \| \leq \| S (\mathbf {A} - \mathbf {B}) \|
$$
Proof. See Theorem III.4.4 of [4].
Lemma 3. Let  $\mathcal{P} = \{p_1,\dots,p_N\}$ ,  $\mathcal{Q} = \{q_1,\dots,q_N\}$  be two probability distributions supported on  $[N]$ , and denote by  $TV(\mathcal{P},\mathcal{Q}) := \frac{1}{2}\sum_{n=1}^{N}|p_n - q_n|$  their total variation distance. If for  $\epsilon \in (0,1/2)$  it holds that  $TV(\mathcal{P},\mathcal{Q}) \leq \epsilon$ , then:
$$
\left| H (\mathcal {P}) - H (\mathcal {Q}) \right| \leq H _ {b} (\epsilon) + \epsilon \cdot \ln (N),
$$
where  $H(\mathcal{P}) \coloneqq -\sum_{n=1}^{N} p_n \ln(p_n)$  is the entropy of  $\mathcal{P}$ , and  $H_b(c) \coloneqq -c \cdot \ln(c) - (1 - c) \cdot \ln(1 - c)$  is the binary entropy of a Bernoulli distribution parameterized by  $c \in [0,1]$ .
Proof. See, e.g., Theorem 11 of [27].
Lemma 4 (Hoeffding inequality in Hilbert space). Let  $X_{1},\ldots ,X_{N}$  be an i.i.d. sequence of random variables whose range is some separable Hilbert space  $\mathcal{H}$ . Suppose that  $\mathbb{E}[X_n] = 0$  and  $\| X_{n}\| \leq c$  for all  $n\in [N]$ . Then, for all  $t\geq 0$ :
$$
P r \left(\left\| \frac {1}{n} \sum_ {n = 1} ^ {N} X _ {n} \right\| \geq t\right) \leq 2 \exp \left(- \frac {N t ^ {2}}{2 c ^ {2}}\right),
$$
where  $\| \cdot \|$  refers to the Hilbert space norm.
Proof. See Section 2.4 of [51].
Lemma 5 (adapted from [35]). Let  $G = (V, E)$  be the perfect binary tree graph, with vertices  $V$  and edges  $E$ , underlying the locally connected tensor network that generates  $\mathcal{W}_{\mathrm{TN}} \in \mathbb{R}^{D_1 \times \dots \times D_N}$  (defined in Section 3.1). For  $\mathcal{K} \subseteq [N]$ , let  $V_{\mathcal{K}} \subseteq V$  and  $V_{K^c} \subseteq \mathcal{V}$  be the leaves in  $G$  corresponding to axes indices  $\mathcal{K}$  and  $\mathcal{K}^c$  of  $\mathcal{W}_{\mathrm{TN}}$ , respectively. Lastly, given a cut  $(A, B)$  of  $V$ , i.e.  $A \subseteq V$  and  $B \subseteq V$  are disjoint and  $A \cup B = V$ , denote by  $C(A, B) := \{\{u, v\} \in E : u \in A, v \in B\}$  the edge-cut set. Then:
$$
\operatorname {r a n k} ([   [ \mathcal {W} _ {\mathrm {T N}}; \mathcal {K}   ]   ]) \leq \min  _ {c u t (A, B) o f V s. t. V _ {\mathcal {K}} \subseteq A, V _ {\mathcal {K}} c \subseteq B} R ^ {| C (A, B) |}.
$$
In particular, if  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N$  , then:
$$
\operatorname {r a n k} \left(\llbracket \mathcal {W} _ {\mathrm {T N}}; \mathcal {K} \rrbracket\right) \leq R.
$$
Proof. See Claim 1 in [35] for the upper bound on the rank of  $\llbracket \mathcal{W}_{\mathrm{TN}}; \mathcal{K} \rrbracket$  for any  $\mathcal{K} \subseteq [N]$ . If  $(\mathcal{K}, \mathcal{K}^c) \in \mathcal{C}_N$ , since there exists a cut  $(A, B)$  of  $V$  such that  $V_{\mathcal{K}} \subseteq A$  and  $V_{\mathcal{K}^c} \subseteq B$  whose edge-cut set is of a singleton, we get that  $\text{rank}(\llbracket \mathcal{W}_{\mathrm{TN}}; \mathcal{K} \rrbracket) \leq R$ .
Lemma 6 (adapted from [35]). Let  $G = (V, E)$  be the perfect  $2^P$ -ary tree graph, with vertices  $V$  and edges  $E$ , underlying the locally connected tensor network that generates  $\mathcal{W}_{TN}^P \in \mathbb{R}^{D_1 \times \dots \times D_{NP}}$  (defined in Appendix J.1.1). Furthermore, let  $\mu : [N]^P \to [N^P]$  be a compatible map from  $P$ -dimensional coordinates to axes indices of  $\mathcal{W}_{TN}^P$  (Definition 5). For  $\mathcal{J} \subseteq [N^P]$ , let  $V_{\mathcal{J}} \subseteq V$  and  $V_{\mathcal{J}^c} \subseteq V$  be the leaves in  $G$  corresponding to axes indices  $\mathcal{J}$  and  $\mathcal{J}^c$  of  $\mathcal{W}_{TN}^P$ , respectively. Lastly, given a cut  $(A, B)$  of  $V$ , i.e.  $A \subseteq V$  and  $B \subseteq V$  are disjoint and  $A \cup B = V$ , denote by  $C(A, B) := \{\{u, v\} \in E : u \in A, v \in B\}$  the edge-cut set. Then:
$$
\operatorname {r a n k} \left(\llbracket \mathcal {W} _ {T N} ^ {P}; \mathcal {J} \rrbracket\right) \leq \min  _ {\text {c u t} (A, B)} \left. \begin{array}{l} V \text {s . t .} V _ {\mathcal {I}} \subseteq A, V _ {\mathcal {I} ^ {\mathrm {c}}} \subseteq B \end{array} R ^ {| C (A, B) |}. \right.
$$
In particular, if  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N^P$  , then:
$$
\operatorname {r a n k} \left(\llbracket \mathcal {W} _ {\mathrm {T N}}; \mu (\mathcal {K}) \rrbracket\right) \leq R.
$$
Proof. See Claim 1 in [35] for the upper bound on the rank of  $\left[\mathbb{W}_{\mathrm{TN}}^{P};\mathcal{J}\right]$  for any  $\mathcal{J} \subseteq [N^P]$ . If  $(\mathcal{K},\mathcal{K}^c) \in \mathcal{C}_N^P$ , since there exists a cut  $(A,B)$  of  $V$  such that  $V_{\mu (\mathcal{K})} \subseteq A$  and  $V_{\mu (K)^c} \subseteq B$  whose edge-cut set is of a singleton, we get that  $\mathrm{rank}(\left[\mathbb{W}_{\mathrm{TN}}^{P};\mu (\mathcal{K})\right]) \leq R$ .
Lemma 7 (adapted from Theorem 3.18 in [25]). Let  $\mathcal{A} \in \mathbb{R}^{D_1 \times \ldots \times D_N}$  and  $\epsilon > 0$ . For each canonical partition  $(\mathcal{K}, \mathcal{K}^c) \in \mathcal{C}_N$ , let  $\sigma_{\mathcal{K},1} \geq \ldots \sigma_{\mathcal{K},D_{\mathcal{K}}}$  be the singular values of  $[\mathcal{A};\mathcal{K}]$ , where  $D_{\mathcal{K}} := \min \{\prod_{n \in \mathcal{K}} D_n, \prod_{n \in \mathcal{K}^c} D_n\}$ . Let  $(n_{\mathcal{K}})_{\mathcal{K} \in \mathcal{C}_N} \in \mathbb{N}^{\mathcal{C}_N}$  be an assignment of an integer to each  $\mathcal{K} \in \mathcal{C}_N$ . For any such assignment, consider the set of tensors with Hierarchical Tucker (HT) rank at most  $(n_{\mathcal{K}})_{\mathcal{K} \in \mathcal{C}_N}$  as follows:
$$
H T \left(\left(n _ {\mathcal {K}}\right) _ {\mathcal {K} \in \mathcal {C} _ {N}}\right) := \left\{\mathcal {V} \in \mathbb {R} ^ {D _ {1} \times \dots \times D _ {N}}: \forall \mathcal {K} \in \mathcal {C} _ {N}, \operatorname {r a n k} ([   [ \mathcal {V}; \mathcal {K} ]   ]) \leq n _ {\mathcal {K}} \right\}.
$$
Suppose that for all  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N$  it holds that  $\sqrt{\sum_{d = n_{\mathcal{K}} + 1}^{D_{\mathcal{K}}}\sigma_{\mathcal{K},d}^2}\leq \frac{\epsilon}{\sqrt{2N - 3}}$ . Then, there exists  $\mathcal{W}\in HT((n_{\mathcal{K}})_{\mathcal{K}\in \mathcal{C}_N})$  satisfying:
$$
\| \mathcal {W} - \mathcal {A} \| \leq \epsilon .
$$
In particular, if for all  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N$  it holds that  $\sqrt{\sum_{d = R + 1}^{D_K}\sigma_{\mathcal{K},d}^2}\leq \frac{\epsilon}{\sqrt{2N - 3}}$ , then there exists  $\mathcal{W}_{\mathrm{TN}}\in \mathbb{R}^{D_1\times \dots \times D_N}$  generated by the locally connected tensor network (defined in Section 3.1) satisfying:
Lemma 8 (adapted from Theorem 3.18 in [25]). Let  $(n_{\mathcal{K}})_{\mathcal{K}\in \mathcal{C}_N}\in \mathbb{N}^{\mathcal{C}_N}$  be an assignment of an integer to each  $\mathcal{K}\in \mathcal{C}_N$ . For any such assignment, consider the set of tensors with Hierarchical Tucker (HT) rank at most  $(n_{\mathcal{K}})_{\mathcal{K}\in \mathcal{C}_N}$  as follows:
$$
H T \left(\left(n _ {\mathcal {K}}\right) _ {\mathcal {K} \in \mathcal {C} _ {N}}\right) := \left\{\mathcal {V} \in \mathbb {R} ^ {D _ {1} \times \dots \times D _ {N}}: \forall \mathcal {K} \in \mathcal {C} _ {N}, \operatorname {r a n k} ([ \mathcal {V}; \mathcal {K} ]) \leq n _ {\mathcal {K}} \right\}.
$$
Consider a locally connected tensor network with varying widths  $(n_{\mathcal{K}})_{\mathcal{K}\in \mathcal{C}_N}$ , i.e. a tensor network conforming to a perfect binary tree graph in which the lengths of inner axes are as follows (in contrast to all being equal to  $R$  as in the locally connected tensor network defined in Section 3.1). An axis corresponding to an edge that connects a node with descendant leaves indexed by  $\mathcal{K}$  to its parent is assigned the length  $n_{\mathcal{K}}$ . Then, every  $\mathcal{A}\in HT((n_{\mathcal{K}})_{\mathcal{K}\in \mathcal{C}_N})$  can be represented by said locally connected tensor network with varying widths, meaning there exists an assignment to the tensors of the tensor network such that it generates  $\mathcal{A}$ . In particular, if  $n_K = R$  for all  $\mathcal{K}\in \mathcal{C}_N$ , then  $\mathcal{A}$  can be generated by the locally connected tensor network with all inner axes being of length  $R$ .
Lemma 9. Let  $\mathcal{V},\mathcal{W}\in \mathbb{R}^{D_1\times \dots \times D_N}$  be tensors such that  $\| \mathcal{V}\| = \| \mathcal{W}\| = 1$  and  $\| \mathcal{V} - \mathcal{W}\| < \frac{1}{2}$ . Then, for any  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N$  it holds that:
$$
\left| Q E (\mathcal {V}; \mathcal {K}) - Q E (\mathcal {W}; \mathcal {K}) \right| \leq H _ {b} \left(\left\| \mathcal {V} - \mathcal {W} \right\|\right) + \left\| \mathcal {V} - \mathcal {W} \right\| \cdot \ln \left(D _ {\mathcal {K}}\right),
$$
where  $H_{b}(c) \coloneqq -(c \cdot \ln(c) + (1 - c) \cdot \ln(1 - c))$  is the binary entropy of a Bernoulli distribution parameterized by  $c \in [0,1]$ , and  $\mathcal{D}_{\mathcal{K}} \coloneqq \min \{\prod_{n \in \mathcal{K}} D_{n}, \prod_{n \in \mathcal{K}^{c}} D_{n}\}$ .
Proof. For any matrix  $\mathbf{M} \in \mathbb{R}^{D_1 \times D_2}$  with  $D \coloneqq \min\{D_1, D_2\}$ , let  $S(\mathbf{M}) = (\sigma_{\mathbf{M},1}, \dots, \sigma_{\mathbf{M},D})$  be the vector consisting of its singular values. First note that
$$
\| \mathcal {V} - \mathcal {W} \| = \| S ([ \mathcal {V} - \mathcal {W}; \mathcal {K} ]) \| \leq \| \mathcal {V} - \mathcal {W} \|.
$$
So by Lemma 2
$$
\left\| S \left(\llbracket \mathcal {V}; \mathcal {K} \rrbracket - S \left(\llbracket \mathcal {W}; \mathcal {K} \rrbracket\right) \right. \right\| \leq \left\| \mathcal {V} - \mathcal {W} \right\|.
$$
Let  $v_{1} \geq \dots \geq v_{D_{\mathcal{K}}}$  and  $w_{1} \geq \dots \geq w_{D_{\mathcal{K}}}$  be the singular values of  $[[\mathcal{V}; \mathcal{K}]]$  and  $[[\mathcal{W}; \mathcal{K}]]$ , respectively. We have by the Cauchy-Schwarz inequality that
$$
\left(\sum_ {d = 1} ^ {D _ {\mathcal {K}}} \left| w _ {d} ^ {2} - v _ {d} ^ {2} \right|\right) ^ {2} = \left(\sum_ {d = 1} ^ {D _ {\mathcal {K}}} \left| w _ {d} - v _ {d} \right| \cdot \left| w _ {d} + v _ {d} \right|\right) ^ {2} \leq \left(\sum_ {d = 1} ^ {D _ {\mathcal {K}}} (w _ {d} - v _ {d}) ^ {2}\right) \left(\sum_ {d = 1} ^ {D _ {\mathcal {K}}} (w _ {d} + v _ {d}) ^ {2}\right).
$$
Now the first term is upper bounded by  $\| \mathcal{V} - \mathcal{W} \|^2$ , and for the second we have
$$
\sum_ {d = 1} ^ {D _ {\mathcal {K}}} (w _ {d} + v _ {d}) ^ {2} = \sum_ {d = 1} ^ {D _ {\mathcal {K}}} w _ {d} ^ {2} + \sum_ {d = 1} ^ {D _ {\mathcal {K}}} v _ {d} ^ {2} + 2 v _ {d} w _ {d} = 2 + 2 \sum_ {d = 1} ^ {D _ {\mathcal {K}}} v _ {d} w _ {d} \leq 4,
$$
where we use the fact that  $\| \mathcal{V}\| = \| \mathcal{W}\| = 1$ , and again Cuachy-Schwarz. Overall we have:
$$
\sum_ {d = 1} ^ {D _ {\mathcal {K}}} \left| w _ {d} ^ {2} - v _ {d} ^ {2} \right| \leq 2 \| \mathcal {V} - \mathcal {W} \|.
$$
Note that the left hand side of the inequality above equals twice the total variation distance between the distributions defined by  $\{w_d^2\}_{d = 1}^{D_\kappa}$  and  $\{v_{d}^{2}\}_{d = 1}^{D_{\kappa}}$ . Therefore by Lemma 3 we have:
$$
\left| Q E (\mathcal {V}; \mathcal {K}) - Q E (\mathcal {W}; \mathcal {K}) \right| = \left| H \left(\left\{w _ {d} ^ {2} \right\}\right) - H \left(\left\{v _ {d} ^ {2} \right\}\right) \right| \leq \| \mathcal {V} - \mathcal {W} \| \cdot \ln \left(D _ {\mathcal {K}}\right) + H _ {b} \left(\| \mathcal {V} - \mathcal {W} \|\right).
$$
Lemma 10. Let  $\mathcal{P} = \{p(x)\}_{x\in [S]}$ , where  $S\in \mathbb{N}$ , be a probability distribution, and denote its entropy by  $H(\mathcal{P}):= \mathbb{E}_{x\sim \mathcal{P}}[\ln (1 / p(x))]$ . Then, for any  $0 < a < 1$ , there exists a subset  $T\subseteq [S]$  such that  $Pr_{x\sim \mathcal{P}}(T^c)\leq a$  and  $|T|\leq e^{\frac{H(\mathcal{P})}{a}}$ .
Proof. By Markov's inequality we have for any  $0 < a < 1$ :
$$
P r _ {x \sim \mathcal {P}} \left(\left\{x: e ^ {- \frac {H (P)}{a}} \geq p (x) \right\}\right) = P r _ {x \sim \mathcal {P}} \left(\left\{x: \ln \left(\frac {1}{p (x)}\right) \geq \frac {H (P)}{a} \right\}\right) \leq a.
$$
Let  $T := \{x : e^{-\frac{H(P)}{a}} \leq p(x)\} \subseteq [S]$ . Note that
$$
e ^ {- \frac {H (\mathcal {P})}{a}} | T | \leq \sum_ {x \in T} p (x) \leq \sum_ {x \in [ S ]} p (x) = 1,
$$
and so  $|T|\leq e^{\frac{H(P)}{a}}$  and  $Pr_{x\sim \mathcal{P}}(T^c)\leq a$  , as required.
M.2 Proof of Theorem 1
If  $\mathcal{A} = 0$  the theorem is trivial, since then  $QE(\mathcal{A};\mathcal{K}) = 0$  for all  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N$ , so we can assume  $\mathcal{A}\neq 0$ . We have:
$$
\begin{array}{l} \left\| \frac {\mathcal {W} _ {\mathrm {T N}}}{\| \mathcal {W} _ {\mathrm {T N}} \|} - \frac {\mathcal {A}}{\| \mathcal {A} \|} \right\| = \frac {1}{\| \mathcal {A} \|} \left\| \frac {\| \mathcal {A} \|}{\| \mathcal {W} _ {\mathrm {T N}} \|} \cdot \mathcal {W} _ {\mathrm {T N}} - \mathcal {A} \right\| \\ \leq \frac {1}{\| \mathcal {A} \|} \left(\left| \frac {\| \mathcal {A} \|}{\| \mathcal {W} _ {\mathrm {T N}} \|} - 1 \right| \cdot \| \mathcal {W} _ {\mathrm {T N}} \| + \| \mathcal {W} _ {\mathrm {T N}} - \mathcal {A} \|\right) \\ = \frac {1}{\| \mathcal {A} \|} \left(\| \mathcal {A} \| - \| \mathcal {W} _ {\mathrm {T N}} \| \right| + \| \mathcal {W} _ {\mathrm {T N}} - \mathcal {A} \|) \\ \leq \frac {2 \epsilon}{\| A \|}. \\ \end{array}
$$
Now, let  $\hat{\mathcal{A}}\coloneqq \frac{\mathcal{A}}{\|\mathcal{A}\|}$  and  $\hat{\mathcal{W}}_{\mathrm{TN}} = \frac{\mathcal{W}_{\mathrm{TN}}}{\|\mathcal{W}_{\mathrm{TN}}\|}$  be normalized versions of  $\mathcal{A}$  and  $\mathcal{W}_{\mathrm{TN}}$ , respectively, and let  $c = \frac{2\epsilon}{\|\mathcal{A}\|}$ . Note that  $c < \frac{2\|\mathcal{A}\|}{4} \frac{1}{\|\mathcal{A}\|} = \frac{1}{2}$ , and therefore by Lemma 9 we have:
$$
\begin{array}{l} \left| Q E (\mathcal {A}; \mathcal {K}) - Q E (\mathcal {W} _ {\mathrm {T N}}; \mathcal {K}) \right| = \left| Q E \left(\hat {\mathcal {A}}; \mathcal {K}\right) - Q E \left(\hat {\mathcal {W}} _ {\mathrm {T N}}; \mathcal {K}\right) \right| \\ \leq c \cdot \ln (D _ {\mathcal {K}}) + H _ {b} (c). \\ \end{array}
$$
By Lemma 5 we have that
$$
Q E \left(\hat {\mathcal {W}} _ {\mathrm {T N}}; \mathcal {K}\right) \leq \ln \left(\operatorname {r a n k} \left(\llbracket \mathcal {W} _ {\mathrm {T N}}; \mathcal {K} \rrbracket\right)\right) \leq \ln (R),
$$
$$
Q E \left(\hat {\mathcal {A}}; \mathcal {K}\right) \leq \ln (R) + c \ln \left(D _ {\mathcal {K}}\right) + H _ {b} (c).
$$
Substituting  $c = \frac{2\epsilon}{\|A\|}$  and invoking the elementary inequality  $H_{b}(x) \leq 2\sqrt{x}$  we obtain
$$
Q E (\mathcal {A}; \mathcal {K}) \leq \ln (R) + \frac {2 \epsilon}{\| \mathcal {A} \|} \cdot \ln (D _ {\mathcal {K}}) + 2 \sqrt {\frac {2 \epsilon}{\| \mathcal {A} \|}},
$$
As for the expected entanglements of a random tensor, let  $\mathcal{A}' \in \mathbb{R}^{D_1 \times \dots \times D_N}$  be drawn according to the uniform distribution over the set of unit norm tensors. According to [52], for any canonical partition  $(\mathcal{K}, \mathcal{K}^c) \in \mathcal{C}_N$  it holds that:
$$
\mathbb {E} [ Q E (\mathcal {A}; \mathcal {K}) ] = \sum_ {k = D _ {\mathcal {K}} ^ {\prime} + 1} ^ {D _ {\mathcal {K}} \cdot D _ {\mathcal {K}} ^ {\prime}} \frac {1}{k} - \frac {D _ {\mathcal {K}} - 1}{2 D _ {\mathcal {K}} ^ {\prime}},
$$
where  $D_{\mathcal{K}}^{\prime} \coloneqq \max \{\prod_{n \in \mathcal{K}} D_n, \prod_{n \in \mathcal{K}^c} D_n\}$  and recall that  $D_{\mathcal{K}} \coloneqq \min \{\prod_{n \in \mathcal{K}} D_n, \prod_{n \in \mathcal{K}^c} D_n\}$ . By the elementary inequality of  $\sum_{k = q}^{p} 1 / k \geq \ln(p) - \ln(q)$  we therefore have that:
$$
\begin{array}{l} \mathbb {E} \left[ Q E \left(\mathcal {A} ^ {\prime}; \mathcal {K}\right) \right] \geq \ln \left(D _ {\mathcal {K}} \cdot D _ {\mathcal {K}} ^ {\prime}\right) - \ln \left(D _ {\mathcal {K}} ^ {\prime} + 1\right) - \frac {D _ {\mathcal {K}} - 1}{2 D _ {\mathcal {K}} ^ {\prime}} \\ = \ln (D _ {\mathcal {K}}) + \ln \left(\frac {D _ {\mathcal {K}} ^ {\prime}}{D _ {\mathcal {K}} ^ {\prime} + 1}\right) - \frac {D _ {\mathcal {K}} - 1}{2 D _ {\mathcal {K}} ^ {\prime}}. \\ \end{array}
$$
Since  $D_{\mathcal{K}} \leq D_{\mathcal{K}}'$  and  $D_{\mathcal{K}} \geq 1$ , it holds that  $(D_{\mathcal{K}} - 1) / 2D_{\mathcal{K}}' \leq 1/2$  and  $\ln(D_{\mathcal{K}}' / (D_{\mathcal{K}}' + 1)) \geq \ln(1/2)$ . Thus:
$$
\mathbb {E} [ Q E (\mathcal {A} ^ {\prime}; \mathcal {K}) ] \geq \ln (D _ {\mathcal {K}}) + \ln \left(\frac {1}{2}\right) - \frac {1}{2} \geq \min  \{| \mathcal {K} |, | \mathcal {K} ^ {c} | \} \cdot \ln \left(\min  _ {n \in [ N ]} D _ {n}\right) + \ln \left(\frac {1}{2}\right) - \frac {1}{2}.
$$
M.3 Proof of Theorem 2
Consider, for each canonical partition  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N$  , the distribution
$$
\mathcal {P} _ {\mathcal {K}} = \left\{p _ {\mathcal {K}} (i) := \frac {\sigma_ {\mathcal {K} , i} ^ {2}}{\| \mathcal {A} \| ^ {2}} \right\} _ {i \in [ D _ {\mathcal {K}} ]},
$$