SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\sigma_{\mathcal{K},1} \geq \sigma_{\mathcal{K},2} \geq \ldots \geq \sigma_{\mathcal{K},D_{\mathcal{K}}}$  are the singular values of  $[\mathcal{A};\mathcal{K}]$  (note that  $\frac{1}{\|\mathcal{A}\|^2} \sum_j \sigma_{\mathcal{K},j}^2 = \frac{\|\mathcal{A}\|^2}{\|\mathcal{A}\|^2} = 1$  so  $\mathcal{P}_{\mathcal{K}}$  is indeed a probability distribution). Denoting by  $H(\mathcal{P}_{\mathcal{K}}) \coloneqq \mathbb{E}_{i \sim \mathcal{P}_{\mathcal{K}}}[\ln(1/p_{\mathcal{K}}(i))]$  the entropy of  $\mathcal{P}_{\mathcal{K}}$ , by assumption:
$$
Q E (\mathcal {A}; \mathcal {K}) = H (\mathcal {P} _ {\mathcal {K}}) \leq \frac {\epsilon^ {2}}{\| \mathcal {A} \| ^ {2} (2 N - 3)} \ln (R),
$$
for all  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N$ . Thus, taking  $a = \frac{\epsilon^2}{\|\mathcal{A}\|^2(2N - 3)}$  we obtain by Lemma 10 that there exists a subset  $T_{\mathcal{K}}\subseteq [D_{\mathcal{K}}]$  such that
$$
\mathcal {P} _ {\mathcal {K}} (T _ {\mathcal {K}} ^ {c}) \leq \frac {\epsilon^ {2}}{(2 N - 3) \| \mathcal {A} \| ^ {2}},
$$
and  $|T_{\mathcal{K}}| \leq e^{\frac{H(\mathcal{P}_{\mathcal{K}})}{c}} = e^{\ln (R)} = R$ . Note that
$$
\mathcal {P} _ {\mathcal {K}} \left(T _ {\mathcal {K}}\right) \leq \sum_ {i = 1} ^ {R} \frac {\sigma_ {i} ^ {2}}{\| \mathcal {A} \| ^ {2}}.
$$
Since this holds for any subset of cardinality at most  $R$ . Taking complements we obtain
$$
\sum_ {i = R + 1} ^ {D _ {\mathcal {K}}} \frac {\sigma_ {i} ^ {2}}{\| \mathcal {A} \| ^ {2}} \leq \mathcal {P} _ {\mathcal {K}} (T _ {\mathcal {K}} ^ {c}),
$$
$$
\sqrt {\sum_ {i = R + 1} ^ {D \mathcal {K}} \sigma_ {\mathcal {K} , i} ^ {2}} \leq \frac {\epsilon}{\sqrt {(2 N - 3)}}.
$$
We can now invoke Lemma 7, which implies that there exists some  $\mathcal{W}_{\mathrm{TN}} \in \mathbb{R}^{D_1 \times \dots \times D_N}$  generated by the locally connected tensor network satisfying:
M.4 Proof of Corollary 1
Notice that the entanglements of a tensor are invariant to multiplication by a constant. In particular,  $QE(\mathcal{D}_{\mathrm{pop}};\mathcal{K}) = QE(\mathcal{D}_{\mathrm{pop}} / \|\mathcal{D}_{\mathrm{pop}}\|; \mathcal{K})$  for any  $(\mathcal{K}, \mathcal{K}^c) \in \mathcal{C}_N$ . Hence, if there exists a canonical partition  $(\mathcal{K}, \mathcal{K}^c) \in \mathcal{C}_N$  under which  $QE(\mathcal{D}_{\mathrm{pop}};\mathcal{K}) > \ln(R) + 2\epsilon \cdot \ln(D_{\mathcal{K}}) + 2\sqrt{2\epsilon}$ , then Theorem 1 implies that  $\min_{\mathcal{W}_{\mathrm{TN}}} \|\mathcal{W}_{\mathrm{TN}} - \mathcal{D}_{\mathrm{pop}} / \|\mathcal{D}_{\mathrm{pop}}\| > \epsilon$ . Now, for any non-zero  $\mathcal{W} \in \mathbb{R}^{D_1 \times \dots D_N}$  generated by the locally connected tensor network, one can also represent  $\mathcal{W} / \|\mathcal{W}\|$  by multiplying any of the tensors constituting the tensor network by  $1 / \|\mathcal{W}\|$  (contraction is a multilinear operation). Thus:
$$
\mathrm {S u b O p t} := \min _ {\mathcal {W} _ {\mathrm {T N}}} \left\| \frac {\mathcal {W} _ {\mathrm {T N}}}{\| \mathcal {W} _ {\mathrm {T N}} \|} - \frac {\mathcal {D} _ {\mathrm {p o p}}}{\| \mathcal {D} _ {\mathrm {p o p}} \|} \right\| \geq \min _ {\mathcal {W} _ {\mathrm {T N}}} \left\| \mathcal {W} _ {\mathrm {T N}} - \frac {\mathcal {D} _ {\mathrm {p o p}}}{\| \mathcal {D} _ {\mathrm {p o p}} \|} \right\| > \epsilon ,
$$
which concludes the first part of the claim, i.e. the necessary condition for low suboptimality in achievable accuracy.
For the sufficient condition, if for all  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N$  it holds that  $QE(\mathcal{D}_{\mathrm{pop}};\mathcal{K})\leq \frac{\epsilon^2}{8N - 12}\cdot \ln (R)$ , then by Theorem 2 there exists an assignment for the locally connected tensor network such that  $\| \mathcal{W}_{\mathrm{TN}} - \mathcal{D}_{\mathrm{pop}} / \| \mathcal{D}_{\mathrm{pop}}\| \| \leq \epsilon /2$ . From the triangle inequality we obtain:
$$
\left\| \frac {\mathcal {W} _ {\mathrm {T N}}}{\| \mathcal {W} _ {\mathrm {T N}} \|} - \frac {\mathcal {D} _ {\text {p o p}}}{\| \mathcal {D} _ {\text {p o p}} \|} \right\| \leq \left\| \mathcal {W} _ {\mathrm {T N}} - \frac {\mathcal {D} _ {\text {p o p}}}{\| \mathcal {D} _ {\text {p o p}} \|} \right\| + \left\| \mathcal {W} _ {\mathrm {T N}} - \frac {\mathcal {W} _ {\mathrm {T N}}}{\| \mathcal {W} _ {\mathrm {T N}} \|} \right\| \leq \frac {\epsilon}{2} + \left\| \mathcal {W} _ {\mathrm {T N}} - \frac {\mathcal {W} _ {\mathrm {T N}}}{\| \mathcal {W} _ {\mathrm {T N}} \|} \right\|. \tag {11}
$$
Since  $\| \mathcal{W}_{\mathrm{TN}} - \mathcal{D}_{\mathrm{pop}} / \| \mathcal{D}_{\mathrm{pop}}\| \| \leq \epsilon /2$  it holds that  $\| \mathcal{W}_{\mathrm{TN}}\| \leq 1 + \epsilon /2$ . Combined with the fact that  $\left\| \mathcal{W}_{\mathrm{TN}} - \frac{\mathcal{W}_{\mathrm{TN}}}{\|\mathcal{W}_{\mathrm{TN}}\|}\right\| = \left\| \mathcal{W}_{\mathrm{TN}}\right\| -1$ , we get that  $\left\| \mathcal{W}_{\mathrm{TN}} - \frac{\mathcal{W}_{\mathrm{TN}}}{\|\mathcal{W}_{\mathrm{TN}}\|}\right\| \leq \epsilon /2$ . Plugging this into Equation (11) yields:
$$
\left\| \frac {\mathcal {W} _ {\mathrm {T N}}}{\| \mathcal {W} _ {\mathrm {T N}} \|} - \frac {\mathcal {D} _ {\mathrm {p o p}}}{\| \mathcal {D} _ {\mathrm {p o p}} \|} \right\| \leq \epsilon ,
$$
and so  $\mathrm{SubOpt} := \min_{\mathcal{W}_{\mathrm{TN}}} \left\| \frac{\mathcal{W}_{\mathrm{TN}}}{\|\mathcal{W}_{\mathrm{TN}}\|} - \frac{\mathcal{D}_{\mathrm{pop}}}{\|\mathcal{D}_{\mathrm{pop}}\|} \right\| \leq \epsilon.$
M.5 Proof of Proposition 2
We have the identity
$$
\left\| \frac {\mathcal {D} _ {\text {p o p}}}{\| \mathcal {D} _ {\text {p o p}} \|} - \frac {\mathcal {D} _ {\text {e m p}}}{\| \mathcal {D} _ {\text {e m p}} \|} \right\| = \left\| \frac {\mathcal {D} _ {\text {p o p}} \| \mathcal {D} _ {\text {e m p}} \| - \mathcal {D} _ {\text {e m p}} \| \mathcal {D} _ {\text {p o p}} \|}{\| \mathcal {D} _ {\text {p o p}} \| \| \mathcal {D} _ {\text {e m p}} \|} \right\| =
$$
$$
\left\| \frac {\mathcal {D} _ {\text {p o p}} \| \mathcal {D} _ {\text {e m p}} \| - \| \mathcal {D} _ {\text {e m p}} \| \mathcal {D} _ {\text {e m p}} + \mathcal {D} _ {\text {e m p}} \| \mathcal {D} _ {\text {e m p}} \| - \mathcal {D} _ {\text {e m p}} \| \mathcal {D} _ {\text {p o p}} \|}{\| \mathcal {D} _ {\text {p o p}} \| \| \mathcal {D} _ {\text {e m p}} \|} \right\|.
$$
By the triangle inequality the above is bounded by
$$
\frac {\left\| \mathcal {D} _ {\mathrm {p o p}} - \mathcal {D} _ {\mathrm {e m p}} \right\|}{\left\| \mathcal {D} _ {\mathrm {p o p}} \right\|} + \frac {\left\| \left\| \mathcal {D} _ {\mathrm {p o p}} \right\| - \left\| \mathcal {D} _ {\mathrm {e m p}} \right\| \right\|}{\left\| \mathcal {D} _ {\mathrm {p o p}} \right\|}.
$$
For  $m \in [M]$ , let  $\mathcal{X}^{(m)} = y^{(m)} \cdot \otimes_{n \in [N]} \mathbf{x}^{(n,m)} - \mathcal{D}_{\mathrm{pop}}$ . These are i.i.d. random variables with  $\mathbb{E}[\mathcal{X}^{(m)}] = 0$  and  $\| \mathcal{X}^{(m)} \| \leq 2$  for all  $m \in [M]$ . Note that
$$
\left\| \frac {1}{M} \sum_ {m = 1} ^ {M} \mathcal {X} ^ {(m)} \right\| = \left\| \mathcal {D} _ {\text {e m p}} - \mathcal {D} _ {\text {p o p}} \right\|,
$$
so by Lemma 4 with  $c = 2, t = \frac{\|\mathcal{D}_{\mathrm{pop}}\|^{\gamma}}{2}$ , assuming  $M \geq \frac{2\ln(\frac{2}{\delta})}{\|\mathcal{D}_{\mathrm{pop}}\|^2\gamma^2}$  we have with probability at least  $1 - \delta$
$$
\left| \left\| \mathcal {D} _ {\mathrm {p o p}} \right\| - \left\| \mathcal {D} _ {\mathrm {e m p}} \right\| \right| \leq \left\| \mathcal {D} _ {p o p} - \mathcal {D} _ {\mathrm {e m p}} \right\| \leq \frac {\left\| \mathcal {D} _ {\mathrm {p o p}} \right\| \gamma}{2},
$$
$$
\frac {\left\| \mathcal {D} _ {\text {p o p}} - \mathcal {D} _ {\text {e m p}} \right\|}{\left\| \mathcal {D} _ {\text {p o p}} \right\|} + \frac {\left\| \left\| \mathcal {D} _ {\text {p o p}} \right\| - \left\| \mathcal {D} _ {\text {e m p}} \right\| \right\|}{\left\| \mathcal {D} _ {\text {p o p}} \right\|} \leq \gamma .
$$
M.6 Proof of Corollary 2
First, by Proposition 2, for  $M \geq \frac{8\ln(\frac{2}{\delta})}{\|\mathcal{D}_{\mathrm{pop}}\|^2\epsilon^2}$  we have that with probability at least  $1 - \delta$ :
$$
\left\| \frac {\mathcal {D} _ {\text {p o p}}}{\| \mathcal {D} _ {\text {p o p}} \|} - \frac {\mathcal {D} _ {\text {e m p}}}{\| \mathcal {D} _ {\text {e m p}} \|} \right\| \leq \frac {\epsilon}{2}. \tag {12}
$$
Now, we establish the necessary condition on the entanglements of  $\mathcal{D}_{\mathrm{emp}}$ . Assume that there exists a canonical partition  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N$  under which
$$
Q E \left(\mathcal {D} _ {\mathrm {e m p}}; \mathcal {K}\right) > \ln (R) + 3 \epsilon \cdot \ln \left(D _ {\mathcal {K}}\right) + 2 \sqrt {3 \epsilon}.
$$
We may view  $\mathcal{D}_{\mathrm{emp}}$  as the population data tensor for the uniform data distribution over the training instances  $\left\{\left(\left(\mathbf{x}^{(1,m)},\dots ,\mathbf{x}^{(N,m)}\right),y^{(m)}\right)\right\}_{m = 1}^{M}$ . Thus, Corollary 1 implies that
$$
\min _ {\mathcal {W} _ {\mathrm {T N}}} \left\| \frac {\mathcal {W} _ {\mathrm {T N}}}{\| \mathcal {W} _ {\mathrm {T N}} \|} - \frac {\mathcal {D} _ {\mathrm {e m p}}}{\| \mathcal {D} _ {\mathrm {e m p}} \|} \right\| > 1. 5 \epsilon .
$$
Using the triangle inequality after adding and subtracting  $\frac{\mathcal{D}_{\mathrm{pop}}}{\|\mathcal{D}_{\mathrm{pop}}\|}$ , it follows that
$$
\min  _ {\mathcal {W} _ {\mathrm {T N}}} \left\{\left\| \frac {\mathcal {W} _ {\mathrm {T N}}}{\| \mathcal {W} _ {\mathrm {T N}} \|} - \frac {\mathcal {D} _ {\text {p o p}}}{\| \mathcal {D} _ {\text {p o p}} \|} \right\| + \left\| \frac {\mathcal {D} _ {\text {p o p}}}{\| \mathcal {D} _ {\text {p o p}} \|} - \frac {\mathcal {D} _ {\text {e m p}}}{\| \mathcal {D} _ {\text {e m p}} \|} \right\| \right\} > 1. 5 \epsilon .
$$
Hence, combined with Equation (12) this concludes the current part of the proof:
$$
\mathrm {S u b O p t} := \min _ {\mathcal {W} _ {\mathrm {T N}}} \left\| \frac {\mathcal {W} _ {\mathrm {T N}}}{\| \mathcal {W} _ {\mathrm {T N}} \|} - \frac {\mathcal {D} _ {\mathrm {p o p}}}{\| \mathcal {D} _ {\mathrm {p o p}} \|} \right\| > 1. 5 \epsilon - \left\| \frac {\mathcal {D} _ {\mathrm {p o p}}}{\| \mathcal {D} _ {\mathrm {p o p}} \|} - \frac {\mathcal {D} _ {\mathrm {e m p}}}{\| \mathcal {D} _ {\mathrm {e m p}} \|} \right\| \geq 1. 5 \epsilon - \frac {\epsilon}{2} = \epsilon .
$$
We turn our attention to the sufficient condition on the entanglements of  $\mathcal{D}_{\mathrm{emp}}$ . Suppose that  $QE(\mathcal{D}_{\mathrm{emp}};\mathcal{K}) \leq \frac{\epsilon^2}{32N - 48} \cdot \ln(R)$  for all canonical partitions  $(\mathcal{K},\mathcal{K}^c) \in \mathcal{C}_N$ . Invoking Corollary 1 while viewing  $\mathcal{D}_{\mathrm{emp}}$  as the population data tensor for the uniform data distribution over the training instances  $\left\{\left(\mathbf{x}^{(1,m)},\dots,\mathbf{x}^{(N,m)}\right),y^{(m)}\right\}_{m=1}^{M}$ , we get that
$$
\min _ {\mathcal {W} _ {\mathrm {T N}}} \left\| \frac {\mathcal {W} _ {\mathrm {T N}}}{\| \mathcal {W} _ {\mathrm {T N}} \|} - \frac {\mathcal {D} _ {\mathrm {e m p}}}{\| \mathcal {D} _ {\mathrm {e m p}} \|} \right\| \leq \frac {\epsilon}{2}.
$$
Combined with Equation (12), and by the triangle inequality, we conclude:
$$
\begin{array}{l} \text {S u b O p t} := \min  _ {\mathcal {W} _ {\mathrm {T N}}} \left\| \frac {\mathcal {W} _ {\mathrm {T N}}}{\| \mathcal {W} _ {\mathrm {T N}} \|} - \frac {\mathcal {D} _ {\mathrm {p o p}}}{\| \mathcal {D} _ {\mathrm {p o p}} \|} \right\| \\ \leq \min _ {\mathcal {W} _ {\mathrm {T N}}} \left\| \frac {\mathcal {W} _ {\mathrm {T N}}}{\| \mathcal {W} _ {\mathrm {T N}} \|} - \frac {\mathcal {D} _ {\mathrm {e m p}}}{\| \mathcal {D} _ {\mathrm {e m p}} \|} \right\| + \left\| \frac {\mathcal {D} _ {\mathrm {p o p}}}{\| \mathcal {D} _ {\mathrm {p o p}} \|} - \frac {\mathcal {D} _ {\mathrm {e m p}}}{\| \mathcal {D} _ {\mathrm {e m p}} \|} \right\| \\ \leq \epsilon . \\ \end{array}
$$
Lastly, the fact that the entanglements of  $\mathcal{D}_{\mathrm{emp}}$  can be evaluated efficiently is discussed in Appendix F.
M.7 Proof of Theorem 3
If  $\mathcal{A} = 0$  the theorem is trivial, since then  $QE(\mathcal{A};\mu (\mathcal{K})) = 0$  for all  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N^P$ , so we can assume  $\mathcal{A}\neq 0$ . We have:
$$
\begin{array}{l} \left\| \frac {\mathcal {W} _ {\mathrm {T N}} ^ {P}}{\| \mathcal {W} _ {\mathrm {T N}} ^ {P} \|} - \frac {\mathcal {A}}{\| \mathcal {A} \|} \right\| = \frac {1}{\| \mathcal {A} \|} \left\| \frac {\| \mathcal {A} \|}{\| \mathcal {W} _ {\mathrm {T N}} ^ {P} \|} \cdot \mathcal {W} _ {\mathrm {T N}} ^ {P} - \mathcal {A} \right\| \\ \leq \frac {1}{\| \mathcal {A} \|} \left(\left| \frac {\| \mathcal {A} \|}{\| \mathcal {W} _ {\mathrm {T N}} ^ {P} \|} - 1 \right| \cdot \| \mathcal {W} _ {\mathrm {T N}} ^ {P} \| + \| \mathcal {W} _ {\mathrm {T N}} ^ {P} - \mathcal {A} \|\right) \\ = \frac {1}{\| \mathcal {A} \|} \left(\left| \| \mathcal {A} \| - \| \mathcal {W} _ {\mathrm {T N}} ^ {P} \| \right| + \| \mathcal {W} _ {\mathrm {T N}} ^ {P} - \mathcal {A} \|\right) \\ \leq \frac {2 \epsilon}{\| A \|}. \\ \end{array}
$$
Now, let  $\hat{\mathcal{A}}\coloneqq \frac{\mathcal{A}}{\|\mathcal{A}\|}$  and  $\hat{\mathcal{W}}_{\mathrm{TN}}^{P} = \frac{\mathcal{W}_{\mathrm{TN}}^{P}}{\|\mathcal{W}_{\mathrm{TN}}^{P}\|}$  be normalized versions of  $\mathcal{A}$  and  $\mathcal{W}_{\mathrm{TN}}^P$  , respectively, and let  $c = \frac{2\epsilon}{\|A\|}$  . Note that  $c <   \frac{2\|\mathcal{A}\|}{4}\frac{1}{\|\mathcal{A}\|} = \frac{1}{2},$  and therefore by Lemma 9 we have:
$$
\begin{array}{l} \left. \left| Q E (\mathcal {A}; \mu (\mathcal {K})) - Q E \left(\mathcal {W} _ {\mathrm {T N}} ^ {P}; \mu (\mathcal {K})\right) \right| = \left| Q E \left(\hat {\mathcal {A}}; \mu (\mathcal {K})\right) - Q E \left(\hat {\mathcal {W}} _ {\mathrm {T N}} ^ {P}; \mu (\mathcal {K})\right) \right| \right. \\ \leq c \cdot \ln \left(D _ {\mu (\mathcal {K})}\right) + H _ {b} (c). \\ \end{array}
$$
By Lemma 6 we have that
$$
Q E \left(\hat {\mathcal {W}} _ {\mathrm {T N}} ^ {P}; \mu (\mathcal {K})\right) \leq \ln (\operatorname {r a n k} \left(\llbracket \mathcal {W} _ {\mathrm {T N}} ^ {\mathcal {P}}; \mu (\mathcal {K}) \rrbracket\right)) \leq \ln (R),
$$
$$
Q E \left(\hat {\mathcal {A}}; \mu (\mathcal {K})\right) \leq \ln (R) + c \ln \left(D _ {\mu (\mathcal {K})}\right) + H _ {b} (c).
$$
$$
Q E (\mathcal {A}; \mu (\mathcal {K})) \leq \ln (R) + \frac {2 \epsilon}{\| \mathcal {A} \|} \cdot \ln (D _ {\mu (\mathcal {K})}) + 2 \sqrt {\frac {2 \epsilon}{\| \mathcal {A} \|}},
$$
As for Equation (7), let  $\mathcal{A}' \in \mathbb{R}^{D_1 \times \dots \times D_{NP}}$  be drawn according to the uniform distribution over the set of unit norm tensors. According to [52], for any canonical partition  $(\mathcal{K}, \mathcal{K}^c) \in \mathcal{C}_N^P$  it holds that:
$$
\mathbb {E} [ Q E (\mathcal {A} ^ {\prime}; \mu (\mathcal {K})) ] = \sum_ {k = D _ {\mu (\mathcal {K})} ^ {\prime} + 1} ^ {D _ {\mu (\mathcal {K})} \cdot D _ {\mu (\mathcal {K})} ^ {\prime}} \frac {1}{k} - \frac {D _ {\mu (\mathcal {K})} - 1}{2 D _ {\mu (\mathcal {K})} ^ {\prime}},
$$
where we denote  $D_{\mu(\mathcal{K})}' \coloneqq \max \{\prod_{n \in \mu(\mathcal{K})} D_n, \prod_{n \in \mu(\mathcal{K})^c} D_n\}$  and recall that  $D_{\mu(\mathcal{K})} \coloneqq \min \{\prod_{n \in \mu(\mathcal{K})} D_n, \prod_{n \in \mu(\mathcal{K})^c} D_n\}$ . By the elementary inequality of  $\sum_{k=q}^{p} 1/k \geq \ln(p) - \ln(q)$  we therefore have that:
$$
\begin{array}{l} \mathbb {E} [ Q E (\mathcal {A} ^ {\prime}; \mu (\mathcal {K})) ] \geq \ln (D _ {\mu (\mathcal {K})} \cdot D _ {\mu (\mathcal {K})} ^ {\prime}) - \ln (D _ {\mu (\mathcal {K})} ^ {\prime} + 1) - \frac {D _ {\mu (\mathcal {K})} - 1}{2 D _ {\mu (\mathcal {K})} ^ {\prime}} \\ = \ln (D _ {\mu (\mathcal {K})}) + \ln \left(\frac {D _ {\mu (\mathcal {K})} ^ {\prime}}{D _ {\mu (\mathcal {K})} ^ {\prime} + 1}\right) - \frac {D _ {\mu (\mathcal {K})} - 1}{2 D _ {\mu (\mathcal {K})} ^ {\prime}}. \\ \end{array}
$$
Since  $D_{\mu (\mathcal{K})}\leq D_{\mu (\mathcal{K})}^{\prime}$  and  $D_{\mu (\mathcal{K})}\geq 1$  , we have that:
$$
\frac {D _ {\mu (\mathcal {K})} - 1}{2 D _ {\mu (\mathcal {K})} ^ {\prime}} \leq \frac {1}{2}, \ln \left(\frac {D _ {\mu (\mathcal {K})} ^ {\prime}}{D _ {\mu (\mathcal {K})} ^ {\prime} + 1}\right) \geq \ln \left(\frac {1}{2}\right).
$$
$$
\begin{array}{l} \mathbb {E} [ Q E (\mathcal {A} ^ {\prime}; \mu (\mathcal {K})) ] \geq \ln (D _ {\mu (\mathcal {K})}) + \ln \left(\frac {1}{2}\right) - \frac {1}{2} \\ \geq \min \{| \mathcal {K} |, | \mathcal {K} ^ {c} | \} \cdot \ln (\min _ {n \in [ N ^ {P} ]} D _ {n}) + \ln \left(\frac {1}{2}\right) - \frac {1}{2}. \\ \end{array}
$$
M.8 Proof of Theorem 4
Let  $\mathcal{C}_{N^P}$  be the one-dimensional canonical partitions of  $[N^P]$  (Definition 2). Note that  $\mu(\mathcal{C}_N^P) := \{\mu(\mathcal{K}) : \mathcal{K} \in \mathcal{C}_N^P\} \subseteq \mathcal{C}_{N^P}$ . For an assignment  $(n_{\mathcal{K}})_{\mathcal{K} \in \mathcal{C}_{N^P}} \in \mathbb{N}^{\mathcal{C}_{N^P}}$  of integers to one-dimensional canonical partitions  $\mathcal{K} \in \mathcal{C}_{N^P}$ , we consider the set of tensors whose matricization with respect to each  $\mathcal{K} \in \mathcal{C}_{N^P}$  has rank at most  $n_{\mathcal{K}}$ . This set is also known in the literature as the set of tensors with Hierarchical Tucker (HT) rank at most  $(n_{\mathcal{K}})_{\mathcal{K} \in \mathcal{C}_{N^P}}$  (cf. [25]). Accordingly, we denote it by:
$$
H T \left(\left(n _ {\mathcal {K}}\right) _ {\mathcal {K} \in \mathcal {C} _ {N ^ {P}}}\right) := \left\{\mathcal {V} \in \mathbb {R} ^ {D _ {1} \times \dots \times D _ {N ^ {P}}}: \forall \mathcal {K} \in \mathcal {C} _ {N ^ {P}}, \operatorname {r a n k} (\llbracket \mathcal {V}; \mathcal {K} \rrbracket) \leq n _ {\mathcal {K}} \right\}.
$$
Now, define  $(n_{\mathcal{K}}^{*})_{\mathcal{K}\in \mathcal{C}_{NP}}\in \mathbb{N}^{\mathcal{C}_{NP}}$  by:
$$
\forall \mathcal {K} \in \mathcal {C} _ {N ^ {P}}: n _ {\mathcal {K}} ^ {*} = \left\{ \begin{array}{l l} R & \text {i f} \mu^ {- 1} (\mathcal {K}) \in \mathcal {C} _ {N} ^ {P} \\ \mathcal {D} _ {\mathcal {K}} & \text {i f} \mu^ {- 1} (\mathcal {K}) \notin \mathcal {C} _ {N} ^ {P} \end{array} \right.,
$$
where  $D_{\mathcal{K}} \coloneqq \min \{\prod_{n \in \mathcal{K}} D_n, \prod_{n \in \mathcal{K}^c} D_n\}$ . We show that for any tensor  $\mathcal{A}$  that satisfies for all  $\mathcal{K} \in \mathcal{C}_N^P$ :
$$
Q E (\mathcal {A}; \mu (\mathcal {K})) \leq \frac {\epsilon^ {2}}{(2 N ^ {P} - 3) \| \mathcal {A} \| ^ {2}} \cdot \ln (R),
$$
there exists a tensor  $\mathcal{V} \in HT\left((n_{\mathcal{K}}^{*})_{\mathcal{K} \in \mathcal{C}_{N^{P}}}\right)$  such that  $\| \mathcal{A} - \mathcal{V} \| \leq \epsilon$ . Consider, for each canonical partition  $(\mathcal{K}, \mathcal{K}^{c}) \in \mathcal{C}_{N}^{P}$ , the distribution
where  $\sigma_{\mathcal{K},1} \geq \sigma_{\mathcal{K},2} \geq \ldots \geq \sigma_{\mathcal{K},D_{\mathcal{K}}}$  are the singular values of  $[\mathcal{A};\mu (\mathcal{K})]$  (note that  $\frac{1}{\|\mathcal{A}\|^2}\sum_j\sigma_{\mathcal{K},j}^2 = \frac{\|\mathcal{A}\|^2}{\|\mathcal{A}\|^2} = 1$  so  $\mathcal{P}_{\mathcal{K}}$  is indeed a probability distribution). Denoting by  $H(\mathcal{P}_{\mathcal{K}}) \coloneqq \mathbb{E}_{i\sim \mathcal{P}_{\mathcal{K}}}[\ln (1 / p_{\mathcal{K}}(i))]$  the entropy of  $\mathcal{P}_{\mathcal{K}}$ , by assumption:
$$
Q E (\mathcal {A}; \mu (\mathcal {K})) = H (\mathcal {P} _ {\mathcal {K}}) \leq \frac {\epsilon^ {2}}{\| \mathcal {A} \| ^ {2} (2 N ^ {P} - 3)} \ln (R),
$$
for all  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N^P$ . Thus, taking  $a = \frac{\epsilon^2}{\|\mathcal{A}\|^2(2N^P - 3)}$  we get by Lemma 10 that there exists a subset  $T_{\mathcal{K}}\subseteq [D_{\mathcal{K}}]$  such that
$$
\mathcal {P} _ {\mathcal {K}} (T _ {\mathcal {K}} ^ {c}) \leq \frac {\epsilon^ {2}}{(2 N ^ {P} - 3) \| \mathcal {A} \| ^ {2}},
$$
and  $|T_{\mathcal{K}}| \leq e^{\frac{H(\mathcal{P}_{\mathcal{K}})}{a}} = e^{\ln (R)} = R$ . Note that
$$
\sqrt {\sum_ {i = R + 1} ^ {D \kappa} \sigma_ {\mathcal {K} , i} ^ {2}} \leq \frac {\epsilon}{\sqrt {(2 N ^ {P} - 3)}}.
$$
We can now invoke Lemma 7 (note that if  $\mu^{-1}(\mathcal{K}) \notin \mathcal{C}_N^P$ , then the requirements of Lemma 7 are trivially fulfilled with respect to the partition  $(\mathcal{K}, \mathcal{K}^c)$  since  $n_{\mathcal{K}}^* = D_{\mathcal{K}}$ ), which implies that there exists some  $\mathcal{W} \in HT\left((n_{\mathcal{K}}^*)_{\mathcal{K} \in \mathcal{C}_{N^P}}\right)$  satisfying:
$$
\| \mathcal {W} - \mathcal {A} \| \leq \epsilon ,
$$
The proof concludes by establishing that for any tensor  $\mathcal{W} \in HT\left((n_{\mathcal{K}}^{*})_{\mathcal{K} \in \mathcal{C}_{NP}}\right)$ , there exists assignment for the tensors constituting the  $P$ -dimensional locally connected tensor network (defined in Figure 6) such that it generates  $\mathcal{W}$ .
To see why this is the case, note that by Lemma 8 any tensor  $\mathcal{W} \in HT\left((n_{\mathcal{K}}^{*})_{\mathcal{K} \in \mathcal{C}_{N^{P}}}\right)$  can be represented by a (one-dimensional) locally connected tensor network with varying widths  $(n_{\mathcal{K}}^{*})_{\mathcal{K} \in \mathcal{C}_{N^{P}}}$ , i.e., a tensor network conforming to a perfect binary tree graph in which the lengths of inner axes are as follows: an axis corresponding to an edge that connects a node with descendant leaves indexed by  $\mathcal{K}$  to its parent is assigned the length  $n_{\mathcal{K}}^{*}$ . We can obtain an equivalent representation of any such tensor as a  $P$ -dimensional locally connected tensor network (described in Appendix J.1.1) via the following procedure. For each node at level  $l \in \{0, P, 2P, \ldots, (L - 1)P\}$  of the tree (recall  $N = 2^L$  for  $L \in \mathbb{N}$ ), contract it with its descendant nodes at levels  $\{l + 1, \ldots, l + (P - 1)\}$ . This results in a new tensor network whose underlying graph is a perfect  $2^{P}$ -ary tree and the remaining edges all correspond to inner axes of lengths equal to  $n_{\mathcal{K}}^{*} = R$  for  $\mathcal{K} \in \mathcal{C}_N^P$ , i.e., in a representation of  $\mathcal{W}$  as a  $P$ -dimensional locally connected tensor network.
<sup>12</sup>For a concrete example, let  $N = 2^L = 4$  and  $P = 2$  (i.e.  $L = 2$ ). In this case, the perfect binary tree underlying the one-dimensional locally connected tensor network of varying widths is of height  $L \cdot P = 4$  and has  $N^P = 16$  leaves. It is converted into a perfect 4-ary tree tensor network of height  $L = 2$  by contracting the root with its two children and the nodes at level two with their children.