SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our theory suggests that the suitability of a data distribution to locally connected neural networks may be enhanced by arranging features such that low entanglement under canonical partitions is attained. Employing a certain surrogate for entanglement, we show that this arrangement can be implemented efficiently, and that it leads to substantial improvements in the prediction accuracies of common locally connected neural networks on various datasets.
Neural network architecture We theoretically analyzed a locally connected neural network with polynomial non-linearity, by employing its equivalence to a tensor network (cf. Section 3.1). Accounting for neural networks with connectivities beyond those considered (e.g. connectivities that are non-local or ones involving skip connections) is an interesting topic for future work. It requires modification of the equivalent tensor network and corresponding modification of the definition of canonical partitions, similarly to the analysis in Appendix J. Another valuable direction is to account for neural networks with other non-linearities, e.g. ReLU. We believe this may be achieved through a generalized notion of tensor networks, successfully used in past work to analyze such architectures [9].
Objective function The analysis in Section 4 assumes a binary soft-margin SVM objective. Extending it to other objective functions, e.g. multi-class SVM, may shed light on the relation between the objective and the requirements for a data distribution to be suitable to neural networks.
Textual data Our experiments (in Section 4.3 and Appendix J.2.3) show that the necessary and sufficient condition we derived for a data distribution to be suitable to a locally connected neural network — namely, low quantum entanglement under canonical partitions of features — is upheld by audio and image datasets. This falls in line with the excellent performance of locally connected neural networks over these data modalities. In contrast, high performant architectures for textual data are typically non-local [61]. Investigating the quantum entanglements that textual data admits and, in particular, under which partitions they are low, may allow designing more efficient architectures with connectivity tailored to textual data.
6.3 Outlook
This work was supported by a Google Research Scholar Award, a Google Research Gift, the Yandex Initiative in Machine Learning, the Israel Science Foundation (grant 1780/21), the Tel Aviv University Center for AI and Data Science, the Adelis Research Fund for Artificial Intelligence, Len Blavatnik and the Blavatnik Family Foundation, and Amnon and Anat Shashua. NR is supported by the Apple Scholars in AI/ML PhD fellowship.
[64] Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209, 2018.
[65] Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:103-114, 2017.
[66] Ya-Hui Zhang. Entanglement entropy of target functions for image classification and convolutional neural network. arXiv preprint arXiv:1710.05520, 2017.
[67] Yitan Zhu, Thomas Brettin, Fangfang Xia, Alexander Partin, Maulik Shukla, Hyunseung Yoo, Yvonne A Evrard, James H Doroshow, and Rick L Stevens. Converting tabular data into images for deep learning with convolutional neural networks. Scientific reports, 11(1):1-11, 2021.
Characterizing formal properties of data distributions that make them suitable for deep learning is a major open problem in the field. A number of papers provide sufficient conditions on a data distribution which imply that it is learnable by certain neural networks [5, 65, 39, 19, 20, 43, 62, 6]. However, these sufficient conditions are restrictive, and are not argued to be necessary for any aspect of learning (i.e. for expressiveness, optimization or generalization). To the best of our knowledge, this paper is the first to derive a verifiable condition on a data distribution that is both necessary and sufficient for aspects of learning to be achievable by a neural network. We note that the condition we derive resembles past attempts to quantify the structure of data via quantum entanglement and mutual information [40, 15, 38, 7, 66, 28, 21]. However, such quantifications have not been formally related to learnability by neural networks.
The current paper follows a long line of research employing tensor networks as theoretical models for studying deep learning. This line includes works analyzing the expressiveness of different neural network architectures [12, 55, 9, 10, 54, 14, 34, 3, 35, 30, 36, 31, 50], their generalization [37], and the implicit regularization induced by their optimization [47, 48, 49, 63, 24]. Similarly to prior works we focus on expressiveness, yet our approach differs in that we incorporate the data distribution into the analysis and tackle the question of what makes data suitable for deep learning.
The algorithm we propose for enhancing the suitability of data to locally connected neural networks can be considered a form of representation learning. Representation learning is a vast field, far too broad to survey here (for an overview see [41]). Our algorithm, which learns a representation via rearrangement of features in the data, is complementary to most representation learning methods in the literature. A notable method that is also based on feature rearrangement is IGTD [67] — a heuristic scheme designed for convolutional neural networks. In contrast to IGTD, our algorithm is theoretically grounded. Moreover, we demonstrate empirically in Section 5 that it leads to higher prediction accuracies.
B Illustration of Canonical Partitions
Figure 4: The canonical partitions of  $[N]$ , for  $N = 2^L$  with  $L \in \mathbb{N}$ . Every  $l \in \{0, \dots, L\}$  contributes  $2^l$  canonical partitions, the  $n$ 'th one induced by  $\mathcal{K} = \{2^{L - l} \cdot (n - 1) + 1, \dots, 2^{L - l} \cdot n\}$ .
C Impossibility Result for Improving the Sufficient Condition in Theorem 2
The sufficient condition in Theorem 2 (from Section 3.2) for approximating  $\mathcal{A} \in \mathbb{R}^{D_1 \times \dots \times D_N}$  requires entanglements to approach zero as the desired approximation error  $\epsilon$  does, in contrast to the necessary condition in Theorem 1 where they approach  $\ln(R)$ . As Proposition 1 shows, this is unavoidable in the absence of further knowledge regarding  $\mathcal{A}$ . However, if for all canonical partitions  $(\mathcal{K}, \mathcal{K}^c) \in \mathcal{C}_N$  the singular values of  $[\mathcal{A}; \mathcal{K}]$  trailing after the  $R$ 'th one are small, then we can also guarantee an assignment for the locally connected tensor network satisfying  $\| \mathcal{W}_{\mathrm{TN}} - \mathcal{A} \| \leq \epsilon$ , while  $QE(\mathcal{A}; \mathcal{K})$  can be on the order of  $\ln(R)$  for all  $(\mathcal{K}, \mathcal{K}^c) \in \mathcal{C}_N$ . Indeed, this follows directly from a result in [25], which we restate as Lemma 7 for convenience.
Proposition 1. Let  $f: \mathbb{R}^2 \to \mathbb{R}_{\geq 0}$  be monotonically increasing in its second variable. Suppose that the following statement holds: if a tensor  $\mathcal{A} \in \mathbb{R}^{D_1 \times \dots \times D_N}$  satisfies  $QE(\mathcal{A}; \mathcal{K}) \leq f(\| \mathcal{A} \|, \epsilon)$  for all canonical partitions  $(\mathcal{K}, \mathcal{K}^c) \in \mathcal{C}_N$  (Definition 2), then there exists an assignment for the tensors constituting the locally connected tensor network (defined in Section 3.1) for which  $\mathcal{W}_{\mathrm{TN}} \in$
$\mathbb{R}^{D_1\times \dots \times D_N}$  upholds:
$$
\| \mathcal {W} _ {\mathrm {T N}} - \mathcal {A} \| \leq \epsilon .
$$
Then, for any  $\mathcal{A} \in \mathbb{R}^{D_1 \times \dots \times D_N}$ , as the desired approximation error  $\epsilon$  goes to zero, so does the sufficient condition on entanglements, i.e.:
$$
\lim  _ {\epsilon \rightarrow 0} f (\| \mathcal {A} \|, \epsilon) = 0.
$$
Proof. Suppose otherwise, i.e. that there exists some  $a > 0$  such that  $\lim_{\epsilon \to 0}f(a,\epsilon) = \inf_{\epsilon >0}f(a,\epsilon) = c > 0$ . Let  $\mathcal{A}$  be a tensor with  $\| \mathcal{A}\| = a$  such that  $QE(A;\mathcal{K}) < c$  for all canonical partitions  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N$ . Then by assumption, for all  $\epsilon >0$  there exist a tensor generated by the locally connected tensor network, which we denote by  $\mathcal{W}_{\mathrm{TN}}(\epsilon)\in \mathbb{R}^{D_1\times \dots \times D_N}$ , that satisfies:
$$
\| \mathcal {W} _ {\mathrm {T N}} (\epsilon) - \mathcal {A} \| \leq \epsilon .
$$
By Lemma 5, for all  $\epsilon >0$  we have that
$$
\operatorname {r a n k} \left(\llbracket \mathcal {W} _ {\mathrm {T N}} (\epsilon); \mathcal {K} \rrbracket\right) \leq R,
$$
so by the lower semicontinuity of the matrix rank, we have that  $\mathrm{rank}([\mathcal{A};\mathcal{K}])\leq R$  as well. But, there are tensors for which this leads to a contradiction, namely tensors with arbitrary low entanglement across all partitions but nearly maximal rank. Indeed, consider tensors of the form
$$
\mathcal {Q} (\delta) = a \cdot \frac {\otimes_ {n = 1} ^ {N} \mathbf {v} ^ {(n)} + \delta \cdot \mathcal {B}}{\| \otimes_ {n = 1} ^ {N} \mathbf {v} ^ {(n)} + \delta \cdot \mathcal {B} \|} \in \mathbb {R} ^ {D _ {1} \times \dots \times D _ {N}},
$$
where  $\{\mathbf{v}^{(n)}\in \mathbb{R}^{D_n}\}_{n = 1}^N$  are (non-zero) vectors,  $\delta >0$  and  $\mathcal{B}\in \mathbb{R}^{D_1\times \dots \times D_N}$  is some tensor with matricizations of maximal rank across all canonical partitions (for a proof of the existence of such a tensor see Claim 3 in [35]). Note that  $\| \mathcal{Q}(\delta)\| = a$  for all  $\delta >0$ . By the triangle inequality have
$$
\left\| \frac {\mathcal {Q} (\delta)}{a} - \frac {\otimes_ {n = 1} ^ {N} \mathbf {v} ^ {(n)}}{\| \otimes_ {n = 1} ^ {N} \mathbf {v} ^ {(n)} \|} \right\| \leq \frac {\delta \cdot \| \mathcal {B} \|}{\| \otimes_ {n = 1} ^ {N} \mathbf {v} ^ {(n)} + \delta \cdot \mathcal {B} \|} + \left\| \frac {\otimes_ {n = 1} ^ {N} \mathbf {v} ^ {(n)}}{\| \otimes_ {n = 1} ^ {N} \mathbf {v} ^ {(n)} + \delta \cdot \mathcal {B} \|} - \frac {\otimes_ {n = 1} ^ {N} \mathbf {v} ^ {(n)}}{\| \otimes_ {n = 1} ^ {N} \mathbf {v} ^ {(n)} \|} \right\|,
$$
$$
\lim  _ {\delta \rightarrow 0} \left\| \frac {\mathcal {Q} (\delta)}{a} - \frac {\otimes_ {n = 1} ^ {N} \mathbf {v} ^ {(n)}}{\| \otimes_ {n = 1} ^ {N} \mathbf {v} ^ {(n)} \|} \right\| = 0.
$$
Thus, from Lemma 9 we know that
$$
\lim  _ {\delta \rightarrow 0} \left| Q E \left(\frac {Q (\delta)}{a}; \mathcal {K}\right) - Q E \left(\frac {\otimes_ {n = 1} ^ {N} \mathbf {v} ^ {(n)}}{\| \otimes_ {n = 1} ^ {N} \mathbf {v} ^ {(n)} \|}; \mathcal {K}\right)\right| = 0
$$
for all  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N$  . Furthermore, for any  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N$  ..
$$
Q E \left(\frac {\otimes_ {n = 1} ^ {N} \mathbf {v} ^ {(n)}}{\| \otimes_ {n = 1} ^ {N} \mathbf {v} ^ {(n)} \|}; \mathcal {K}\right) = 0,
$$
and therefore by Theorem 1 for sufficiently small  $\delta >0$  we have
$$
Q E (\mathcal {Q} (\delta); \mathcal {K}) <   c,
$$
for all  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N$ . However,  $\mathrm{rank}([\mathcal{Q}(\delta);\mathcal{K}])$  is (nearly) maximal for all canonical partitions. Indeed, by the triangle inequality for matrix rank we get
$$
\operatorname {r a n k} \left(\llbracket \mathcal {Q} (\delta); \mathcal {K} \rrbracket\right) \geq \operatorname {r a n k} \left(\llbracket \mathcal {B}; \mathcal {K} \rrbracket\right) - \operatorname {r a n k} \left(\llbracket \otimes_ {n = 1} ^ {N} \mathbf {v} ^ {(n)}; \mathcal {K} \rrbracket\right) = D _ {\mathcal {K}} - 1,
$$
where  $D_{\mathcal{K}} \coloneqq \min \{\prod_{n \in \mathcal{K}} D_n, \prod_{n \in \mathcal{K}^c} D_n\}$ .
D Quantum Entanglement as a Measure of Dependence Between Features
In this appendix, we provide intuition behind the following statement made in Section 4.2: the quantum entanglement of  $\mathcal{D}_{\mathrm{pop}}$  with respect to an arbitrary partition of its axes  $(\mathcal{K},\mathcal{K}^c)$ , where  $\mathcal{K}\subseteq [N]$ , is a measure of dependence between the data features indexed by  $\mathcal{K}$  and those indexed by  $\mathcal{K}^c$ . Consider the case where the features indexed by  $\mathcal{K}$  are statistically independent of those indexed by  $\mathcal{K}^c$ , and all features are statistically independent of the label  $y$ . Then  $QE(\mathcal{D}_{\mathrm{pop}};\mathcal{K}) = 0$  (see Lemma 1 below for proof of this fact). If the features are not statistically independent of  $y$ , but when conditioned on  $y$  the features indexed by  $\mathcal{K}$  are statistically independent of those indexed by  $\mathcal{K}^c$ , then  $QE(\mathcal{D}_{\mathrm{pop}};\mathcal{K})\leq \ln (2)$  (this fact is also established by Lemma 1 below). In general, the higher  $QE(\mathcal{D}_{\mathrm{pop}};\mathcal{K})$  is, the farther the distribution is from the aforementioned situations of independence, implying stronger dependence between the features indexed by  $\mathcal{K}$  and those indexed by  $\mathcal{K}^c$ .
Lemma 1. Consider the classification setting of Section 4.1, and let  $\mathcal{K} \subseteq [N]$ . If the features indexed by  $\mathcal{K}$  are independent of those indexed by  $\mathcal{K}^c$  (i.e.  $(\mathbf{x}^{(n)})_{n \in \mathcal{K}}$  are independent of  $(\mathbf{x}^{(n)})_{n \in \mathcal{K}^c}$ ), conditioned on the label  $y$ , then  $QE(\mathcal{D}_{\mathrm{pop}}; \mathcal{K}) \leq \ln(2)$ . Furthermore, if the features are also independent of  $y$ , then  $QE(\mathcal{D}_{\mathrm{pop}}; \mathcal{K}) = 0$ .
Proof. For brevity, denote  $\mathbf{X} \coloneqq (\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(N)})$ ,  $\mathbf{X}_{\mathcal{K}} \coloneqq (\mathbf{x}^{(n)})_{n \in \mathcal{K}}$ , and  $\mathbf{X}_{\mathcal{K}^c} \coloneqq (\mathbf{x}^{(n)})_{n \in \mathcal{K}^c}$ . By the law of total expectation, the entry of  $\mathcal{D}_{\mathrm{pop}}$  corresponding to an index tuple  $(i_1, \ldots, i_N) \in [D_1] \times \dots \times [D_N]$  satisfies:
$$
\begin{array}{l} \left(\mathcal {D} _ {\mathrm {p o p}}\right) _ {i _ {1}, \dots , i _ {N}} = \mathbb {E} _ {\mathbf {X}, y} \left[ y \cdot \prod_ {n = 1} ^ {N} \mathbf {x} _ {i _ {n}} ^ {(n)} \right] \\ = \mathbb {P} (y = 1) \cdot \mathbb {E} _ {\mathbf {X}} \left[ \prod_ {n = 1} ^ {N} \mathbf {x} _ {i _ {n}} ^ {(n)} | y = 1 \right] - \mathbb {P} (y = - 1) \cdot \mathbb {E} _ {\mathbf {X}} \left[ \prod_ {n = 1} ^ {N} \mathbf {x} _ {i _ {n}} ^ {(n)} | y = - 1 \right]. \\ \end{array}
$$
Thus, if the features indexed by  $\mathcal{K}$  are independent of those indexed by  $\mathcal{K}^c$  given  $y$ , it holds that:
$$
\begin{array}{l} \left(\mathcal {D} _ {\text {p o p}}\right) _ {i _ {1}, \dots , i _ {N}} = \mathbb {P} (y = 1) \cdot \mathbb {E} _ {\mathbf {X} _ {\mathcal {K}}} \left[ \prod_ {n \in \mathcal {K}} \mathbf {x} _ {i _ {n}} ^ {(n)} | y = 1 \right] \cdot \mathbb {E} _ {\mathbf {X} _ {\mathcal {K} ^ {c}}} \left[ \prod_ {n \in \mathcal {K} ^ {c}} \mathbf {x} _ {i _ {n}} ^ {(n)} | y = 1 \right] \\ - \mathbb {P} (y = - 1) \cdot \mathbb {E} _ {\mathbf {X} _ {\mathcal {K}}} \left[ \prod_ {n \in \mathcal {K}} \mathbf {x} _ {i _ {n}} ^ {(n)} | y = - 1 \right] \cdot \mathbb {E} _ {\mathbf {X} _ {\mathcal {K} ^ {c}}} \left[ \prod_ {n \in \mathcal {K} ^ {c}} \mathbf {x} _ {i _ {n}} ^ {(n)} | y = - 1 \right]. \tag {4} \\ \end{array}
$$
This implies that we can write  $\llbracket \mathcal{D}_{\mathrm{pop}};\mathcal{K}\rrbracket \in \mathbb{R}^{\prod_{n\in \mathcal{K}}D_n\times \prod_{n\in \mathcal{K}^c}D_n}$  — the arrangement of  $\mathcal{D}_{\mathrm{pop}}$  as a matrix whose rows correspond to axes indexed by  $\mathcal{K}$  and columns correspond to the remaining axes — as a weighted sum of two outer products. Specifically, define  $\mathbf{v}_{\mathcal{K}},\mathbf{u}_{\mathcal{K}}\in \mathbb{R}^{\prod_{n\in \mathcal{K}}D_n}$  to be the vectors holding for each possible indexing tuple  $(i_n)_{n\in \mathcal{K}}$  the values  $\mathbb{E}_{\mathbf{X}_{\mathcal{K}}}\big[\prod_{n\in \mathcal{K}}\mathbf{x}_{i_n}^{(n)}|y = 1\big]$  and  $\mathbb{E}_{\mathbf{X}_{\mathcal{K}}}\big[\prod_{n\in \mathcal{K}}\mathbf{x}_{i_n}^{(n)}|y = -1\big]$ , respectively (with the arrangement of entries in the vectors being consistent with the rows of  $\llbracket \mathcal{D}_{\mathrm{pop}};\mathcal{K}\rrbracket$ ). Furthermore, let  $\mathbf{v}_{\mathcal{K}^c},\mathbf{u}_{\mathcal{K}^c}\in \mathbb{R}^{\prod_{n\in \mathcal{K}^c}D_n}$  be defined analogously by replacing  $\mathcal{K}$  with  $\mathcal{K}^c$  (with the arrangement of entries in the vectors being consistent with the columns of  $\llbracket \mathcal{D}_{\mathrm{pop}};\mathcal{K}\rrbracket$ ). Then, by Equation (4):
$$
\llbracket \mathcal {D} _ {\text {p o p}}; \mathcal {K} \rrbracket = \mathbb {P} (y = 1) \cdot \mathbf {v} _ {\mathcal {K}} \otimes \mathbf {v} _ {\mathcal {K} ^ {c}} - \mathbb {P} (y = - 1) \cdot \mathbf {u} _ {\mathcal {K}} \otimes \mathbf {u} _ {\mathcal {K} ^ {c}}.
$$
Since each outer product forms a rank one matrix, the subadditivity of rank implies that the rank of  $\llbracket \mathcal{D}_{\mathrm{pop}};\mathcal{K}\rrbracket$  is at most two. As a result,  $\llbracket \mathcal{D}_{\mathrm{pop}};\mathcal{K}\rrbracket$  has at most two non-zero singular values and  $QE(\mathcal{D}_{\mathrm{pop}};\mathcal{K})\leq \ln (2)$  (the entropy of a distribution is at most the natural logarithm of the size of its support).
If, in addition, all features in the data are independent of the label  $y$ , then  $\mathbf{v}_{\mathcal{K}} = \mathbf{u}_{\mathcal{K}}$  and  $\mathbf{v}_{\mathcal{K}^c} = \mathbf{u}_{\mathcal{K}^c}$  meaning  $\llbracket \mathcal{D}_{\mathrm{pop}};\mathcal{K}\rrbracket = (\mathbb{P}(y = 1) - \mathbb{P}(y = -1))\cdot \mathbf{v}_{\mathcal{K}}\otimes \mathbf{v}_{\mathcal{K}^c}$ . In this case, the rank of  $\llbracket \mathcal{D}_{\mathrm{pop}};\mathcal{K}\rrbracket$  is at most one, so  $QE(\mathcal{D}_{\mathrm{pop}};\mathcal{K}) = 0$ .
E Efficiently Computable Criterion for Low Suboptimality in Achievable Accuracy
In this appendix, we provide the formal claims deferred from Section 4.2. Specifically, Proposition 2 shows that  $\mathcal{D}_{\mathrm{pop}} / \| \mathcal{D}_{\mathrm{pop}}\|$  is, with high probability, well-approximated by the normalized empirical data tensor  $\mathcal{D}_{\mathrm{emp}} / \| \mathcal{D}_{\mathrm{emp}}\|$ . Corollary 2 establishes the efficiently computable criterion (low entanglement under canonical partitions for  $\mathcal{D}_{\mathrm{emp}}$ ), that with high probability is both necessary and sufficient for low suboptimality in achievable accuracy.
Proposition 2. Consider the classification setting of Section 4.1, and let  $\delta \in (0,1)$  and  $\gamma > 0$ . If the training set size  $M$  satisfies  $M \geq \frac{2\ln\left(\frac{2}{\delta}\right)}{\|\mathcal{D}_{\mathrm{pop}}\|^2\gamma^2}$ , then with probability at least  $1 - \delta$ :
$$
\left\| \frac {\mathcal {D} _ {\mathrm {p o p}}}{\| \mathcal {D} _ {\mathrm {p o p}} \|} - \frac {\mathcal {D} _ {\mathrm {e m p}}}{\| \mathcal {D} _ {\mathrm {e m p}} \|} \right\| \leq \gamma
$$
Proof sketch (proof in Appendix M.5). A standard generalization of the Hoeffding inequality to random vectors in a Hilbert space yields a high probability bound on  $\| \mathcal{D}_{\mathrm{emp}} - \mathcal{D}_{\mathrm{pop}} \|$ , which is then translated to a bound on the normalized tensors.
Algorithm 1 Entanglement Computation for the Empirical Data Tensor
1: Input:  $\mathcal{X} := \{(\mathbf{x}^{(1,m)},\dots,\mathbf{x}^{(N,m)})\}_{m=1}^{M} - M \in \mathbb{N}$  data instances comprising  $N \in \mathbb{N}$  features each,  $\mathcal{K} \subseteq [N]$  — subset of feature indices
2: Output:  $QE(\mathcal{D}_{\mathrm{emp}}; \mathcal{K})$
3: Compute  $\mathbf{G}^{(\mathcal{K})},\mathbf{G}^{(\mathcal{K}^c)}\in \mathbb{R}^{M\times M}$  given element-wise by:
$$
\forall i, j \in [ M ]: \mathbf {G} _ {i, j} ^ {(\mathcal {K})} = y ^ {(i)} y ^ {(j)} \cdot \left\langle \otimes_ {n \in \mathcal {K}} \mathbf {x} ^ {(n, i)}, \otimes_ {n \in \mathcal {K}} \mathbf {x} ^ {(n, j)} \right\rangle = y ^ {(i)} y ^ {(j)} \cdot \prod_ {n \in \mathcal {K}} \left\langle \mathbf {x} ^ {(n, i)}, \mathbf {x} ^ {(n, j)} \right\rangle
$$
$$
\forall i, j \in [ M ]: \mathbf {G} _ {i, j} ^ {(\mathcal {K} ^ {c})} = \left\langle \otimes_ {n \in \mathcal {K} ^ {c}} \mathbf {x} ^ {(n, i)}, \otimes_ {n \in \mathcal {K} ^ {c}} \mathbf {x} ^ {(n, j)} \right\rangle = \prod_ {n \in \mathcal {K} ^ {c}} \left\langle \mathbf {x} ^ {(n, i)}, \mathbf {x} ^ {(n, j)} \right\rangle
$$
4: Compute eigenvalue decompositions of  $\mathbf{G}^{(\mathcal{K})}$  and  $\mathbf{G}^{(K^c)}$ , i.e.:
$$
\mathbf {G} ^ {(\mathcal {K})} = \mathbf {U} ^ {(\mathcal {K})} \mathbf {S} ^ {(\mathcal {K})} \left(\mathbf {U} ^ {(\mathcal {K})}\right) ^ {\top}
$$
$$
\mathbf {G} ^ {(\mathcal {K} ^ {c})} = \mathbf {U} ^ {(\mathcal {K} ^ {c})} \mathbf {S} ^ {(\mathcal {K} ^ {c})} \left(\mathbf {U} ^ {(\mathcal {K} ^ {c})}\right) ^ {\top}
$$
where  $\mathbf{U}^{(\mathcal{K})},\mathbf{U}^{(\mathcal{K}^c)}\in \mathbb{R}^{M\times M}$  are orthogonal matrices and  $\mathbf{S}^{(\mathcal{K})},\mathbf{S}^{(\mathcal{K}^c)}\in \mathbb{R}^{M\times M}$  are diagonal holding the eigenvalues of  $\mathbf{G}^{(\mathcal{K})}$  and  $\mathbf{G}^{(\mathcal{K}^c)}$ , respectively
5: Compute  $\mathbf{Q} = \left(\mathbf{S}^{(\mathcal{K})}\right)^{\frac{1}{2}}\left(\mathbf{U}^{(\mathcal{K})}\right)^{\top}\mathbf{U}^{(\mathcal{K}^c)}\left(\mathbf{S}^{(\mathcal{K}^c)}\right)^{\frac{1}{2}}\in \mathbb{R}^{M\times M}$
6: Compute a singular value decomposition of  $\mathbf{Q}$  to obtain its singular values  $\sigma_{1}(\mathbf{Q}),\dots,\sigma_{M}(\mathbf{Q})$
7: Let  $\rho_{m} := \sigma_{m}^{2}(\mathbf{Q}) / \sum_{m' = 1}^{M}\sigma_{m'}^{2}(\mathbf{Q})$  for  $m\in [M]$
8: return  $QE(\mathcal{D}_{\mathrm{emp}}; \mathcal{K}) = -\sum_{m=1}^{M} \rho_m \ln(\rho_m)$  (if  $\mathbf{Q} = 0$ , then return 0)
Corollary 2. Consider the setting and notation of Corollary 1, with  $\epsilon \in (0,1 / 6]$ . For  $\delta \in (0,1)$  suppose that the training set size  $M$  satisfies  $M\geq \frac{8\ln(\frac{2}{\delta})}{\|\mathcal{D}_{\mathrm{pop}}\|^2\epsilon^2}$ . Then, with probability at least  $1 - \delta$  the following hold. First, if there exists a canonical partition  $(\mathcal{K},\mathcal{K}^{c})\in \mathcal{C}_{N}$  (Definition 2) under which  $QE(\mathcal{D}_{\mathrm{emp}};\mathcal{K}) > \ln (R) + 3\epsilon \cdot \ln (D_{\mathcal{K}}) + 2\sqrt{3}\epsilon$ , then:
$$
\operatorname {S u b O p t} > \epsilon .
$$
Second, if for all  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N$  it holds that  $QE(\mathcal{D}_{\mathrm{emp}};\mathcal{K})\leq \frac{\epsilon^2}{32N - 48}\cdot \ln (R)$ , then:
$$
\operatorname {S u b O p t} \leq \epsilon .
$$
Moreover, the conditions above on the entanglements of  $\mathcal{D}_{\mathrm{emp}}$  can be evaluated efficiently (in  $\mathcal{O}(DNM^2 +NM^3)$  time  $\mathcal{O}(DNM + M^2)$  and memory, where  $D\coloneqq \max_{n\in [N]}D_n$ ).
Proof sketch (proof in Appendix M.6). Implied by Corollary 1, Proposition 2 with  $\gamma = \frac{\epsilon}{2}$  and Algorithm 1 in Appendix F.
F Efficiently Computing Entanglements of the Empirical Data Tensor