SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For a given tensor, its entanglement with respect to a partition of axes (Definition 1) is determined by the singular values of its arrangement as a matrix according to the partition. Since the empirical data tensor  $\mathcal{D}_{\mathrm{emp}}$  (Equation (2)) has size exponential in the number of features  $N$ , it is infeasible to naively compute its entanglement (or even explicitly store it in memory). Fortunately, as shown in [40], the specific form of the empirical data tensor admits an efficient algorithm for computing entanglements, without explicitly manipulating an exponentially large matrix. Specifically, the algorithm runs in  $\mathcal{O}(DNM^2 + M^3)$  time and requires  $\mathcal{O}(DNM + M^2)$  memory, where  $D := \max_{n \in [N]} D_n$  is the maximal feature dimension (i.e. axis length of  $\mathcal{D}_{\mathrm{emp}}$ ),  $N$  is the number of features in the data (i.e. number of axes that  $\mathcal{D}_{\mathrm{emp}}$  has), and  $M$  is the number of training instances. For completeness, we outline the method in Algorithm 1 while referring the interested reader to Appendix A in [40] for further details.
G Practical Algorithm via Surrogate for Entanglement
To efficiently implement the scheme from Section 5.1, we replace entanglement with a surrogate measure of dependence. The surrogate is based on the Pearson correlation coefficient for multivariate features [45], and its agreement with entanglement is demonstrated empirically in Appendix I. Theoretically supporting this agreement is left for future work.
Definition 4. Given a set of  $M \in \mathbb{N}$  instances  $\mathcal{X} \coloneqq \{(\mathbf{x}^{(1,m)},\dots,\mathbf{x}^{(N,m)}) \in (\mathbb{R}^D)^N\}_{m=1}^M$ , denote by  $p_{n,n'}$  the multivariate Pearson correlation between features  $n, n' \in [N]$ . For  $\mathcal{K} \subseteq [N]$ , the surrogate entanglement of  $\mathcal{X}$  with respect to the partition  $(\mathcal{K}, \mathcal{K}^c)$ , denoted  $SE(\mathcal{X};\mathcal{K})$ , is the sum of absolute values of Pearson correlation coefficients between pairs of features, the first belonging to  $\mathcal{K}$  and the second to  $\mathcal{K}^c \coloneqq [N] \setminus \mathcal{K}$ . That is,  $SE(\mathcal{X};\mathcal{K}) \coloneqq \sum_{n \in \mathcal{K}, n' \in \mathcal{K}^c} |p_{n,n'}|$ .
As shown by Proposition 3 below, replacing entanglement with surrogate entanglement in the scheme from Section 5.1 converts each search for a balanced partition minimizing average entanglement into a minimum balanced cut problem. Although the minimum balanced cut problem is NP-hard (see, e.g., [23]), it enjoys a wide array of well-established approximation tools, particularly ones designed for large scale [29, 57]. We therefore obtain a practical algorithm for enhancing the suitability of a data distribution to locally connected neural networks — see Algorithm 2.
Proposition 3. For any  $\bar{\mathcal{K}}\subseteq [N]$  of even size, the following optimization problem can be framed as a minimum balanced cut problem over a complete graph with  $|\bar{\mathcal{K}} |$  vertices:
$$
\min  _ {\mathcal {K} \subset \bar {\mathcal {K}}, | \mathcal {K} | = | \bar {\mathcal {K}} | / 2} \frac {1}{2} \left[ S E (\mathcal {X}; \mathcal {K}) + S E (\mathcal {X}; \bar {\mathcal {K}} \backslash \mathcal {K}) \right]. \tag {5}
$$
Specifically, there exists a complete undirected weighted graph with vertices  $\bar{\mathcal{K}}$  and edge weights  $w:\bar{\mathcal{K}}\times \bar{\mathcal{K}}\to \mathbb{R}$  such that for any  $\mathcal{K}\subset \bar{\mathcal{K}}$ , the weight of the cut in the graph induced by  $\mathcal{K}$  —  $\sum_{n\in \mathcal{K},n'\in \bar{\mathcal{K}}\setminus \mathcal{K}}w(\{n,n'\})$  — is equal, up to an additive constant, to the term minimized in Equation (5), i.e. to  $\frac{1}{2} [SE(\mathcal{X};\mathcal{K}) + SE(\mathcal{X};\bar{\mathcal{K}}\setminus \mathcal{K})]$ .
Proof. Consider the complete undirected graph whose vertices are  $\bar{\mathcal{K}}$  and where the weight of an edge  $\{n, n'\} \in \bar{\mathcal{K}} \times \bar{\mathcal{K}}$  is  $w(\{n, n'\}) = |p_{n,n'}|$  (recall that  $p_{n,n'}$  stands for the multivariate Pearson correlation between features  $n$  and  $n'$  in  $\mathcal{X}$ ). For any  $\mathcal{K} \subset \bar{\mathcal{K}}$  it holds that:
$$
\sum_ {n \in \mathcal {K}, n ^ {\prime} \in \bar {\mathcal {K}} \backslash \mathcal {K}} w (\{n, n ^ {\prime} \}) = \frac {1}{2} \left[ S E (\mathcal {X}; \mathcal {K}) + S E (\mathcal {X}; \bar {\mathcal {K}} \backslash \mathcal {K}) \right] - \frac {1}{2} S E (\mathcal {X}; \bar {\mathcal {K}}),
$$
where  $\frac{1}{2} SE\big(\mathcal{X};\bar{\mathcal{K}}\big)$  does not depend on  $\kappa$ . This concludes the proof.
H Definition of the Multivariate Pearson Correlation from [45]
Appendix G introduces a surrogate measure for entanglement based on the multivariate Pearson correlation from [45]. For completeness, this appendix provides its formal definition.
Given a set of  $M \in \mathbb{N}$  instances  $\mathcal{X} := \{(\mathbf{x}^{(1,m)},\dots,\mathbf{x}^{(N,m)}) \in (\mathbb{R}^D)^N\}_{m=1}^M$ , let  $\Sigma^{(n)}$  be the empirical covariance matrix of feature  $n \in [N]$  and  $\Sigma^{(n,n')}$  be the empirical cross-covariance matrix of features  $n, n' \in [N]$ , i.e.:
$$
\Sigma^ {(n)} := \frac {1}{M} \sum_ {m - 1} ^ {M} \left(\mathbf {x} ^ {(n, m)} - \boldsymbol {\mu} ^ {(n)}\right) \otimes \left(\mathbf {x} ^ {(n, m)} - \boldsymbol {\mu} ^ {(n)}\right),
$$
$$
\Sigma^ {(n, n ^ {\prime})} := \frac {1}{M} \sum_ {m - 1} ^ {M} \left(\mathbf {x} ^ {(n, m)} - \boldsymbol {\mu} ^ {(n)}\right) \otimes \left(\mathbf {x} ^ {(n ^ {\prime}, m)} - \boldsymbol {\mu} ^ {(n ^ {\prime})}\right),
$$
where  $\mu^{(n)}\coloneqq \frac{1}{M}\sum_{m = 1}^{M}\mathbf{x}^{(n,m)}$  for  $n\in [N]$ . With this notation, the multivariate Pearson correlation of  $n,n^{\prime}\in [N]$  from [45] is defined by  $p_{n,n^{\prime}}\coloneqq \mathrm{trace}\bigl (\Sigma^{(n,n^{\prime})}\bigr) / \mathrm{trace}\bigl ((\Sigma^{(n)}\Sigma^{(n^{\prime})})^{1 / 2}\bigr)$ .
For completeness, Appendix H provides a formal definition of the multivariate Pearson correlation.
Algorithm 2 Enhancing Suitability of Data to Locally Connected Neural Networks
1: Input:  $\mathcal{X} := \{(\mathbf{x}^{(1,m)},\ldots ,\mathbf{x}^{(N,m)})\}_{m = 1}^{M}$  —  $M\in \mathbb{N}$  data instances comprising  $N\in \mathbb{N}$  features
2: Output: Permutation  $\pi : [N] \to [N]$  to apply to feature indices
3: Let  $\mathcal{K}_{0,1} \coloneqq [N]$  and denote  $L \coloneqq \log_2(N)$
4: # We assume for simplicity that  $N$  is a power of two, otherwise one may add constant features
5: for  $l = 0, \dots, L - 1$ ,  $n = 1, \dots, 2^l$  do
6: Using a reduction to a minimum balanced cut problem (Proposition 3), find an approximate solution  $\mathcal{K}_{l + 1,2n - 1} \subset \mathcal{K}_{l,n}$  for:
$$
\min  _ {\mathcal {K} \subset \mathcal {K} _ {l, n}, | \mathcal {K} | = | \mathcal {K} _ {l, n} | / 2} \frac {1}{2} [ S E (\mathcal {X}; \mathcal {K}) + S E (\mathcal {X}; \mathcal {K} _ {l, n} \setminus \mathcal {K}) ]
$$
7: Let  $\mathcal{K}_{l+1,2n} := \mathcal{K}_{l,n} \setminus \mathcal{K}_{l+1,2n-1}$
8: end for
9: # At this point,  $\mathcal{K}_{L,1},\ldots ,\mathcal{K}_{L,N}$  each contain a single feature index
10: return  $\pi$  that maps  $k\in \mathcal{K}_{L,n}$  to  $n$ , for every  $n\in [N]$ .
I Entanglement and Surrogate Entanglement Are Strongly Correlated
In Appendix G, we introduced a surrogate entanglement measure (Definition 4) to facilitate efficient implementation of the feature arrangement search scheme from Section 5.1. Figure 5 supports the viability of the chosen surrogate, demonstrating empirically that it is strongly correlated with entanglement of the empirical data tensor (Definition 1 and Equation (2)).
Figure 5: Surrogate entanglement (Definition 4) is strongly correlated with the entanglement (Definition 1) of the empirical data tensor. Presented are average entanglement and average surrogate entanglement under canonical partitions, admitted by the Speech Commands audio datasets [64] considered in Figure 3. Remarkably, the Pearson correlation between the quantities is 0.974. For further details see caption of Figure 3 as well as Appendix L.
J Extension to Arbitrary Dimensional Models and Data
In this appendix, we extend our theoretical analysis and experiments, including the algorithm for enhancing the suitability of data to locally connected neural networks, from one-dimensional (sequential) models and data to  $P$ -dimensional models and data (such as two-dimensional image data or three-dimensional video data), for  $P \in \mathbb{N}$ . Specifically, Appendix J.1 extends Section 3, Appendix J.2 extends Section 4 and Appendix J.3 extends Section 5.
To ease presentation, we consider  $P$ -dimensional data instances whose feature vectors are associated with coordinates  $(n_1,\ldots ,n_P)\in [N]^P$ , where  $N = 2^L$  for some  $L\in \mathbb{N}$  (if this is not the case we may add constant features as needed).
Figure 6: The analyzed tensor network equivalent to a locally connected neural network operating over  $P$ -dimensional data, for  $P = 2$ . (a) The tensor network adheres to a perfect  $2^{P}$ -ary tree connectivity with  $N^{P}$  leaf nodes, where  $N = 2^{L}$  for some  $L \in \mathbb{N}$ , and generates  $\mathcal{W}_{\mathrm{TN}}^{P} \in \mathbb{R}^{D_{1} \times \dots \times D_{NP}}$ . Axes corresponding to open edges are indexed such that open edges descendant to any node of the tree have contiguous indices. The lengths of axes corresponding to inner (non-open) edges are equal to  $R \in \mathbb{N}$ , referred to as the width of the tensor network. (b) Exemplar  $\mu : [N]^{P} \to [N^{P}]$  compatible with the locally connected tensor network (Definition 5), mapping  $P$ -dimensional coordinates to axes indices of  $\mathcal{W}_{\mathrm{TN}}^{P}$ . (c) Contracting  $\mathcal{W}_{\mathrm{TN}}^{P}$  with vectors  $\{\mathbf{x}^{(n_{1}, \ldots, n_{P})}\}_{n_{1}, \ldots, n_{P} \in [N]}$  according to a compatible  $\mu$  produces  $\langle \otimes_{n=1}^{NP} \mathbf{x}^{\mu^{-1}(n)}, \mathcal{W}_{\mathrm{TN}}^{P} \rangle$ . Performing these contractions can be viewed as a forward pass of a certain locally connected neural network (with polynomial non-linearity) over the data instance  $\{\mathbf{x}^{(n_{1}, \ldots, n_{P})}\}_{n_{1}, \ldots, n_{P} \in [N]}$  (see, e.g., [12, 10, 35, 49]).
μ : [8] × [8] → [64]
1	2	5	6	17	18	21	22
3	4	7	8	19	20	23	24
9	10	13	14	25	26	29	30
11	12	15	16	27	28	31	32
33	34	37	38	49	50	53	54
35	36	39	40	51	52	55	56
41	42	45	46	57	58	61	62
43	44	47	48	59	60	63	64
J.1 Low Entanglement Under Canonical Partitions Is Necessary and Sufficient for Fitting Tensor
We introduce the locally connected tensor network equivalent to a locally connected neural network that operates over  $P$ -dimensional data (Appendix J.1.1). Subsequently, we establish a necessary and sufficient condition required for it to fit a given tensor (Appendix J.1.2), generalizing the results of Section 3.2.
J.1.1 Tensor Network Equivalent to a Locally Connected Neural Network
For  $P$ -dimensional data, the locally connected tensor network we consider (defined in Section 3.1 for one-dimensional data) has an underlying perfect  $2^{P}$ -ary tree graph of height  $L$ . We denote the tensor it generates by  $\mathcal{W}_{\mathrm{TN}}^P \in \mathbb{R}^{D_1 \times \dots \times D_{NP}}$ . Figure 6(a) provides its diagrammatic definition. As in the one-dimensional case, the lengths of axes corresponding to inner edges are taken to be  $R \in \mathbb{N}$ , referred to as the width of the tensor network.
The axes of  $\mathcal{W}_{\mathrm{TN}}^P$  are associated with  $P$ -dimensional coordinates through a bijective function  $\mu : [N]^P \to [N^P]$ .
Definition 5. We say that a bijective function  $\mu : [N]^P \to [N^P]$  is compatible with the locally connected tensor network if, for any node in the tensor network, the coordinates mapped to indices of  $\mathcal{W}_{\mathrm{TN}}^P$ 's axes descendant to that node form a contiguous  $P$ -dimensional cubic block in  $[N]^P$  (e.g., square block when  $P = 2$ ) — see Figure 6(b) for an illustration. With slight abuse of notation, for  $\mathcal{K} \subseteq [N]^P$  we denote  $\mu(\mathcal{K}) := \{\mu(n_1, \ldots, n_P) : (n_1, \ldots, n_P) \in \mathcal{K}\} \subseteq [N^P]$ .
Contracting the locally connected tensor network with  $\{\mathbf{x}^{(n_1,\dots,n_P)}\in \mathbb{R}^{D_\mu (n_1,\dots,n_P)}\}_{n_1,\ldots ,n_P\in [N]}$  according to a compatible  $\mu$  , as depicted in Figure 6(c), can be viewed as a forward pass of the data instance  $\{\mathbf{x}^{(n_1,\dots,n_P)}\}_{n_1,\dots ,n_P\in [N]}$  through a locally connected neural network (with polynomial non-linearity), which produces the scalar  $\langle \otimes_{n = 1}^{N^P}\mathbf{x}^{\mu^{-1}(n)},\mathcal{W}_{\mathrm{TN}}^P\rangle$  (see, e.g., [12, 10, 35, 49]).
Figure 7: The canonical partitions of  $[N]^P$ , for  $P = 2$  and  $N = 2^L$  with  $L \in \mathbb{N}$ . Every  $l \in \{0, \dots, L\}$  contributes  $2^{l \cdot P}$  canonical partitions, each induced by  $\mathcal{K} = \times_{p=1}^{P} \{2^{L-l} \cdot (n_p - 1) + 1, \dots, 2^{L-l} \cdot n_p\}$  for  $n_1, \dots, n_P \in [2^l]$ .
J.1.2 Necessary and Sufficient Condition for Fitting Tensor
The ability of the locally connected tensor network, defined in Appendix J.1.1, to fit (i.e. represent) a tensor is determined by the entanglements that the tensor admits under partitions of its axes, induced by the following canonical partitions of  $[N]^P$ .
Definition 6. The canonical partitions of  $[N]^P$ , illustrated in Figure 7 for  $P = 2$ , are: $^{10}$
$$
\mathcal {C} _ {N} ^ {P} := \left\{\left(\mathcal {K}, \mathcal {K} ^ {c}\right): \mathcal {K} = \times_ {p = 1} ^ {P} \left\{2 ^ {L - l} \cdot \left(n _ {p} - 1\right) + 1, \dots , 2 ^ {L - l} \cdot n _ {p} \right\}, \right.
$$
$$
l \in \left\{0, \dots , L \right\}, n _ {1}, \dots , n _ {P} \in \left[ 2 ^ {l} \right] \}.
$$
With the definition of canonical partitions for  $P$ -dimensional data in place, Theorem 3 generalizes Theorem 1. In particular, suppose that  $\mathcal{W}_{\mathrm{TN}}^P$  — the tensor generated by the locally connected tensor network — well-approximates  $\mathcal{A} \in \mathbb{R}^{D_1 \times \dots \times D_{N^P}}$ . Then, given a compatible  $\mu : [N]^P \to [N^P]$  (Definition 5), Theorem 3 establishes that the entanglement of  $\mathcal{A}$  with respect to  $(\mu(\mathcal{K}), \mu(\mathcal{K})^c)$ , where  $(\mathcal{K}, \mathcal{K}^c) \in \mathcal{C}_N^P$ , cannot be much larger than  $\ln(R)$ , whereas the expected entanglement attained by a random tensor with respect to  $(\mu(\mathcal{K}), \mu(\mathcal{K})^c)$  is on the order of  $\min\{|K|, |\mathcal{K}^c|\}$  (which is linear in  $N^P$  for some canonical partitions).
In the other direction, Theorem 4 implies that low entanglement under partitions of axes induced by canonical partitions of  $[N]^P$  is not only necessary for a tensor to be fit by the locally connected tensor network, but also sufficient.
Theorem 3. Let  $\mathcal{W}_{TN}^P \in \mathbb{R}^{D_1 \times \dots \times D_{N^P}}$  be a tensor generated by the locally connected tensor network defined in Appendix J.1.1, and  $\mu : [N]^P \to [N^P]$  be a compatible map from  $P$ -dimensional coordinates to axes indices of  $\mathcal{W}_{TN}^P$  (Definition 5). For any  $\mathcal{A} \in \mathbb{R}^{D_1 \times \dots \times D_{N^P}}$  and  $\epsilon \in [0, \| \mathcal{A} \| / 4]$ , if  $\| \mathcal{W}_{TN}^P - \mathcal{A} \| \leq \epsilon$ , then for all canonical partitions  $(\mathcal{K}, \mathcal{K}^c) \in \mathcal{C}_N^P$  (Definition 6):
$$
Q E (\mathcal {A}; \mu (\mathcal {K})) \leq \ln (R) + \frac {2 \epsilon}{\| \mathcal {A} \|} \cdot \ln \left(D _ {\mu (\mathcal {K})}\right) + 2 \sqrt {\frac {2 \epsilon}{\| \mathcal {A} \|}}, \tag {6}
$$
For sets  $S_{1},\ldots ,S_{P}$  , we denote their Cartesian product by  $\times_{p = 1}^{P}S_{p}$
where  $D_{\mu(\mathcal{K})} \coloneqq \min \{\prod_{n \in \mu(\mathcal{K})} D_n, \prod_{n \in \mu(\mathcal{K})^c} D_n\}$ . In contrast, a random  $\mathcal{A}' \in \mathbb{R}^{D_1 \times \dots \times D_{N^P}}$ , drawn according to the uniform distribution over the set of unit norm tensors, satisfies for all canonical partitions  $(\mathcal{K}, \mathcal{K}^c) \in \mathcal{C}_N^P$ :
$$
\mathbb {E} \left[ Q E \left(\mathcal {A} ^ {\prime}; \mu (\mathcal {K})\right) \right] \geq \min  \left\{\left| \mathcal {K} \right|, \left| \mathcal {K} ^ {c} \right| \right\} \cdot \ln \left(\min  _ {n \in [ N ^ {P} ]} D _ {n}\right) + \ln \left(\frac {1}{2}\right) - \frac {1}{2}. \tag {7}
$$
Proof sketch (proof in Appendix M.7). The proof is analogous to that of Theorem 1.
Theorem 4. Let  $\mathcal{A} \in \mathbb{R}^{D_1 \times \dots \times D_{N^P}}$  and  $\epsilon > 0$ . Suppose that for all canonical partitions  $(\mathcal{K}, \mathcal{K}^c) \in \mathcal{C}_N^P$  (Definition 6) it holds that  $QE(\mathcal{A}; \mu(\mathcal{K})) \leq \frac{\epsilon^2}{(2N^P - 3)\|\mathcal{A}\|^2} \cdot \ln(R)$ , where  $\mu : [N]^P \to [N^P]$  is compatible with the locally connected tensor network (Definition 5). Then, there exists an assignment for the tensors constituting the locally connected tensor network (defined in Appendix J.1.1) such that it generates  $\mathcal{W}_{TN}^P \in \mathbb{R}^{D_1 \times \dots \times D_{N^P}}$  satisfying:
$$
\left\| \mathcal {W} _ {T N} ^ {P} - \mathcal {A} \right\| \leq \epsilon .
$$
Proof sketch (proof in Appendix M.8). The claim follows through a reduction from the locally connected tensor network for  $P$ -dimensional data to that for one-dimensional data (defined in Section 3.1), i.e. from perfect  $2^{P}$ -ary to perfect binary tree tensor networks. Specifically, we consider a modified locally connected tensor network for one-dimensional data, where axes corresponding to different inner edges can vary in length (as opposed to all having length  $R$ ). We then show, by arguments analogous to those in the proof of Theorem 2, that it can approximate  $\mathcal{A}$  while having certain inner axes, related to the canonical partitions of  $[N]^{P}$ , of lengths at most  $R$ . The proof concludes by establishing that, any tensor represented by such a locally connected tensor network for one-dimensional data can be represented via the locally connected tensor network for  $P$ -dimensional data (where the length of each axis corresponding to an inner edge is  $R$ ).
J.2 Low Entanglement Under Canonical Partitions Is Necessary and Sufficient for Accurate Prediction
In this appendix, we consider the locally connected tensor network from Appendix J.1.1 in a machine learning setting, and extend the results and experiments of Section 4 from one-dimensional to  $P$ -dimensional models and data.
J.2.1 Accurate Prediction Is Equivalent to Fitting Data Tensor
The locally connected tensor network generating  $\mathcal{W}_{\mathrm{TN}}^P \in \mathbb{R}^{D_1 \times \dots \times D_{N^P}}$  is equivalent to a locally connected neural network operating over  $P$ -dimensional data (see Appendix J.1.1). Specifically, a forward pass of the latter over  $\{\mathbf{x}^{(n_1, \ldots, n_P)} \in \mathbb{R}^{D_\mu(n_1, \ldots, n_P)}\}_{n_1, \ldots, n_P \in [N]}$  yields  $\langle \otimes_{n=1}^{N^P} \mathbf{x}^{\mu^{-1}(n)}, \mathcal{W}_{\mathrm{TN}}^P \rangle$ , for a compatible  $\mu : [N]^P \to [N^P]$  (Definition 5). Suppose we are given a training set of labeled instances  $\{\{\mathbf{x}^{((n_1, \ldots, n_P), m)}\}_{n_1, \ldots, n_P \in [N]}, y^{(m)}\}_{m=1}^M$  drawn i.i.d. from some distribution, where  $y^{(m)} \in \{1, -1\}$  for  $m \in [M]$ . Learning the parameters of the neural network through the soft-margin support vector machine (SVM) objective amounts to optimizing:
$$
\min _ {\| \mathcal {W} _ {\mathrm {T N}} ^ {P} \| \leq B} \frac {1}{M} \sum_ {m = 1} ^ {M} \max  \left\{0, 1 - y ^ {(m)} \left\langle \otimes_ {n = 1} ^ {N ^ {P}} \mathbf {x} ^ {(\mu^ {- 1} (n), m)}, \mathcal {W} _ {\mathrm {T N}} ^ {P} \right\rangle \right\},
$$
for a predetermined constant  $B > 0$ . This objective generalizes Equation (1) from one-dimensional to  $P$ -dimensional model and data. Assume that instances are normalized, i.e.  $\| \mathbf{x}^{((n_1,\dots ,n_P),m)}\| \leq 1$  for all  $n_1,\ldots ,n_P\in [N],m\in [M]$ , and that  $B\leq 1$ . By a derivation analogous to that succeeding Equation (1) in Section 4.1, if follows that minimizing the SVM objective is equivalent to minimizing  $\left\| \frac{\mathcal{W}_{\mathrm{TN}}^P}{\|\mathcal{W}_{\mathrm{TN}}^P\|} -\frac{\mathcal{D}_{\mathrm{emp}}^P}{\|\mathcal{D}_{\mathrm{emp}}^P\|}\right\|$ , where:
$$
\mathcal {D} _ {\mathrm {e m p}} ^ {P} := \frac {1}{M} \sum_ {m = 1} ^ {M} y ^ {(m)} \cdot \otimes_ {n = 1} ^ {N ^ {P}} \mathbf {x} ^ {(\mu^ {- 1} (n), m)} \tag {8}
$$
extends the notion of empirical data tensor to  $P$ -dimensional data. In other words, the accuracy achievable over the training data is determined by the extent to which  $\frac{\mathcal{W}_{\mathrm{TN}}^P}{\|\mathcal{W}_{\mathrm{TN}}^P\|}$  can fit the normalized empirical data tensor  $\frac{\mathcal{D}_{\mathrm{emp}}^P}{\|\mathcal{D}_{\mathrm{emp}}^P\|}$ .
The same arguments apply to the population loss, in which case  $\mathcal{D}_{\mathrm{emp}}^P$  is replaced by the population data tensor:
$$
\mathcal {D} _ {\text {p o p}} ^ {P} := \mathbb {E} _ {\left\{\mathbf {x} ^ {(n _ {1}, \dots , n _ {P})} \right\} _ {n _ {1}, \dots , n _ {P} \in [ N ]}, y} \left[ y \cdot \otimes_ {n = 1} ^ {N ^ {P}} \mathbf {x} ^ {\mu^ {- 1} (n)} \right]. \tag {9}
$$
The achievable accuracy over the population is therefore determined by the extent to which  $\frac{\mathcal{W}_{\mathrm{TN}}^P}{\|\mathcal{W}_{\mathrm{TN}}^P\|}$  can fit the normalized population data tensor  $\frac{\mathcal{D}_{\mathrm{pop}}^P}{\|\mathcal{D}_{\mathrm{pop}}^P\|}$ . Accordingly, we refer to the minimal distance from it as the supobtimality in achievable accuracy, generalizing Definition 3 from Section 4.1.
Definition 7. In the context of the classification setting above, the suboptimality in achievable accuracy is:
$$
\mathrm {S u b O p t} ^ {P} := \min _ {\mathcal {W} _ {\mathrm {T N}} ^ {P}} \left\| \frac {\mathcal {W} _ {\mathrm {T N}} ^ {P}}{\| \mathcal {W} _ {\mathrm {T N}} ^ {P} \|} - \frac {\mathcal {D} _ {\mathrm {p o p}} ^ {P}}{\| \mathcal {D} _ {\mathrm {p o p}} ^ {P} \|} \right\|.
$$
J.2.2 Necessary and Sufficient Condition for Accurate Prediction
In the classification setting of Appendix J.2.1, by invoking Theorems 3 and 4 from Appendix J.1.2, we conclude that the suboptimality in achievable accuracy is small if and only if the population (empirical) data tensor  $\mathcal{D}_{\mathrm{pop}}^P (\mathcal{D}_{\mathrm{emp}}^P)$  admits low entanglement under the canonical partitions of features (Definition 6). Specifically, we establish Corollary 3, Proposition 4 and Corollary 4, which generalize Corollary 1, Proposition 2 and Corollary 2 from Section 4.2, respectively, to  $P$ -dimensional model and data.
Corollary 3. Consider the classification setting of Appendix J.2.1, and let  $\epsilon \in [0,1/4]$ . If there exists a canonical partition  $(\mathcal{K},\mathcal{K}^c) \in \mathcal{C}_N^P$  (Definition 6) under which  $QE\big(\mathcal{D}_{pop}^P;\mu (\mathcal{K})\big) > \ln (R) + 2\epsilon \cdot \ln (D_{\mu (\mathcal{K})}) + 2\sqrt{2\epsilon}$ , where  $D_{\mu (\mathcal{K})} \coloneqq \min \{\prod_{n\in \mu (\mathcal{K})}D_n,\prod_{n\in \mu (\mathcal{K})^c}D_n\}$ , then:
$$
\operatorname {S u b O p t} ^ {P} > \epsilon .
$$
Conversely, if for all  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N^P$  it holds that  $QE\big(\mathcal{D}_{pop}^{P};\mu (\mathcal{K})\big)\leq \frac{\epsilon^{2}}{8N^{P} - 12}\cdot \ln (R),$  then:
$$
\operatorname {S u b O p t} ^ {P} \leq \epsilon .
$$
Proof. Implied by Theorems 3 and 4 after accounting for  $\mathcal{W}_{\mathrm{TN}}^P$  being normalized in the suboptimality in achievable accuracy, as done in the proof of Corollary 1.