SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yotam Alexander*, Nimrod De La Vega*, Noam Razin, Nadav Cohen
{yotama,nimrodd,noamrazin}@mail.tau.ac.il,cohennadav@cs.tau.ac.il
The question of what makes a data distribution suitable for deep learning is a fundamental open problem. Focusing on locally connected neural networks (a prevalent family of architectures that includes convolutional and recurrent neural networks as well as local self-attention models), we address this problem by adopting theoretical tools from quantum physics. Our main theoretical result states that a certain locally connected neural network is capable of accurate prediction over a data distribution if and only if the data distribution admits low quantum entanglement under certain canonical partitions of features. As a practical application of this result, we derive a preprocessing method for enhancing the suitability of a data distribution to locally connected neural networks. Experiments with widespread models over various datasets demonstrate our findings. We hope that our use of quantum entanglement will encourage further adoption of tools from physics for formally reasoning about the relation between deep learning and real-world data.<sup>1</sup>
Deep learning is delivering unprecedented performance when applied to data modalities involving images, text and audio. On the other hand, it is known both theoretically and empirically [53, 1] that there exist data distributions over which deep learning utterly fails. The question of what makes a data distribution suitable for deep learning is a fundamental open problem in the field.
A prevalent family of deep learning architectures is that of locally connected neural networks. It includes, among others: (i) convolutional neural networks, which dominate the area of computer vision; (ii) recurrent neural networks, which were the most common architecture for sequence (e.g. text and audio) processing, and are experiencing a resurgence by virtue of S4 models [26]; and (iii) local variants of self-attention neural networks [46]. Conventional wisdom postulates that data distributions suitable for locally connected neural networks are those exhibiting a "local nature," and there have been attempts to formalize this intuition [66, 28, 15]. However, to the best of our knowledge, there are no characterizations providing necessary and sufficient conditions for a data distribution to be suitable to a locally connected neural network.
A seemingly distinct scientific discipline tying distributions and computational models is quantum physics. There, distributions of interest are described by tensors, and the associated computational models are tensor networks. While there is shortage in formal tools for assessing the suitability of data distributions to deep learning architectures, there exists a widely accepted theory that allows for assessing the suitability of tensors to tensor networks. The theory is based on the notion of quantum
<sup>1</sup>Due to lack of space, a significant portion of the paper is deferred to the appendices. We refer the reader to [2] for a self-contained version of the text.
entanglement, which quantifies dependencies that a tensor admits under partitions of its axes (for a given tensor  $\mathcal{A}$  and a partition of its axes to sets  $\kappa$  and  $\kappa^c$ , the entanglement is a non-negative number quantifying the dependence that  $\mathcal{A}$  admits between  $\kappa$  and  $\kappa^c$ ).
In this paper, we apply the foregoing theory to a tensor network equivalent to a certain locally connected neural network, and derive theorems by which fitting a tensor is possible if and only if the tensor admits low entanglement under certain canonical partitions of its axes. We then consider the tensor network in a machine learning context, and find that its ability to attain low approximation error, i.e. to express a solution with low population loss, is determined by its ability to fit a particular tensor defined by the data distribution, whose axes correspond to features. Combining the latter finding with the former theorems, we conclude that a locally connected neural network is capable of accurate prediction over a data distribution if and only if the data distribution admits low entanglement under canonical partitions of features. Experiments with different datasets corroborate this conclusion, showing that the accuracy of common locally connected neural networks (including modern convolutional, recurrent, and local self-attention neural networks) is inversely correlated to the entanglement under canonical partitions of features in the data (the lower the entanglement, the higher the accuracy, and vice versa).
The above results bring forth a recipe for enhancing the suitability of a data distribution to locally connected neural networks: given a dataset, search for an arrangement of features which leads to low entanglement under canonical partitions, and then arrange the features accordingly. Unfortunately, the above search is computationally prohibitive. However, if we employ a certain correlation-based measure as a surrogate for entanglement, i.e. as a gauge for dependence between sides of a partition of features, then the search converts into a succession of minimum balanced cut problems, thereby admitting use of well-established graph theoretical tools, including ones designed for large scale [29, 57]. We empirically evaluate this approach on various datasets, demonstrating that it substantially improves prediction accuracy of common locally connected neural networks (including modern convolutional, recurrent, and local self-attention neural networks).
The data modalities to which deep learning is most commonly applied — namely ones involving images, text and audio — are often regarded as natural (as opposed to, for example, tabular data fusing heterogeneous information). We believe the difficulty in explaining the suitability of such modalities to deep learning may be due to a shortage in tools for formally reasoning about natural data. Concepts and tools from physics — a branch of science concerned with formally reasoning about natural phenomena — may be key to overcoming said difficulty. We hope that our use of quantum entanglement will encourage further research along this line.
For simplicity, the main text treats locally connected neural networks whose input data is one dimensional (e.g. text or audio). We defer to Appendix J an extension of the analysis and experiments to models intaking data of arbitrary dimension (e.g. two-dimensional images). Due to lack of space, we also defer our review of related work to Appendix A.
We use  $\| \cdot \|$  and  $\langle \cdot, \cdot \rangle$  to denote the Euclidean (Frobenius) norm and inner product, respectively. We shorthand  $[N] \coloneqq \{1, \dots, N\}$ , where  $N \in \mathbb{N}$ . The complement of  $\mathcal{K} \subseteq [N]$  is denoted by  $\mathcal{K}^c$ , i.e.  $\mathcal{K}^c \coloneqq [N] \setminus \mathcal{K}$ .
2.1 Tensors and Tensor Networks
For our purposes, a tensor is an array with multiple axes  $\mathcal{A} \in \mathbb{R}^{D_1 \times \dots \times D_N}$ , where  $N \in \mathbb{N}$  is its order and  $D_1, \ldots, D_N \in \mathbb{N}$  are its axes lengths. The  $(d_1, \ldots, d_N)$ 'th entry of  $\mathcal{A}$  is denoted  $\mathcal{A}_{d_1, \ldots, d_N}$ .
Contraction between tensors is a generalization of multiplication between matrices. Two matrices  $\mathbf{A} \in \mathbb{R}^{D_1 \times D_2}$  and  $\mathbf{B} \in \mathbb{R}^{D_1' \times D_2'}$  can be multiplied if  $D_2 = D_1'$ , in which case we get a matrix in  $\mathbb{R}^{D_1 \times D_2'}$  holding  $\sum_{d=1}^{D_2} \mathbf{A}_{d_1, d} \cdot \mathbf{B}_{d, d_2'}$  in entry  $(d_1, d_2') \in [D_1] \times [D_2']$ . More generally, two tensors  $\mathcal{A} \in \mathbb{R}^{D_1 \times \dots \times D_N}$  and  $\mathcal{B} \in \mathbb{R}^{D_1' \times \dots \times D_{N'}}$  can be contracted along axis  $n \in [N]$  of  $\mathcal{A}$  and  $n' \in [N']$  of  $\mathcal{B}$  if  $D_n = D_{n'}'$ , in which case we get a tensor of size  $D_1 \times \dots D_{n-1} \times D_{n+1} \times \dots \times D_N \times D_1' \times \dots \times D_{n'-1}' \times D_{n'+1}' \dots \times D_{N'}'$  holding  $\sum_{d=1}^{D_n} \mathcal{A}_{d_1, \ldots, d_{n-1}, d, d_{n+1}, \ldots, d_N} \cdot \mathcal{B}_{d_1', \ldots, d_{n'-1}', d, d_{n'+1}', \ldots, d_{N'}}$  in the entry indexed by  $\{d_k \in [D_k]\}_{k \in [N] \setminus \{n\}}$  and  $\{d_k' \in [D_k']\}_{k \in [N'] \setminus \{n'\}}$ .
Tensor networks are prominent computational models for fitting (i.e. representing) tensors. More specifically, a tensor network is a weighted graph that describes formation of a (typically high-order)
(b)
Figure 1: Tensor networks form a graphical language for fitting (i.e. representing) tensors through tensor contractions. Tensor network definition: Every node in a tensor network is associated with a tensor, whose order is equal to the number of edges emanating from the node. An edge connecting two nodes specifies contraction between the tensors associated with the nodes (Section 2.1), where the weight of the edge signifies the respective axes lengths. Tensor networks may also contain open edges, i.e. edges that are connected to a node on one side and are open on the other. The number of such open edges is equal to the order of the tensor produced by contracting the tensor network. Illustrations: Presented are exemplar tensor network diagrams of: (a) an order  $N$  tensor  $\mathcal{A} \in \mathbb{R}^{D_1 \times \dots \times D_N}$ ; (b) a vector-matrix multiplication between  $\mathbf{M} \in \mathbb{R}^{D_1 \times D_2}$  and  $\mathbf{v} \in \mathbb{R}^{D_2}$ , which results in the vector  $\mathbf{M}\mathbf{v} \in \mathbb{R}^{D_1}$ ; and (c) a tensor network generating  $\mathcal{W} \in \mathbb{R}^{D_1 \times D_2 \times D_3}$ .
tensor via contractions between (typically low-order) tensors. As customary (cf. [42]), we will present tensor networks via graphical diagrams to avoid cumbersome notation — see Figure 1 for details.
2.2 Quantum Entanglement
In quantum physics, the distribution of possible states for a multi-particle ("many body") system is described by a tensor, whose axes are associated with individual particles. A key property of the distribution is the dependence it admits under a given partition of the particles (i.e. between a given set of particles and its complement). This dependence is formalized through the notion of quantum entanglement, defined using the distribution's description as a tensor — see Definition 1 below.
Quantum entanglement lies at the heart of a widely accepted theory which allows assessing the ability of a tensor network to fit a given tensor (cf. [16, 35]). In Section 3 we specialize this theory to a tensor network equivalent to a certain locally connected neural network. The specialized theory will be used (in Section 4) to establish our main theoretical contribution: a necessary and sufficient condition for when the locally connected neural network is capable of accurate prediction over a data distribution.
Definition 1. For a tensor  $\mathcal{A} \in \mathbb{R}^{D_1 \times \dots \times D_N}$  and subset of its axes  $\mathcal{K} \subseteq [N]$ , let  $[\mathcal{A};\mathcal{K}] \in \mathbb{R}^{\prod_{n \in \mathcal{K}} D_n \times \prod_{n \in \mathcal{K}^c} D_n}$  be the arrangement of  $\mathcal{A}$  as a matrix where rows correspond to axes  $\mathcal{K}$  and columns correspond to the remaining axes  $\mathcal{K}^c \coloneqq [N] \backslash \mathcal{K}$ . Denote by  $\sigma_1 \geq \dots \geq \sigma_{D_\mathcal{K}} \in \mathbb{R}_{\geq 0}$  the singular values of  $[\mathcal{A};\mathcal{K}]$ , where  $D_\mathcal{K} \coloneqq \min\{\prod_{n \in \mathcal{K}} D_n, \prod_{n \in \mathcal{K}^c} D_n\}$ . The quantum entanglement $^2$  of  $\mathcal{A}$  with respect to the partition  $(\mathcal{K},\mathcal{K}^c)$  is the entropy of the distribution  $\{\rho_d \coloneqq \sigma_d^2 / \sum_{d'=1}^{D_\mathcal{K}} \sigma_{d'}^2\}_{d=1}^{D_\mathcal{K}}$ , i.e.  $QE(\mathcal{A};\mathcal{K}) \coloneqq -\sum_{d=1}^{D_\mathcal{K}} \rho_d \ln(\rho_d)$ . By convention, if  $\mathcal{A} = 0$  then  $QE(\mathcal{A};\mathcal{K}) = 0$ .
3 Low Entanglement Under Canonical Partitions Is Necessary and Sufficient for Fitting Tensor
In this section, we prove that a tensor network equivalent to a certain locally connected neural network can fit a tensor if and only if the tensor admits low entanglement under certain canonical partitions of its axes. We begin by introducing the tensor network (Section 3.1). Subsequently, we establish the necessary and sufficient condition required for it to fit a given tensor (Section 3.2). For conciseness, the treatment in this section is limited to one-dimensional (sequential) models; see Appendix J.1 for an extension to arbitrary dimensions.
3.1 Tensor Network Equivalent to a Locally Connected Neural Network
Let  $N \in \mathbb{N}$ , and for simplicity suppose that  $N = 2^L$  for some  $L \in \mathbb{N}$ . We consider a tensor network with an underlying perfect binary tree graph of height  $L$ , which generates  $\mathcal{W}_{\mathrm{TN}} \in \mathbb{R}^{D_1 \times \dots \times D_N}$ . Figure 2(a) provides its diagrammatic definition. For simplicity of presentation, the lengths of axes corresponding to inner (non-open) edges are taken to all be equal to some  $R \in \mathbb{N}$ , referred to as the width of the tensor network.
There exist multiple notions of entanglement in quantum physics (see, e.g., [35]). The one we consider is the most common, known as entanglement entropy.
Figure 2: The analyzed tensor network equivalent to a locally connected neural network. (a) We consider a tensor network adhering to a perfect binary tree connectivity with  $N = 2^L$  leaf nodes, for  $L \in \mathbb{N}$ , generating  $\mathcal{W}_{\mathrm{TN}} \in \mathbb{R}^{D_1 \times \dots \times D_N}$ . Axes corresponding to open edges are indexed such that open edges descendant to any node of the tree have contiguous indices. The lengths of axes corresponding to inner (non-open) edges are equal to  $R \in \mathbb{N}$ , referred to as the width of the tensor network. (b) Contracting  $\mathcal{W}_{\mathrm{TN}}$  with vectors  $\mathbf{x}^{(1)} \in \mathbb{R}^{D_1}, \ldots, \mathbf{x}^{(N)} \in \mathbb{R}^{D_N}$  produces  $\langle \otimes_{n=1}^{N} \mathbf{x}^{(n)}, \mathcal{W}_{\mathrm{TN}} \rangle$ . Performing these contractions from leaves to root can be viewed as a forward pass of a data instance  $(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(N)})$  through a certain locally connected neural network (with polynomial non-linearity; see, e.g., [12, 10, 35, 49]). Accordingly, we call the tensor network generating  $\mathcal{W}_{\mathrm{TN}}$  a locally connected tensor network.
As identified by previous works, the tensor network depicted in Figure 2(a) is equivalent to a certain locally connected neural network (with polynomial non-linearity — see, e.g., [12, 10, 35, 49]). In particular, contracting the tensor network with vectors  $\mathbf{x}^{(1)} \in \mathbb{R}^{D_1}, \ldots, \mathbf{x}^{(N)} \in \mathbb{R}^{D_N}$ , as illustrated in Figure 2(b), can be viewed as a forward pass of the data instance  $(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(N)})$  through a locally connected neural network, whose hidden layers are of width  $R$ . This computation results in a scalar equal to  $\left\langle \otimes_{n=1}^{N} \mathbf{x}^{(n)}, \mathcal{W}_{\mathrm{TN}} \right\rangle$ , where  $\otimes$  stands for the outer product. In light of its equivalence to a locally connected neural network, we will refer to the tensor network as a locally connected tensor network. We note that for the equivalent neural network to be practical (in terms of memory and runtime), the width  $R$  needs to be of moderate size (typically no more than a few thousands). Specifically,  $R$  cannot be exponential in the order  $N$ , meaning  $\ln(R)$  needs to be much smaller than  $N$ .
By virtue of the locally connected tensor network's equivalence to a deep neural network, it has been paramount for the study of expressiveness and generalization in deep learning [12, 9, 10, 13, 14, 54, 34, 35, 3, 30, 31, 36, 47, 48, 49, 50]. Although the equivalent deep neural network (which has polynomial non-linearity) is less common than other neural networks (e.g., ones with ReLU non-linearity), it has demonstrated competitive performance in practice [8, 11, 55, 58, 22]. More importantly, its theoretical analyses, through the equivalence to the locally connected tensor network, brought forth numerous insights that were demonstrated empirically and led to development of practical tools for common locally connected architectures. Continuing this line, we will demonstrate our theoretical insights through experiments with widespread convolutional, recurrent and local self-attention architectures (Section 4.3), and employ our theory for deriving an algorithm that enhances the suitability of a data distribution to said architectures (Section 5).
3.2 Necessary and Sufficient Condition for Fitting Tensor
Herein we show that the ability of the locally connected tensor network (defined in Section 3.1) to fit (i.e. represent) a given tensor is determined by the entanglements that the tensor admits under the below-defined canonical partitions of  $[N]$ . Note that each canonical partition comprises a subset of contiguous indices, so low entanglement under canonical partitions can be viewed as a formalization of locality.
Definition 2. The canonical partitions of  $[N] = [2^L]$  (illustrated in Figure 4 of Appendix B) are:
$$
\mathcal {C} _ {N} := \left\{\left(\mathcal {K}, \mathcal {K} ^ {c}\right): \mathcal {K} = \left\{2 ^ {L - l} \cdot (n - 1) + 1, \dots , 2 ^ {L - l} \cdot n \right\}, l \in \left\{0, \dots , L \right\}, n \in \left[ 2 ^ {l} \right] \right\}.
$$
By appealing to known upper bounds on the entanglements that a given tensor network supports [16, 35], we now establish that if the locally connected tensor network can fit a given tensor, that tensor
3For any  $\{\mathbf{x}^{(n)}\in \mathbb{R}^{D_n}\}_{n = 1}^N$  , the outer product  $\otimes_{n = 1}^{N}\mathbf{x}^{(n)}\in \mathbb{R}^{D_1\times \dots \times D_N}$  is defined element-wise by  $[\otimes_{n = 1}^{N}\mathbf{x}^{(n)}]_{d_1,\ldots ,d_N} = \prod_{n = 1}^{N}\mathbf{x}_{d_n}^{(n)}$  , where  $d_{1}\in [D_{1}],\ldots ,d_{N}\in [D_{N}]$
must admit low entanglement under the canonical partitions of its axes. Namely, suppose that  $\mathcal{W}_{\mathrm{TN}}$  — the tensor generated by the locally connected tensor network — well-approximates an order  $N$  tensor  $\mathcal{A}$ . Then, Theorem 1 below shows that the entanglement of  $\mathcal{A}$  with respect to every canonical partition  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N$  cannot be much larger than  $\ln (R)$  (recall that  $R$  is the width of the locally connected tensor network, and that in practical settings  $\ln (R)$  is much smaller than  $N$ ), whereas the expected entanglement of a random tensor with respect to  $(\mathcal{K},\mathcal{K}^c)$  is on the order of  $\min \{| \mathcal{K} |, |\mathcal{K}^c | \}$  (which is linear in  $N$  for most canonical partitions).
In the other direction, Theorem 2 below implies that low entanglement under the canonical partitions is not only necessary for a tensor to be fit by the locally connected tensor network, but also sufficient.
Theorem 1. Let  $\mathcal{W}_{\mathrm{TN}} \in \mathbb{R}^{D_1 \times \dots \times D_N}$  be a tensor generated by the locally connected tensor network defined in Section 3.1, and let  $\mathcal{A} \in \mathbb{R}^{D_1 \times \dots \times D_N}$ . For any  $\epsilon \in [0, \| \mathcal{A} \|/4]$ , if  $\| \mathcal{W}_{\mathrm{TN}} - \mathcal{A} \| \leq \epsilon$ , then for all canonical partitions  $(\mathcal{K}, \mathcal{K}^c) \in \mathcal{C}_N$  (Definition 2) it holds that  $QE(\mathcal{A}; \mathcal{K}) \leq \ln(R) + \frac{2\epsilon}{\|\mathcal{A}\|} \cdot \ln(D_{\mathcal{K}}) + 2\sqrt{\frac{2\epsilon}{\|\mathcal{A}\|}}$ , where  $D_{\mathcal{K}} := \min\{\prod_{n \in \mathcal{K}} D_n, \prod_{n \in \mathcal{K}^c} D_n\}$ . In contrast, a random tensor  $\mathcal{A}' \in \mathbb{R}^{D_1 \times \dots \times D_N}$ , drawn according to the uniform distribution over the set of unit norm tensors, satisfies  $\mathbb{E}[QE(\mathcal{A}'; \mathcal{K})] \geq \min\{| \mathcal{K}|, |\mathcal{K}^c| \}\cdot \ln(\min_{n \in [N]} D_n) + \ln\left(\frac{1}{2}\right) - \frac{1}{2}$  for all canonical partitions  $(\mathcal{K}, \mathcal{K}^c) \in \mathcal{C}_N$ .
Proof sketch (proof in Appendix M.2). In general, the entanglements that a tensor network supports can be upper bounded through cuts in its graph [16, 35]. For the locally connected tensor network, these bounds imply that  $QE(\mathcal{W}_{\mathrm{TN}};\mathcal{K}) \leq \ln(R)$  for any canonical partition  $(\mathcal{K},\mathcal{K}^c)$ . The upper bounds on the entanglements of  $\mathcal{A}$  then follow by showing that if  $\mathcal{W}_{\mathrm{TN}}$  and  $\mathcal{A}$  are close, then so are their entanglements. The lower bounds on the expected entanglements of a random tensor are derived based on a characterization from [52].
Theorem 2. Let  $\mathcal{A} \in \mathbb{R}^{D_1 \times \dots \times D_N}$  and  $\epsilon > 0$ . Suppose that for all canonical partitions  $(\mathcal{K}, \mathcal{K}^c) \in \mathcal{C}_N$  (Definition 2) it holds that  $QE(\mathcal{A}; \mathcal{K}) \leq \frac{\epsilon^2}{(2N - 3)\|\mathcal{A}\|^2} \cdot \ln(R)$ . Then, there exists an assignment for the tensors constituting the locally connected tensor network (defined in Section 3.1) such that it generates  $\mathcal{W}_{\mathrm{TN}} \in \mathbb{R}^{D_1 \times \dots \times D_N}$  satisfying  $\|\mathcal{W}_{\mathrm{TN}} - \mathcal{A}\| \leq \epsilon$ .
Proof sketch (proof in Appendix M.3). We show that if  $\mathcal{A}$  has low entanglement under a canonical partition  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N$ , then the singular values of  $[\mathcal{A};\mathcal{K}]$  must decay rapidly (recall that  $[\mathcal{A};\mathcal{K}]$  is the arrangement of  $\mathcal{A}$  as a matrix where rows correspond to axes indexed by  $\mathcal{K}$  and columns correspond to the remaining axes). The approximation guarantee is then obtained through a construction from [25], which is based on truncated singular value decompositions of every  $[\mathcal{A};\mathcal{K}]$  for  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N$ .
4 Low Entanglement Under Canonical Partitions Is Necessary and Sufficient for Accurate Prediction
In this section we consider the locally connected tensor network from Section 3.1 in a machine learning setting. We show that attaining low population loss amounts to fitting a tensor defined by the data distribution, whose axes correspond to features (Section 4.1). Applying the theorems of Section 3.2, we then conclude that the locally connected tensor network is capable of accurate prediction if and only if the data distribution admits low entanglement under canonical partitions of features (Section 4.2). This conclusion is corroborated through experiments, demonstrating that the performance of common locally connected neural networks (including convolutional, recurrent, and local self-attention neural networks) is inversely correlated with the entanglement under canonical partitions of features in the data (Section 4.3). For conciseness, the treatment in this section is limited to one-dimensional (sequential) models and data; see Appendix J.2 for an extension to arbitrary dimensions.
<sup>4</sup>If  $\mathcal{A} = 0$  ,then  $\epsilon = 0$  .In this case, the expression  $\epsilon /\| \mathcal{A}\|$  is by convention equal to zero.