Title: What Makes Data Suitable for a Locally Connected Neural Network? A Necessary and Sufficient Condition Based on Quantum Entanglement.
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Learning

Locally Connected Neural Networks

Data Distributions

Quantum Entanglement

Tensor Networks
Abstract: The question of what makes a data distribution suitable for deep learning is a fundamental open problem. Focusing on locally connected neural networks (a prevalent family of architectures that includes convolutional and recurrent neural networks as well as local self-attention models), we address this problem by adopting theoretical tools from quantum physics. Our main theoretical result states that a certain locally connected neural network is capable of accurate prediction over a data distribution if and only if the data distribution admits low quantum entanglement under certain canonical partitions of features. As a practical application of this result, we derive a preprocessing method for enhancing the suitability of a data distribution to locally connected neural networks. Experiments with widespread models over various datasets demonstrate our findings. We hope that our use of quantum entanglement will encourage further adoption of tools from physics for formally reasoning about the relation between deep learning and real-world data.

What Makes Data Suitable for a Locally Connected Neural Network? A Necessary and Sufficient Condition Based on Quantum Entanglement

Abstract

The question of what makes a data distribution suitable for deep learning is a fundamental open problem. Focusing on locally connected neural networks (a prevalent family of architectures that includes convolutional and recurrent neural networks as well as local self-attention models), we address this problem by adopting theoretical tools from quantum physics. Our main theoretical result states that a certain locally connected neural network is capable of accurate prediction over a data distribution if and only if the data distribution admits low quantum entanglement under certain canonical partitions of features. As a practical application of this result, we derive a preprocessing method for enhancing the suitability of a data distribution to locally connected neural networks. Experiments with widespread models over various datasets demonstrate our findings. We hope that our use of quantum entanglement will encourage further adoption of tools from physics for formally reasoning about the relation between deep learning and real-world data. 1

1 Introduction

Deep learning is delivering unprecedented performance when applied to data modalities involving images, text and audio. On the other hand, it is known both theoretically and empirically [53, 1] that there exist data distributions over which deep learning utterly fails. The question of what makes a data distribution suitable for deep learning is a fundamental open problem in the field.

A prevalent family of deep learning architectures is that of locally connected neural networks . It includes, among others: (i) convolutional neural networks, which dominate the area of computer vision; (ii) recurrent neural networks, which were the most common architecture for sequence ( e.g. text and audio) processing, and are experiencing a resurgence by virtue of S4 models [26]; and (iii) local variants of self-attention neural networks [46]. Conventional wisdom postulates that data distributions suitable for locally connected neural networks are those exhibiting a 'local nature,' and there have been attempts to formalize this intuition [66, 28, 15]. However, to the best of our knowledge, there are no characterizations providing necessary and sufficient conditions for a data distribution to be suitable to a locally connected neural network.

A seemingly distinct scientific discipline tying distributions and computational models is quantum physics . There, distributions of interest are described by tensors , and the associated computational models are tensor networks . While there is shortage in formal tools for assessing the suitability of data distributions to deep learning architectures, there exists a widely accepted theory that allows for assessing the suitability of tensors to tensor networks. The theory is based on the notion of quantum

entanglement , which quantifies dependencies that a tensor admits under partitions of its axes (for a given tensor A and a partition of its axes to sets K and K c , the entanglement is a non-negative number quantifying the dependence that A admits between K and K c ).

In this paper, we apply the foregoing theory to a tensor network equivalent to a certain locally connected neural network, and derive theorems by which fitting a tensor is possible if and only if the tensor admits low entanglement under certain canonical partitions of its axes. We then consider the tensor network in a machine learning context, and find that its ability to attain low approximation error, i.e. to express a solution with low population loss, is determined by its ability to fit a particular tensor defined by the data distribution, whose axes correspond to features. Combining the latter finding with the former theorems, we conclude that a locally connected neural network is capable of accurate prediction over a data distribution if and only if the data distribution admits low entanglement under canonical partitions of features . Experiments with different datasets corroborate this conclusion, showing that the accuracy of common locally connected neural networks (including modern convolutional, recurrent, and local self-attention neural networks) is inversely correlated to the entanglement under canonical partitions of features in the data (the lower the entanglement, the higher the accuracy, and vice versa).

The above results bring forth a recipe for enhancing the suitability of a data distribution to locally connected neural networks: given a dataset, search for an arrangement of features which leads to low entanglement under canonical partitions, and then arrange the features accordingly. Unfortunately, the above search is computationally prohibitive. However, if we employ a certain correlationbased measure as a surrogate for entanglement, i.e. as a gauge for dependence between sides of a partition of features, then the search converts into a succession of minimum balanced cut problems, thereby admitting use of well-established graph theoretical tools, including ones designed for large scale [29, 57]. We empirically evaluate this approach on various datasets, demonstrating that it substantially improves prediction accuracy of common locally connected neural networks (including modern convolutional, recurrent, and local self-attention neural networks).

The data modalities to which deep learning is most commonly applied - namely ones involving images, text and audio - are often regarded as natural (as opposed to, for example, tabular data fusing heterogeneous information). We believe the difficulty in explaining the suitability of such modalities to deep learning may be due to a shortage in tools for formally reasoning about natural data. Concepts and tools from physics - a branch of science concerned with formally reasoning about natural phenomena - may be key to overcoming said difficulty. We hope that our use of quantum entanglement will encourage further research along this line.

2 Preliminaries

For simplicity, the main text treats locally connected neural networks whose input data is one dimensional ( e.g. text or audio). We defer to Appendix J an extension of the analysis and experiments to models intaking data of arbitrary dimension ( e.g. two-dimensional images). Due to lack of space, we also defer our review of related work to Appendix A.

We use ‖·‖ and 〈· , ·〉 to denote the Euclidean (Frobenius) norm and inner product, respectively. We shorthand [ N ] := { 1 , . . . , N } , where N ∈ N . The complement of K ⊆ [ N ] is denoted by K c , i.e. K c := [ N ] \ K .

2.1 Tensors and Tensor Networks

For our purposes, a tensor is an array with multiple axes A ∈ R D 1 ×···× D N , where N ∈ N is its order and D 1 , . . . , D N ∈ N are its axes lengths . The ( d 1 , . . . , d N ) 'th entry of A is denoted A d 1 ,...,d N .

Contraction between tensors is a generalization of multiplication between matrices. Two matrices A ∈ R D 1 × D 2 and B ∈ R D ′ 1 × D ′ 2 can be multiplied if D 2 = D ′ 1 , in which case we get a matrix in R D 1 × D ′ 2 holding ∑ D 2 d =1 A d 1 ,d · B d,d ′ 2 in entry ( d 1 , d ′ 2 ) ∈ [ D 1 ] × [ D ′ 2 ] . More generally, two tensors A ∈ R D 1 ×···× D N and B ∈ R D ′ 1 ×···× D ′ N ′ can be contracted along axis n ∈ [ N ] of A and n ′ ∈ [ N ′ ] of B if D n = D ′ n ′ , in which case we get a tensor of size D 1 ×··· D n -1 × D n +1 ×···× D N × D ′ 1 × · · · × D ′ n ′ -1 × D ′ n ′ +1 · · · × D ′ N ′ holding ∑ D n d =1 A d 1 ,...,d n -1 ,d,d n +1 ,...,d N · B d ′ 1 ,...,d ′ n ′ -1 ,d,d ′ n ′ +1 ,...,d ′ N ′ in the entry indexed by { d k ∈ [ D k ] } k ∈ [ N ] \{ n } and { d ′ k ∈ [ D ′ k ] } k ∈ [ N ′ ] \{ n ′ } .

Tensor networks are prominent computational models for fitting ( i.e. representing) tensors. More specifically, a tensor network is a weighted graph that describes formation of a (typically high-order)

𝐷

!

𝐷

"

𝐌

𝐷

!

𝐯

𝐌𝐯

𝐷

#

(a)

(b)

(c)

𝐷

!

𝐷

$

W

A

𝐷

#

𝐷

#

tensor via contractions between (typically low-order) tensors. As customary ( cf. [42]), we will present tensor networks via graphical diagrams to avoid cumbersome notation - see Figure 1 for details.

2.2 Quantum Entanglement

In quantum physics, the distribution of possible states for a multi-particle ('many body') system is described by a tensor, whose axes are associated with individual particles. A key property of the distribution is the dependence it admits under a given partition of the particles ( i.e. between a given set of particles and its complement). This dependence is formalized through the notion of quantum entanglement , defined using the distribution's description as a tensor - see Definition 1 below.

Quantum entanglement lies at the heart of a widely accepted theory which allows assessing the ability of a tensor network to fit a given tensor ( cf. [16, 35]). In Section 3 we specialize this theory to a tensor network equivalent to a certain locally connected neural network. The specialized theory will be used (in Section 4) to establish our main theoretical contribution: a necessary and sufficient condition for when the locally connected neural network is capable of accurate prediction over a data distribution.

Definition 1. For a tensor A ∈ R D 1 ×···× D N and subset of its axes K ⊆ [ N ] , let J A ; K K ∈ R ∏ n ∈K D n × ∏ n ∈K c D n be the arrangement of A as a matrix where rows correspond to axes K and columns correspond to the remaining axes K c := [ N ] \K . Denote by σ 1 ≥ · · · ≥ σ D K ∈ R ≥ 0 the singular values of J A ; K K , where D K := min { ∏ n ∈K D n , ∏ n ∈K c D n } . The quantum entanglement 2 of A with respect to the partition ( K , K c ) is the entropy of the distribution { ρ d := σ 2 d / ∑ D K d ′ =1 σ 2 d ′ } D K d =1 , i.e. QE ( A ; K ) := -∑ D K d =1 ρ d ln( ρ d ) . By convention, if A = 0 then QE ( A ; K ) = 0 .

3 Low Entanglement Under Canonical Partitions Is Necessary and Sufficient for Fitting Tensor

In this section, we prove that a tensor network equivalent to a certain locally connected neural network can fit a tensor if and only if the tensor admits low entanglement under certain canonical partitions of its axes. We begin by introducing the tensor network (Section 3.1). Subsequently, we establish the necessary and sufficient condition required for it to fit a given tensor (Section 3.2). For conciseness, the treatment in this section is limited to one-dimensional (sequential) models; see Appendix J.1 for an extension to arbitrary dimensions.

3.1 Tensor Network Equivalent to a Locally Connected Neural Network

Let N ∈ N , and for simplicity suppose that N = 2 L for some L ∈ N . We consider a tensor network with an underlying perfect binary tree graph of height L , which generates W TN ∈ R D 1 ×···× D N . Figure 2(a) provides its diagrammatic definition. For simplicity of presentation, the lengths of axes corresponding to inner (non-open) edges are taken to all be equal to some R ∈ N , referred to as the width of the tensor network.

<latexi sh

1_b

64="7

H

3Q

YgTSWkdq

Z

E

+

>A

C

c

VBN

8

Ur

/o

w

JpKj

P

Ff0I

2

X

RGv mD

M

O

9

y

u

z

5

L

n

<latexi sh

1_b

64="0Yv yKN

/V

oZ9fO

gmnP

Q

R8

>A

C

H

c

BJS

G

3WuI

L

U

E

F

T

D

wj

d

z

5

X

Mp r+

7

q

k

2

<latexi sh

1_b

64="7

H

3Q

YgTSWkdq

Z

E

+

>A

C

c

VBN

8

Ur

/o

w

JpKj

P

Ff0I

2

X

RGv mD

M

O

9

y

u

z

5

L

n

(a)

𝐷

!

𝑅

𝑅

𝐷

"

W

TN

𝐷

#

𝑅

𝑅

𝐷

$

𝐷

%&#

𝑅

𝑅

𝐷

%&"

𝐷

%&!

𝑅

𝑅

𝐷

%

𝑅

𝑅

𝐷

!

(b)

𝐷

"

⊗

N

n

=1

x

(

n

)

,

W

TN

〉

W

TN

𝐷

#

𝐷

#$"

𝐱

(𝟏)

𝐱

(𝟐)

𝐱

(𝑵$𝟏)

𝐱

(𝑵)

As identified by previous works, the tensor network depicted in Figure 2(a) is equivalent to a certain locally connected neural network (with polynomial non-linearity - see, e.g. , [12, 10, 35, 49]). In particular, contracting the tensor network with vectors x (1) ∈ R D 1 , . . . , x ( N ) ∈ R D N , as illustrated in Figure 2(b), can be viewed as a forward pass of the data instance ( x (1) , . . . , x ( N ) ) through a locally connected neural network, whose hidden layers are of width R . This computation results in a scalar equal to 〈 ⊗ N n =1 x ( n ) , W TN 〉 , where ⊗ stands for the outer product. 3 In light of its equivalence to a locally connected neural network, we will refer to the tensor network as a locally connected tensor network . We note that for the equivalent neural network to be practical (in terms of memory and runtime), the width R needs to be of moderate size (typically no more than a few thousands). Specifically, R cannot be exponential in the order N , meaning ln( R ) needs to be much smaller than N .

By virtue of the locally connected tensor network's equivalence to a deep neural network, it has been paramount for the study of expressiveness and generalization in deep learning [12, 9, 10, 13, 14, 54, 34, 35, 3, 30, 31, 36, 47, 48, 49, 50]. Although the equivalent deep neural network (which has polynomial non-linearity) is less common than other neural networks ( e.g. , ones with ReLU non-linearity), it has demonstrated competitive performance in practice [8, 11, 55, 58, 22]. More importantly, its theoretical analyses, through the equivalence to the locally connected tensor network, brought forth numerous insights that were demonstrated empirically and led to development of practical tools for common locally connected architectures. Continuing this line, we will demonstrate our theoretical insights through experiments with widespread convolutional, recurrent and local selfattention architectures (Section 4.3), and employ our theory for deriving an algorithm that enhances the suitability of a data distribution to said architectures (Section 5).

3.2 Necessary and Sufficient Condition for Fitting Tensor

Herein we show that the ability of the locally connected tensor network (defined in Section 3.1) to fit ( i.e. represent) a given tensor is determined by the entanglements that the tensor admits under the below-defined canonical partitions of [ N ] . Note that each canonical partition comprises a subset of contiguous indices, so low entanglement under canonical partitions can be viewed as a formalization of locality.

Definition 2. The canonical partitions of [ N ] = [2 L ] (illustrated in Figure 4 of Appendix B) are:

By appealing to known upper bounds on the entanglements that a given tensor network supports [16, 35], we now establish that if the locally connected tensor network can fit a given tensor, that tensor

must admit low entanglement under the canonical partitions of its axes. Namely, suppose that W TN -the tensor generated by the locally connected tensor network - well-approximates an order N tensor A . Then, Theorem 1 below shows that the entanglement of A with respect to every canonical partition ( K , K c ) ∈ C N cannot be much larger than ln( R ) (recall that R is the width of the locally connected tensor network, and that in practical settings ln( R ) is much smaller than N ), whereas the expected entanglement of a random tensor with respect to ( K , K c ) is on the order of min {|K| , |K c |} (which is linear in N for most canonical partitions).

In the other direction, Theorem 2 below implies that low entanglement under the canonical partitions is not only necessary for a tensor to be fit by the locally connected tensor network, but also sufficient.

Theorem 1. Let W TN ∈ R D 1 ×···× D N be a tensor generated by the locally connected tensor network defined in Section 3.1, and let A ∈ R D 1 ×···× D N . For any glyph[epsilon1] ∈ [0 , ‖A‖ / 4] , if ‖W TN -A‖ ≤ glyph[epsilon1] , then for all canonical partitions ( K , K c ) ∈ C N (Definition 2) it holds that QE ( A ; K ) ≤ ln( R ) + 2 glyph[epsilon1] ‖A‖ · ln( D K ) + 2 √ 2 glyph[epsilon1] ‖A‖ , where D K := min { ∏ n ∈K D n , ∏ n ∈K c D n } . 4 In contrast, a random tensor A ′ ∈ R D 1 ×···× D N , drawn according to the uniform distribution over the set of unit norm tensors, satisfies E [ QE ( A ′ ; K )] ≥ min {|K| , |K c |}· ln(min n ∈ [ N ] D n )+ln ( 1 2 ) -1 2 for all canonical partitions ( K , K c ) ∈ C N .

Proof sketch (proof in Appendix M.2). In general, the entanglements that a tensor network supports can be upper bounded through cuts in its graph [16, 35]. For the locally connected tensor network, these bounds imply that QE ( W TN ; K ) ≤ ln( R ) for any canonical partition ( K , K c ) . The upper bounds on the entanglements of A then follow by showing that if W TN and A are close, then so are their entanglements. The lower bounds on the expected entanglements of a random tensor are derived based on a characterization from [52].

Theorem 2. Let A ∈ R D 1 ×···× D N and glyph[epsilon1] > 0 . Suppose that for all canonical partitions ( K , K c ) ∈ C N (Definition 2) it holds that QE ( A ; K ) ≤ glyph[epsilon1] 2 (2 N -3) ‖A‖ 2 · ln( R ) . 5 Then, there exists an assignment for the tensors constituting the locally connected tensor network (defined in Section 3.1) such that it generates W TN ∈ R D 1 ×···× D N satisfying ‖W TN -A‖ ≤ glyph[epsilon1] .

Proof sketch (proof in Appendix M.3). We show that if A has low entanglement under a canonical partition ( K , K c ) ∈ C N , then the singular values of J A ; K K must decay rapidly (recall that J A ; K K is the arrangement of A as a matrix where rows correspond to axes indexed by K and columns correspond to the remaining axes). The approximation guarantee is then obtained through a construction from [25], which is based on truncated singular value decompositions of every J A ; K K for ( K , K c ) ∈ C N .

4 Low Entanglement Under Canonical Partitions Is Necessary and Sufficient for Accurate Prediction

In this section we consider the locally connected tensor network from Section 3.1 in a machine learning setting. We show that attaining low population loss amounts to fitting a tensor defined by the data distribution, whose axes correspond to features (Section 4.1). Applying the theorems of Section 3.2, we then conclude that the locally connected tensor network is capable of accurate prediction if and only if the data distribution admits low entanglement under canonical partitions of features (Section 4.2). This conclusion is corroborated through experiments, demonstrating that the performance of common locally connected neural networks (including convolutional, recurrent, and local self-attention neural networks) is inversely correlated with the entanglement under canonical partitions of features in the data (Section 4.3). For conciseness, the treatment in this section is limited to one-dimensional (sequential) models and data; see Appendix J.2 for an extension to arbitrary dimensions.

4.1 Accurate Prediction Is Equivalent to Fitting Data Tensor

As discussed in Section 3.1, the locally connected tensor network generating W TN ∈ R D 1 ×···× D N is equivalent to a locally connected neural network, whose forward pass over a data instance ( x (1) , . . . , x ( N ) ) yields 〈 ⊗ N n =1 x ( n ) , W TN 〉 , where x (1) ∈ R D 1 , . . . , x ( N ) ∈ R D N . Motivated by this fact, we consider a binary classification setting, in which the label y of the instance ( x (1) , . . . , x ( N ) ) is either 1 or -1 , and the prediction ˆ y is taken to be the sign of the output of the neural network, i.e. ˆ y = sign (〈 ⊗ N n =1 x ( n ) , W TN 〉) .

Suppose we are given a training set of labeled instances {( ( x (1 ,m ) , . . . , x ( N,m ) ) , y ( m ) )} M m =1 drawn i.i.d. from some distribution, and we would like to learn the parameters of the neural network through the soft-margin support vector machine (SVM) objective, i.e. by optimizing:

for a predetermined constant B > 0 . We assume instances are normalized, i.e. the distribution is such that all vectors constituting an instance have norm no greater than one. We also assume that B ≤ 1 . In this case ∣ ∣ y ( m ) 〈 ⊗ N n =1 x ( n,m ) , W TN 〉∣ ∣ ≤ 1 , so our optimization problem can be expressed as min ‖W TN ‖≤ B 1 -〈D emp , W TN 〉 , where is referred to as the empirical data tensor . This means that the accuracy over the training data is determined by how large the inner product 〈D emp , W TN 〉 is.

Disregarding the degenerate case of D emp = 0 (in which the optimized objective is constant), the inner products 〈 D emp , W TN 〉 and 〈 D emp / ‖D emp ‖ , W TN 〉 differ by only a multiplicative (positive) constant, so fitting the training data amounts to optimizing max ‖W TN ‖≤ B 〈D emp / ‖D emp ‖ , W TN 〉 . If W TN can represent some W , then it can also represent c ·W for every c ∈ R . Thus, we may equivalently optimize max W TN 〈D emp / ‖D emp ‖ , W TN / ‖W TN ‖〉 and multiply the result by B . Fitting the training data therefore boils down to minimizing ∥ ∥ W TN ‖W TN ‖ -D emp ‖D emp ‖ ∥ ∥ . In other words, the accuracy achievable over the training data is determined by the extent to which W TN ‖W TN ‖ can fit the normalized empirical data tensor D emp ‖D emp ‖ .

The arguments above are independent of the training set size M , and in fact apply to the population loss as well, in which case D emp is replaced by the population data tensor :

Disregarding the degenerate case of D pop = 0 ( i.e. that in which the population loss is constant), it follows that the achievable accuracy over the population is determined by the extent to which W TN ‖W TN ‖ can fit the normalized population data tensor D pop ‖D pop ‖ . We refer to the minimal distance from it as the suboptimality in achievable accuracy .

Definition 3. In the context of the classification setting above, the suboptimality in achievable accuracy is SubOpt := min W TN ∥ ∥ ∥ W TN ‖W TN ‖ -D pop ‖D pop ‖ ∥ ∥ ∥ . 4.2 Necessary and Sufficient Condition for Accurate Prediction

In the classification setting of Section 4.1, by invoking Theorems 1 and 2 from Section 3.2, we conclude that the suboptimality in achievable accuracy is small if and only if the population data tensor D pop admits low entanglement under the canonical partitions of its axes (Definition 2). This is formalized in Corollary 1 below. The quantum entanglement of D pop with respect to an arbitrary partition of its axes ( K , K c ) , where K ⊆ [ N ] , is a measure of dependence between the data features indexed by K and those indexed by K c -see Appendix D for intuition behind this. Thus, Corollary 1 implies that the suboptimality in achievable accuracy is small if and only if the data admits low dependence under the canonical partitions of features. Since canonical partitions comprise a subset with contiguous indices, we obtain a formalization of the intuition by which data distributions suitable for locally connected neural networks are those exhibiting a 'local nature.'

Directly evaluating the conditions required by Corollary 1 - low entanglement under canonical partitions for D pop -is impractical, since: (i) D pop is defined via an unknown data distribution

(Equation (3)); and (ii) computing the entanglements involves taking singular value decompositions of matrices with size exponential in the number of input variables N . Fortunately, as Proposition 2 in Appendix E shows, D pop is with high probability well-approximated by the empirical data tensor D emp . Moreover, the entanglement of D emp under any partition can be computed efficiently, without explicitly storing or manipulating an exponentially large matrix - see Appendix F for an algorithm (originally proposed in [40]). Overall, we obtain an efficiently computable criterion (low entanglement under canonical partitions for D emp ), that with high probability is both necessary and sufficient for low suboptimality in achievable accuracy (see Corollary 2 in Appendix E for a formalization).

Corollary 1. Consider the classification setting of Section 4.1, and let glyph[epsilon1] ∈ [0 , 1 / 4] . If there exists a canonical partition ( K , K c ) ∈ C N (Definition 2) under which QE ( D pop ; K ) > ln( R ) + 2 glyph[epsilon1] · ln( D K ) + 2 √ 2 glyph[epsilon1] , where R is the width of the locally connected tensor network and D K := min { ∏ n ∈K D n , ∏ n ∈K c D n } , then SubOpt > glyph[epsilon1] . Conversely, if for all ( K , K c ) ∈ C N it holds that QE ( D pop ; K ) ≤ glyph[epsilon1] 2 8 N -12 · ln( R ) , then SubOpt ≤ glyph[epsilon1] .

Proof sketch (proof in Appendix M.4). The result follows from Theorems 1 and 2 after accounting for the normalization of W TN in the definition of SubOpt (Definition 3).

4.3 Empirical Demonstration

Corollary 2 establishes that, with high probability, the locally connected tensor network (from Section 3.1) can achieve high prediction accuracy if and only if the empirical data tensor (Equation (2)) admits low entanglement under canonical partitions of its axes. We corroborate our formal analysis through experiments, demonstrating that its conclusions carry over to common locally connected architectures. Namely, applying convolutional neural networks, S4 (a popular recurrent neural network; see [26]), and a local self-attention model [46] to different datasets, we show that the achieved test accuracy is inversely correlated with the entanglements of the empirical data tensor under canonical partitions. Below is a description of experiments with one-dimensional ( i.e. sequential) models and data. Additional experiments with two-dimensional (imagery) models and data are given in Appendix J.2.3.

Discerning the relation between entanglements of the empirical data tensor and performance (prediction accuracy) of locally connected neural networks requires datasets admitting different entanglements. A potential way to acquire such datasets is as follows. First, select a dataset on which locally connected neural networks perform well, in the hopes that it admits low entanglement under canonical partitions; natural candidates are datasets comprising images, text or audio. Subsequently, create 'shuffled' variants of the dataset by repeatedly swapping the position of two features chosen at random. 6 This erodes the original arrangement of features in the data, and is expected to yield higher entanglement under canonical partitions.

We followed the blueprint above for a binary classification version of the Speech Commands audio dataset [64]. Figure 3 presents test accuracies achieved by a convolutional neural network, S4, and a local self-attention model, as well as average entanglement under canonical partitions of the empirical data tensor, against the number of random feature swaps performed to create the dataset. As expected, when the number of swaps increases, the average entanglement under canonical partitions becomes higher. At the same time, in accordance with our theory, the prediction accuracies of the locally connected neural networks substantially deteriorate, showing an inverse correlation with the entanglement under canonical partitions.

5 Enhancing Suitability of Data to Locally Connected Neural Networks

Our analysis (Sections 3 and 4) suggests that a data distribution is suitable for locally connected neural networks if and only if it admits low entanglement under canonical partitions of features. Motivated by this observation, we derive a preprocessing algorithm aimed to enhance the suitability of a data distribution to locally connected neural networks (Section 5.1 and Appendix G). Empirical evaluations demonstrate that it significantly improves prediction accuracies of common locally connected neural networks on various datasets (Section 5.2). For conciseness, the treatment in this section is limited to one-dimensional (sequential) models and data; see Appendix J.3 for an extension to arbitrary dimensions.

(∐{}{]̂∐⌈(∑∐√√]√]}{√





⋂˜√√(̂̂√√∐̂glyph[arrowbt]({⌋}

〉{√∐{˜⌈˜⌉˜{√(⊎{̂˜√(

⊕}̂∐⌈∖√√˜{√]}{

/

/

/

/











⊗⊗

⋃





5.1 Search for Feature Arrangement With Low Entanglement Under Canonical Partitions

Our analysis naturally leads to a recipe for enhancing the suitability of a data distribution to locally connected neural networks: given a dataset, search for an arrangement of features which leads to low entanglement under canonical partitions, and then arrange the features accordingly. Formally, suppose we have M ∈ N training instances {( ( x (1 ,m ) , . . . , x ( N,m ) ) , y ( m ) )} M m =1 , where y ( m ) ∈ { 1 , -1 } and x ( n,m ) ∈ R D for n ∈ [ N ] , m ∈ [ M ] , with D ∈ N . Assume without loss of generality that N is a power of two (if this is not the case we may add constant features as needed). The aforementioned recipe boils down to a search for a permutation π : [ N ] → [ N ] , which when applied to feature indices leads the empirical data tensor D emp (Equation (2)) to admit low entanglement under the canonical partitions of its axes (Definition 2).

A greedy realization of the foregoing search is as follows. Initially, partition the features into two equally sized sets K 1 , 1 ⊂ [ N ] and K 1 , 2 := [ N ] \ K 1 , 1 such that the entanglement of D emp with respect to ( K 1 , 1 , K 1 , 2 ) is minimal. That is, find K 1 , 1 ∈ argmin K⊂ [ N ] , |K| = N/ 2 QE ( D emp ; K ) . The permutation π will map K 1 , 1 to coordinates { 1 , . . . , N 2 } and K 1 , 2 to { N 2 +1 , . . . , N } . Then, partition K 1 , 1 into two equally sized sets K 2 , 1 ⊂ K 1 , 1 and K 2 , 2 := K 1 , 1 \ K 2 , 1 such that the average of entanglements induced by these sets is minimal, i.e. K 2 , 1 ∈ argmin K⊂K 1 , 1 , |K| = |K 1 , 1 | / 2 1 2 [ QE ( D emp ; K )+ QE ( D emp ; K 1 , 1 \ K ) ] . The permutation π will map K 2 , 1 to coordinates { 1 , . . . , N 4 } and K 2 , 2 to { N 4 +1 , . . . , N 2 } . A partition of K 1 , 2 into two equally sized sets K 2 , 3 and K 2 , 4 is obtained similarly, where π will map K 2 , 3 to coordinates { N 2 +1 , . . . , 3 N 4 } and K 2 , 4 to { 3 N 4 +1 , . . . , N } . Continuing in the same fashion, until we reach subsets K L, 1 , . . . , K L,N consisting of a single feature index each, fully specifies the permutation π .

Unfortunately, the step lying at the heart of the above scheme - finding a balanced partition that minimizes average entanglement - is computationally prohibitive, and we are not aware of any tools that alleviate the computational difficulty. However, as discussed in Appendix G, if one replaces entanglement with an appropriate surrogate measure, then each search for a balanced partition minimizing average entanglement converts into a minimum balanced cut problem , which enjoys a wide array of established approximation tools [29]. We thus obtain a practical algorithm for enhancing the suitability of a data distribution to locally connected neural networks.

5.2 Experiments

We empirically evaluate our feature rearrangement method, detailed in Appendix G, using common locally connected neural networks - a convolutional neural network, S4 (popular recurrent neural network; see [26]), and a local self-attention model [46] - over randomly permuted audio datasets (Section 5.2.1) and several tabular datasets (Section 5.2.2). For brevity, we defer some experiments and implementation details to Appendices K and L. Additional experiments with two-dimensional data are given in Appendix J.3.3.

5.2.1 Randomly Permuted Audio Datasets

Section 4.3 demonstrated that audio data admits low entanglement under canonical partitions of features, and that randomly permuting the position of features leads this entanglement to increase,

while substantially degrading the prediction accuracy of locally connected neural networks. A sensible test for our method is to evaluate its ability to recover performance lost due to the random permutation of features.

For the Speech Commands dataset [64], Table 1 compares prediction accuracies of locally connected neural networks on the data: (i) subject to a random permutation of features; (ii) attained after rearranging the randomly permuted features via our method; and (iii) attained after rearranging the randomly permuted features via IGTD [67] - a heuristic scheme designed for convolutional neural networks (see Appendix A). As can be seen, our method leads to significant improvements, surpassing those brought forth by IGTD. Note that the performance lost due to the random permutation of features is not entirely recovered. 7 We believe this relates to phenomena outside the scope of the theory underlying our method (Sections 3 and 4), for example translation invariance in data being beneficial in terms of generalization. Investigation of such phenomena and suitable modification of our method are regarded as promising directions for future work.

The number of features in the audio dataset used in the experiment above is 2048. We demonstrate the scalability of Algorithm 2 by including in Appendix K an experiment over audio data with 50,000 features. In this experiment, instances of the minimum balanced cut problem encountered in Algorithm 2 entail graphs with up to 25 · 10 8 edges. They are solved using the well-known edge sparsification algorithm of [57] that preserves weights of cuts, allowing for configurable compute and memory consumption (the more resources are consumed, the more accurate the solution will be).

5.2.2 Tabular Datasets

The prediction accuracies of locally connected neural networks on tabular data, i.e. on data in which features are arranged arbitrarily, is known to be subpar [56]. Tables 2 and 5 report results of experiments with locally connected neural networks over standard tabular benchmarks (namely 'semeion', 'isolet' and 'dna' [60]), demonstrating that arranging features via our method leads to significant improvements in prediction accuracies, surpassing improvements brought forth by IGTD (a heuristic scheme designed for convolutional neural networks [67]). Note that our method does not lead to state of the art prediction accuracies on the evaluated benchmarks. 8 However, the results suggest that it renders locally connected neural networks a viable option for tabular data. This option is particularly appealing in when the number of features is large settings, where many alternative approaches ( e.g. ones involving fully connected neural networks) are impractical.

6 Conclusion

6.1 Summary

The question of what makes a data distribution suitable for deep learning is a fundamental open problem. Focusing on locally connected neural networks - a prevalent family of deep learning architectures that includes as special cases convolutional neural networks, recurrent neural networks (in particular the recent S4 models) and local self-attention models - we address this problem by

adopting theoretical tools from quantum physics. Our main theoretical result states that a certain locally connected neural network is capable of accurate prediction ( i.e. can express a solution with low population loss) over a data distribution if and only if the data distribution admits low quantum entanglement under certain canonical partitions of features. Experiments with widespread locally connected neural networks corroborate this finding.

Our theory suggests that the suitability of a data distribution to locally connected neural networks may be enhanced by arranging features such that low entanglement under canonical partitions is attained. Employing a certain surrogate for entanglement, we show that this arrangement can be implemented efficiently, and that it leads to substantial improvements in the prediction accuracies of common locally connected neural networks on various datasets.

6.2 Limitations and Future Work

Neural network architecture We theoretically analyzed a locally connected neural network with polynomial non-linearity, by employing its equivalence to a tensor network ( cf. Section 3.1). Accounting for neural networks with connectivities beyond those considered ( e.g. connectivities that are non-local or ones involving skip connections) is an interesting topic for future work. It requires modification of the equivalent tensor network and corresponding modification of the definition of canonical partitions, similarly to the analysis in Appendix J. Another valuable direction is to account for neural networks with other non-linearities, e.g. ReLU. We believe this may be achieved through a generalized notion of tensor networks, successfully used in past work to analyze such architectures [9].

Objective function The analysis in Section 4 assumes a binary soft-margin SVM objective. Extending it to other objective functions, e.g. multi-class SVM, may shed light on the relation between the objective and the requirements for a data distribution to be suitable to neural networks.

Textual data Our experiments (in Section 4.3 and Appendix J.2.3) show that the necessary and sufficient condition we derived for a data distribution to be suitable to a locally connected neural network - namely, low quantum entanglement under canonical partitions of features - is upheld by audio and image datasets. This falls in line with the excellent performance of locally connected neural networks over these data modalities. In contrast, high performant architectures for textual data are typically non-local [61]. Investigating the quantum entanglements that textual data admits and, in particular, under which partitions they are low, may allow designing more efficient architectures with connectivity tailored to textual data.

6.3 Outlook

The data modalities to which deep learning is most commonly applied - namely ones involving images, text and audio - are often regarded as natural (as opposed to, for example, tabular data fusing heterogeneous information). We believe the difficulty in explaining the suitability of such modalities to deep learning may be due to a shortage in tools for formally reasoning about natural data. Concepts and tools from physics - a branch of science concerned with formally reasoning about natural phenomena - may be key to overcoming said difficulty. We hope that our use of quantum entanglement will encourage further research along this line.

Acknowledgments and Disclosure of Funding

This work was supported by a Google Research Scholar Award, a Google Research Gift, the Yandex Initiative in Machine Learning, the Israel Science Foundation (grant 1780/21), the Tel Aviv University Center for AI and Data Science, the Adelis Research Fund for Artificial Intelligence, Len Blavatnik and the Blavatnik Family Foundation, and Amnon and Anat Shashua. NR is supported by the Apple Scholars in AI/ML PhD fellowship.

References

Emmanuel Abbe and Colin Sandon. Provable limitations of deep learning. arXiv preprint arXiv:1812.06369 , 2018.

Yotam Alexander, Nimrod De La Vega, Noam Razin, and Nadav Cohen. What makes data suitable for a locally connected neural network? a necessary and sufficient condition based on quantum entanglement. arXiv preprint arXiv:2303.11249 , 2023.

Emilio Rafael Balda, Arash Behboodi, and Rudolf Mathar. A tensor analysis on dense connectivity via convolutional arithmetic circuits. Preprint , 2018.

Rajendra Bhatia. Matrix analysis . Springer-Verlag, 1997.

Alon Brutzkus and Amir Globerson. Globally optimal gradient descent for a convnet with gaussian inputs. In International Conference on Machine Learning (ICML) , pages 605-614, 2017.

Alon Brutzkus, Amir Globerson, Eran Malach, Alon Regev Netser, and Shai Shalev-Schwartz. Efficient learning of CNNs using patch based features. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 2336-2356. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/brutzkus22a.html .

Song Cheng, Jing Chen, and Lei Wang. Information perspective to probabilistic modeling: Boltzmann machines versus born machines. Entropy , 20(8):583, 2018.

Nadav Cohen and Amnon Shashua. Simnets: A generalization of convolutional networks. Advances in Neural Information Processing Systems (NeurIPS), Deep Learning Workshop , 2014.

Nadav Cohen and Amnon Shashua. Convolutional rectifier networks as generalized tensor decompositions. International Conference on Machine Learning (ICML) , 2016.

Nadav Cohen and Amnon Shashua. Inductive bias of deep convolutional networks through pooling geometry. International Conference on Learning Representations (ICLR) , 2017.

Nadav Cohen, Or Sharir, and Amnon Shashua. Deep simnets. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

Nadav Cohen, Or Sharir, and Amnon Shashua. On the expressive power of deep learning: A tensor analysis. Conference On Learning Theory (COLT) , 2016.

Nadav Cohen, Or Sharir, Yoav Levine, Ronen Tamari, David Yakira, and Amnon Shashua. Analysis and design of convolutional networks via hierarchical tensor decompositions. Intel Collaborative Research Institute for Computational Intelligence (ICRI-CI) Special Issue on Deep Learning Theory , 2017.

Nadav Cohen, Ronen Tamari, and Amnon Shashua. Boosting dilated convolutional networks with mixed tensor decompositions. International Conference on Learning Representations (ICLR) , 2018.

Ian Convy, William Huggins, Haoran Liao, and K Birgitta Whaley. Mutual information scaling for tensor network machine learning. Machine learning: science and technology , 3(1):015017, 2022.

Shawn X Cui, Michael H Freedman, Or Sattath, Richard Stong, and Greg Minton. Quantum max-flow/mincut. Journal of Mathematical Physics , 57(6):062206, 2016.

Wei Dai, Chia Dai, Shuhui Qu, Juncheng Li, and Samarjit Das. Very deep convolutional neural networks for raw waveforms. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages 421-425. IEEE, 2017.

Persi Diaconis and Mehrdad Shahshahani. Generating a random permutation with random transpositions. Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete , 57(2):159-179, 1981.

Simon S Du and Surbhi Goel. Improved learning of one-hidden-layer convolutional neural networks with overlaps. arXiv preprint arXiv:1805.07798 , 2018.

Simon S Du, Jason D Lee, Yuandong Tian, Aarti Singh, and Barnabas Poczos. Gradient descent learns one-hidden-layer cnn: Don't be afraid of spurious local minima. In International Conference on Machine Learning (ICML) , pages 1339-1348, 2018.

Anatoly Dymarsky and Kirill Pavlenko. Tensor network to learn the wave function of data. Physical Review Research , 4(4):043111, 2022.

Timo Felser, Marco Trenti, Lorenzo Sestini, Alessio Gianelle, Davide Zuliani, Donatella Lucchesi, and Simone Montangero. Quantum-inspired machine learning on high-energy physics data. npj Quantum Information , 7(1):1-8, 2021.

Michael R Garey and David S Johnson. Computers and intractability. A Guide to the Theory of NPCompleteness , 1979.

Rong Ge, Yunwei Ren, Xiang Wang, and Mo Zhou. Understanding deflation process in over-parametrized tensor decomposition. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Lars Grasedyck. Hierarchical singular value decomposition of tensors. SIAM journal on matrix analysis and applications , 31(4):2029-2054, 2010.

Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. International Conference on Learning Representations (ICLR) , 2022.

Siu-Wai Ho and Raymond W Yeung. The interplay between entropy and variational distance. IEEE Transactions on Information Theory , 56(12):5906-5929, 2010.

Zhih-Ahn Jia, Lu Wei, Yu-Chun Wu, Guang-Can Guo, and Guo-Ping Guo. Entanglement area law for shallow and deep quantum neural network states. New Journal of Physics , 22(5):053022, 2020.

George Karypis and Vipin Kumar. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM Journal on scientific Computing , 20(1):359-392, 1998.

Valentin Khrulkov, Alexander Novikov, and Ivan Oseledets. Expressive power of recurrent neural networks. International Conference on Learning Representations (ICLR) , 2018.

Valentin Khrulkov, Oleksii Hrinchuk, and Ivan Oseledets. Generalized tensor models for recurrent neural networks. International Conference on Learning Representations (ICLR) , 2019.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.

Yoav Levine, Or Sharir, and Amnon Shashua. Benefits of depth for long-term memory of recurrent networks. International Conference on Learning Representations (ICLR) Workshop , 2018.

Yoav Levine, David Yakira, Nadav Cohen, and Amnon Shashua. Deep learning and quantum entanglement: Fundamental connections with implications to network design. International Conference on Learning Representations (ICLR) , 2018.

Yoav Levine, Or Sharir, Nadav Cohen, and Amnon Shashua. Quantum entanglement in deep learning architectures. To appear in Physical Review Letters , 2019.

Jingling Li, Yanchao Sun, Jiahao Su, Taiji Suzuki, and Furong Huang. Understanding generalization in deep learning via tensor methods. In International Conference on Artificial Intelligence and Statistics , pages 504-515. PMLR, 2020.

Sirui Lu, Márton Kanász-Nagy, Ivan Kukuljan, and J Ignacio Cirac. Tensor networks and efficient descriptions of classical data. arXiv preprint arXiv:2103.06872 , 2021.

Eran Malach and Shai Shalev-Shwartz. A provably correct algorithm for deep learning that actually works. arXiv preprint arXiv:1803.09522 , 2018.

John Martyn, Guifre Vidal, Chase Roberts, and Stefan Leichenauer. Entanglement and tensor networks for supervised image classification. arXiv preprint arXiv:2007.06082 , 2020.

Kento Nozawa and Issei Sato. Empirical evaluation and theoretical analysis for representation learning: A survey. arXiv preprint arXiv:2204.08226 , 2022.

Román Orús. A practical introduction to tensor networks: Matrix product states and projected entangled pair states. Annals of physics , 349:117-158, 2014.

Samet Oymak and Mahdi Soltanolkotabi. End-to-end learning of a convolutional neural network via deep tensor decomposition. arXiv preprint arXiv:1805.06523 , 2018.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W , 2017.

Giovanni Puccetti. Measuring linear correlation between random vectors. Available at SSRN 3116066 , 2022.

Jack Rae and Ali Razavi. Do transformers need deep long-range memory? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Online, July 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/2020.acl-main.672 .

Noam Razin and Nadav Cohen. Implicit regularization in deep learning may not be explainable by norms. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Noam Razin, Asaf Maman, and Nadav Cohen. Implicit regularization in tensor factorization. International Conference on Machine Learning (ICML) , 2021.

Noam Razin, Asaf Maman, and Nadav Cohen. Implicit regularization in hierarchical tensor factorization and deep convolutional neural networks. International Conference on Machine Learning (ICML) , 2022.

Noam Razin, Tom Verbin, and Nadav Cohen. On the ability of graph neural networks to model interactions between vertices. arXiv preprint arXiv:2211.16494 , 2022.

Lorenzo Rosasco, Mikhail Belkin, and Ernesto De Vito. On learning with integral operators. J. Mach. Learn. Res. , 11:905-934, mar 2010. ISSN 1532-4435.

Siddhartha Sen. Average entropy of a quantum subsystem. Physical review letters , 77(1):1, 1996.

Shai Shalev-Shwartz, Ohad Shamir, and Shaked Shammah. Failures of gradient-based deep learning. In International Conference on Machine Learning , pages 3067-3075. PMLR, 2017.

Or Sharir and Amnon Shashua. On the expressive power of overlapping architectures of deep learning. International Conference on Learning Representations (ICLR) , 2018.

Or Sharir, Ronen Tamari, Nadav Cohen, and Amnon Shashua. Tensorial mixture models. arXiv preprint arXiv:1610.04167 , 2016.

Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion , 81:84-90, 2022.

Daniel A Spielman and Shang-Hua Teng. Spectral sparsification of graphs. SIAM Journal on Computing , 40(4):981-1025, 2011.

E Miles Stoudenmire. Learning relevant features of data with multi-scale tensor networks. Quantum Science and Technology , 3(3):034003, 2018.

Edwin Stoudenmire and David J Schwab. Supervised learning with tensor networks. In Advances in Neural Information Processing Systems (NeurIPS) , 2016.

Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. Openml: networked science in machine learning. SIGKDD Explorations , 15(2):49-60, 2013. doi: 10.1145/2641190.2641198. URL http://doi.acm.org/10.1145/2641190.264119 .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Qingcan Wang et al. Exponential convergence of the deep neural network approximation for analytic functions. arXiv preprint arXiv:1807.00297 , 2018.

Xiang Wang, Chenwei Wu, Jason D Lee, Tengyu Ma, and Rong Ge. Beyond lazy training for overparameterized tensor decomposition. Advances in Neural Information Processing Systems , 33:2193421944, 2020.

Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209 , 2018.

Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks , 94:103-114, 2017.

Ya-Hui Zhang. Entanglement entropy of target functions for image classification and convolutional neural network. arXiv preprint arXiv:1710.05520 , 2017.

Yitan Zhu, Thomas Brettin, Fangfang Xia, Alexander Partin, Maulik Shukla, Hyunseung Yoo, Yvonne A Evrard, James H Doroshow, and Rick L Stevens. Converting tabular data into images for deep learning with convolutional neural networks. Scientific reports , 11(1):1-11, 2021.

A Related Work

Characterizing formal properties of data distributions that make them suitable for deep learning is a major open problem in the field. A number of papers provide sufficient conditions on a data distribution which imply that it is learnable by certain neural networks [5, 65, 39, 19, 20, 43, 62, 6]. However, these sufficient conditions are restrictive, and are not argued to be necessary for any aspect of learning ( i.e. for expressiveness, optimization or generalization). To the best of our knowledge, this paper is the first to derive a verifiable condition on a data distribution that is both necessary and sufficient for aspects of learning to be achievable by a neural network. We note that the condition we derive resembles past attempts to quantify the structure of data via quantum entanglement and mutual information [40, 15, 38, 7, 66, 28, 21]. However, such quantifications have not been formally related to learnability by neural networks.

The current paper follows a long line of research employing tensor networks as theoretical models for studying deep learning. This line includes works analyzing the expressiveness of different neural network architectures [12, 55, 9, 10, 54, 14, 34, 3, 35, 30, 36, 31, 50], their generalization [37], and the implicit regularization induced by their optimization [47, 48, 49, 63, 24]. Similarly to prior works we focus on expressiveness, yet our approach differs in that we incorporate the data distribution into the analysis and tackle the question of what makes data suitable for deep learning.

The algorithm we propose for enhancing the suitability of data to locally connected neural networks can be considered a form of representation learning. Representation learning is a vast field, far too broad to survey here (for an overview see [41]). Our algorithm, which learns a representation via rearrangement of features in the data, is complementary to most representation learning methods in the literature. A notable method that is also based on feature rearrangement is IGTD [67] - a heuristic scheme designed for convolutional neural networks. In contrast to IGTD, our algorithm is theoretically grounded. Moreover, we demonstrate empirically in Section 5 that it leads to higher prediction accuracies.

B Illustration of Canonical Partitions

𝑙 = 1

𝑙 = 2

𝑙 = 𝐿

𝑙 = 0

K

∈

∈

K

c

C Impossibility Result for Improving the Sufficient Condition in Theorem 2

The sufficient condition in Theorem 2 (from Section 3.2) for approximating A ∈ R D 1 ×···× D N requires entanglements to approach zero as the desired approximation error glyph[epsilon1] does, in contrast to the necessary condition in Theorem 1 where they approach ln( R ) . As Proposition 1 shows, this is unavoidable in the absence of further knowledge regarding A . However, if for all canonical partitions ( K , K c ) ∈ C N the singular values of J A ; K K trailing after the R 'th one are small, then we can also guarantee an assignment for the locally connected tensor network satisfying ‖W TN -A‖ ≤ glyph[epsilon1] , while QE ( A ; K ) can be on the order of ln( R ) for all ( K , K c ) ∈ C N . Indeed, this follows directly from a result in [25], which we restate as Lemma 7 for convenience.

Proposition 1. Let f : R 2 → R ≥ 0 be monotonically increasing in its second variable. Suppose that the following statement holds: if a tensor A ∈ R D 1 ×···× D N satisfies QE ( A ; K ) ≤ f ( ‖A‖ , glyph[epsilon1] ) for all canonical partitions ( K , K c ) ∈ C N (Definition 2), then there exists an assignment for the tensors constituting the locally connected tensor network (defined in Section 3.1) for which W TN ∈

R D 1 ×···× D N upholds:

Then, for any A ∈ R D 1 ×···× D N , as the desired approximation error glyph[epsilon1] goes to zero, so does the sufficient condition on entanglements, i.e.:

Proof. Suppose otherwise, i.e. that there exists some a > 0 such that lim glyph[epsilon1] → 0 f ( a, glyph[epsilon1] ) = inf glyph[epsilon1]> 0 f ( a, glyph[epsilon1] ) = c > 0 . Let A be a tensor with ‖A‖ = a such that QE ( A ; K ) < c for all canonical partitions ( K , K c ) ∈ C N . Then by assumption, for all glyph[epsilon1] > 0 there exist a tensor generated by the locally connected tensor network, which we denote by W TN ( glyph[epsilon1] ) ∈ R D 1 ×···× D N , that satisfies:

By Lemma 5, for all glyph[epsilon1] > 0 we have that

J K so by the lower semicontinuity of the matrix rank, we have that rank( J A ; K K )) ≤ R as well. But, there are tensors for which this leads to a contradiction, namely tensors with arbitrary low entanglement across all partitions but nearly maximal rank. Indeed, consider tensors of the form

∥ ∥ ∥ ∥ Q ( δ ) a -⊗ N n =1 v ( n ) ‖⊗ N n =1 v ( n ) ‖ ∥ ∥ ∥ ∥ ≤ δ · ‖B‖ ‖⊗ N n =1 v ( n ) + δ · B‖ + ∥ ∥ ∥ ∥ ⊗ N n =1 v ( n ) ‖⊗ N n =1 v ( n ) + δ · B‖ -⊗ N n =1 v ( n ) ‖⊗ N n =1 v ( n ) ‖ ∥ ∥ ∥ ∥ , and so lim δ → 0 ∥ ∥ ∥ ∥ Q ( δ ) a -⊗ N n =1 v ( n ) ‖⊗ N n =1 v ( n ) ‖ ∥ ∥ ∥ ∥ = 0 . Thus, from Lemma 9 we know that where { v ( n ) ∈ R D n } N n =1 are (non-zero) vectors, δ > 0 and B ∈ R D 1 ×···× D N is some tensor with matricizations of maximal rank across all canonical partitions (for a proof of the existence of such a tensor see Claim 3 in [35]). Note that ‖Q ( δ ) ‖ = a for all δ > 0 . By the triangle inequality have

for all ( K , K c ) ∈ C N . Furthermore, for any ( K , K c ) ∈ C N :

and therefore by Theorem 1 for sufficiently small δ > 0 we have

for all ( K , K c ) ∈ C N . However, rank( J Q ( δ ); K K ) is (nearly) maximal for all canonical partitions. Indeed, by the triangle inequality for matrix rank we get

D Quantum Entanglement as a Measure of Dependence Between Features

In this appendix, we provide intuition behind the following statement made in Section 4.2: the quantum entanglement of D pop with respect to an arbitrary partition of its axes ( K , K c ) , where K ⊆ [ N ] , is a measure of dependence between the data features indexed by K and those indexed by K c . Consider the case where the features indexed by K are statistically independent of those indexed by K c , and all features are statistically independent of the label y . Then QE ( D pop ; K ) = 0 (see Lemma 1 below for proof of this fact). If the features are not statistically independent of y , but when conditioned on y the features indexed by K are statistically independent of those indexed by K c , then QE ( D pop ; K ) ≤ ln(2) (this fact is also established by Lemma 1 below). In general, the higher QE ( D pop ; K ) is, the farther the distribution is from the aforementioned situations of independence, implying stronger dependence between the features indexed by K and those indexed by K c .

Lemma 1. Consider the classification setting of Section 4.1, and let K ⊆ [ N ] . If the features indexed by K are independent of those indexed by K c (i.e. ( x ( n ) ) n ∈K are independent of ( x ( n ) ) n ∈K c ), conditioned on the label y , then QE ( D pop ; K ) ≤ ln(2) . Furthermore, if the features are also independent of y , then QE ( D pop ; K ) = 0 .

Proof. For brevity, denote X := ( x (1) , . . . , x ( N ) ) , X K := ( x ( n ) ) n ∈K , and X K c := ( x ( n ) ) n ∈K c . By the law of total expectation, the entry of D pop corresponding to an index tuple ( i 1 , . . . , i N ) ∈ [ D 1 ] ×··· × [ D N ] satisfies:

Thus, if the features indexed by K are independent of those indexed by K c given y , it holds that:

This implies that we can write J D pop ; K K ∈ R ∏ n ∈K D n × ∏ n ∈K c D n -the arrangement of D pop as a matrix whose rows correspond to axes indexed by K and columns correspond to the remaining axes - as a weighted sum of two outer products. Specifically, define v K , u K ∈ R ∏ n ∈K D n to be the vectors holding for each possible indexing tuple ( i n ) n ∈K the values E X K [∏ n ∈K x ( n ) i n | y = 1 ] and E X K [∏ n ∈K x ( n ) i n | y = -1 ] , respectively (with the arrangement of entries in the vectors being consistent with the rows of J D pop ; K K ). Furthermore, let v K c , u K c ∈ R ∏ n ∈K c D n be defined analogously by replacing K with K c (with the arrangement of entries in the vectors being consistent with the columns of J D pop ; K K ). Then, by Equation (4):

Since each outer product forms a rank one matrix, the subaddititivty of rank implies that the rank of J D pop ; K K is at most two. As a result, J D pop ; K K has at most two non-zero singular values and QE ( D pop ; K ) ≤ ln(2) (the entropy of a distribution is at most the natural logarithm of the size of its support).

If, in addition, all features in the data are independent of the label y , then v K = u K and v K c = u K c , meaning J D pop ; K K = ( P ( y = 1) -P ( y = -1)) · v K ⊗ v K c . In this case, the rank of J D pop ; K K is at most one, so QE ( D pop ; K ) = 0 .

E Efficiently Computable Criterion for Low Suboptimality in Achievable Accuracy

In this appendix, we provide the formal claims deferred from Section 4.2. Specifically, Proposition 2 shows that D pop / ‖D pop ‖ is, with high probability, well-approximated by the normalized empirical data tensor D emp / ‖D emp ‖ . Corollary 2 establishes the efficiently computable criterion (low entanglement under canonical partitions for D emp ), that with high probability is both necessary and sufficient for low suboptimality in achievable accuracy.

Proposition 2. Consider the classification setting of Section 4.1, and let δ ∈ (0 , 1) and γ > 0 . If the training set size M satisfies M ≥ 2 ln( 2 δ ) ‖D pop ‖ 2 γ 2 , then with probability at least 1 -δ :

Proof sketch (proof in Appendix M.5). A standard generalization of the Hoeffding inequality to random vectors in a Hilbert space yields a high probability bound on ‖D emp -D pop ‖ , which is then translated to a bound on the normalized tensors.

Algorithm 1 Entanglement Computation for the Empirical Data Tensor

1: Input: X := { ( x (1 ,m ) , . . . , x ( N,m ) ) } M m =1 -M ∈ N data instances comprising N ∈ N features each, K ⊆ [ N ] -subset of feature indices

2: Output: QE ( D emp ; K )

3: Compute G ( K ) , G ( K c ) ∈ R M × M given element-wise by:

4: Compute eigenvalue decompositions of G ( K ) and G ( K c ) , i.e. :

where U ( K ) , U ( K c ) ∈ R M × M are orthogonal matrices and S ( K ) , S ( K c ) ∈ R M × M are diagonal holding the eigenvalues of G ( K ) and G ( K c ) , respectively

5: Compute Q = ( S ( K ) ) 1 2 ( U ( K ) ) glyph[latticetop] U ( K c ) ( S ( K c ) ) 1 2 ∈ R M × M

6: Compute a singular value decomposition of Q to obtain its singular values σ 1 ( Q ) , . . . , σ M ( Q )

7: Let ρ m := σ 2 m ( Q ) / ∑ M m ′ =1 σ 2 m ′ ( Q ) for m ∈ [ M ]

8: return QE ( D emp ; K ) = -∑ M m =1 ρ m ln( ρ m ) (if Q = 0 , then return 0 )

Corollary 2. Consider the setting and notation of Corollary 1, with glyph[epsilon1] ∈ (0 , 1 / 6] . For δ ∈ (0 , 1) , suppose that the training set size M satisfies M ≥ 8 ln( 2 δ ) ‖D pop ‖ 2 glyph[epsilon1] 2 . Then, with probability at least 1 -δ the following hold. First, if there exists a canonical partition ( K , K c ) ∈ C N (Definition 2) under which QE ( D emp ; K ) > ln( R ) + 3 glyph[epsilon1] · ln( D K ) + 2 √ 3 glyph[epsilon1] , then:

Second, if for all ( , c ) N it holds that QE ( emp ; ) glyph[epsilon1] 2 ln( R ) , then:

Moreover, the conditions above on the entanglements of D emp can be evaluated efficiently (in O ( DNM 2 + NM 3 ) time O ( DNM + M 2 ) and memory, where D := max n ∈ [ N ] D n ).

Proof sketch (proof in Appendix M.6). Implied by Corollary 1, Proposition 2 with γ = glyph[epsilon1] 2 and Algorithm 1 in Appendix F.

F Efficiently Computing Entanglements of the Empirical Data Tensor

For a given tensor, its entanglement with respect to a partition of axes (Definition 1) is determined by the singular values of its arrangement as a matrix according to the partition. Since the empirical data tensor D emp (Equation (2)) has size exponential in the number of features N , it is infeasible to naively compute its entanglement (or even explicitly store it in memory). Fortunately, as shown in [40], the specific form of the empirical data tensor admits an efficient algorithm for computing entanglements, without explicitly manipulating an exponentially large matrix. Specifically, the algorithm runs in O ( DNM 2 + M 3 ) time and requires O ( DNM + M 2 ) memory, where D := max n ∈ [ N ] D n is the maximal feature dimension ( i.e. axis length of D emp ), N is the number of features in the data ( i.e. number of axes that D emp has), and M is the number of training instances. For completeness, we outline the method in Algorithm 1 while referring the interested reader to Appendix A in [40] for further details.

G Practical Algorithm via Surrogate for Entanglement

To efficiently implement the scheme from Section 5.1, we replace entanglement with a surrogate measure of dependence. The surrogate is based on the Pearson correlation coefficient for multivariate features [45], 9 and its agreement with entanglement is demonstrated empirically in Appendix I. Theoretically supporting this agreement is left for future work.

Definition 4. Given a set of M ∈ N instances X := { ( x (1 ,m ) , . . . , x ( N,m ) ) ∈ ( R D ) N } M m =1 , denote by p n,n ′ the multivariate Pearson correlation between features n, n ′ ∈ [ N ] . For K ⊆ [ N ] , the surrogate entanglement of X with respect to the partition ( K , K c ) , denoted SE ( X ; K ) , is the sum of absolute values of Pearson correlation coefficients between pairs of features, the first belonging to K and the second to K c := [ N ] \ K . That is, SE ( X ; K ) := ∑ n ∈K ,n ′ ∈K c | p n,n ′ | .

As shown by Proposition 3 below, replacing entanglement with surrogate entanglement in the scheme from Section 5.1 converts each search for a balanced partition minimizing average entanglement into a minimum balanced cut problem . Although the minimum balanced cut problem is NP-hard (see, e.g. , [23]), it enjoys a wide array of well-established approximation tools, particularly ones designed for large scale [29, 57]. We therefore obtain a practical algorithm for enhancing the suitability of a data distribution to locally connected neural networks - see Algorithm 2.

Proposition 3. For any s K ⊆ [ N ] of even size, the following optimization problem can be framed as a minimum balanced cut problem over a complete graph with | s K| vertices:

Specifically, there exists a complete undirected weighted graph with vertices s K and edge weights w : s K × s K → R such that for any K ⊂ s K , the weight of the cut in the graph induced by K -∑ n ∈K ,n ′ ∈ s K\K w ( { n, n ′ } ) -is equal, up to an additive constant, to the term minimized in Equation (5) , i.e. to 1 2 [ SE ( X ; K ) + SE ( X ; s K\K )] .

Proof. Consider the complete undirected graph whose vertices are s K and where the weight of an edge { n, n ′ } ∈ s K× s K is w ( { n, n ′ } ) = | p n,n ′ | (recall that p n,n ′ stands for the multivariate Pearson correlation between features n and n ′ in X ). For any K ⊂ s K it holds that:

where 1 2 SE ( X ; s K ) does not depend on K . This concludes the proof.

H Definition of the Multivariate Pearson Correlation from [45]

Appendix G introduces a surrogate measure for entanglement based on the multivariate Pearson correlation from [45]. For completeness, this appendix provides its formal definition.

Given a set of M ∈ N instances X := { ( x (1 ,m ) , . . . , x ( N,m ) ) ∈ ( R D ) N } M m =1 , let Σ ( n ) be the empirical covariance matrix of feature n ∈ [ N ] and Σ ( n,n ′ ) be the empirical cross-covariance matrix of features n, n ′ ∈ [ N ] , i.e. :

where µ ( n ) := 1 M ∑ M m =1 x ( n,m ) for n ∈ [ N ] . With this notation, the multivariate Pearson correlation of n, n ′ ∈ [ N ] from [45] is defined by p n,n ′ := trace ( Σ ( n,n ′ ) ) / trace ( (Σ ( n ) Σ ( n ′ ) ) 1 / 2 ) .

Algorithm 2 Enhancing Suitability of Data to Locally Connected Neural Networks

1: Input: X := { ( x (1 ,m ) , . . . , x ( N,m ) ) } M m =1 -M ∈ N data instances comprising N ∈ N features

2: Output: Permutation π : [ N ] → [ N ] to apply to feature indices

3: Let K 0 , 1 := [ N ] and denote L := log 2 ( N )

4: # We assume for simplicity that N is a power of two, otherwise one may add constant features

5: for l = 0 , . . . , L -1 , n = 1 , . . . , 2 l do

6: Using a reduction to a minimum balanced cut problem (Proposition 3), find an approximate solution K l +1 , 2 n -1 ⊂ K l,n for:

7: Let K l +1 , 2 n := K l,n \ K l +1 , 2 n -1

8: end for

9: # At this point, K L, 1 , . . . , K L,N each contain a single feature index

10: return π that maps k ∈ K L,n to n , for every n ∈ [ N ]

I Entanglement and Surrogate Entanglement Are Strongly Correlated

In Appendix G, we introduced a surrogate entanglement measure (Definition 4) to facilitate efficient implementation of the feature arrangement search scheme from Section 5.1. Figure 5 supports the viability of the chosen surrogate, demonstrating empirically that it is strongly correlated with entanglement of the empirical data tensor (Definition 1 and Equation (2)).

0

500

1000

1500

2000

2500

3000

3500

4000

Number of Feature Swaps in Data

0.4

0.6

0.8

1.0

Entanglement Under

Canonical Partitions

Entanglement

Surrogate Entanglement

5.2e+04

5.5e+04

5.8e+04

6.0e+04

6.2e+04

6.5e+04

7.0e+04

Surrogate Entanglement

Under Canonical Partitions

J Extension to Arbitrary Dimensional Models and Data

In this appendix, we extend our theoretical analysis and experiments, including the algorithm for enhancing the suitability of data to locally connected neural networks, from one-dimensional (sequential) models and data to P -dimensional models and data (such as two-dimensional image data or three-dimensional video data), for P ∈ N . Specifically, Appendix J.1 extends Section 3, Appendix J.2 extends Section 4 and Appendix J.3 extends Section 5.

To ease presentation, we consider P -dimensional data instances whose feature vectors are associated with coordinates ( n 1 , . . . , n P ) ∈ [ N ] P , where N = 2 L for some L ∈ N (if this is not the case we may add constant features as needed).

<latexi sh

1_b

64="fUPQH

T+

LwjG3/

u

V

D8CYM0

>A

c

F

v

5

NS

gq

m

E

y

k

XI

2oJ

Zp

B

9

r

z

W

7

nK

R

O

d

<latexi sh

1_b

64="GdDXj/U

3I

0

cqW9

ELAS

>

C

n

V

N

FJ

rP

ugk

w

Z

R

p

B

M

ym o5

H

O

v

T

Q

z

2

8

7

+Y

K

f

<latexi sh

1_b

64="GdDXj/U

3I

0

cqW9

ELAS

>

C

n

V

N

FJ

rP

ugk

w

Z

R

p

B

M

ym o5

H

O

v

T

Q

z

2

8

7

+Y

K

f

<latexi sh

1_b

64="F

5

Amqg

To

M

v

pP

f

O

Nc

>

C

n

Z

7S

B

I

W

0GQ2

Vd

E

Yy

r

J

k

z

Xw

H

9/

8

j

R

u

U

K

D

+

3

<latexi sh

1_b

64="GdDXj/U

3I

0

cqW9

ELAS

>

C

n

V

N

FJ

rP

ugk

w

Z

R

p

B

M

ym o5

H

O

v

T

Q

z

2

8

7

+Y

K

f

(c)

W

2

TN

(a)

𝐷

!

𝐷

"

𝐷

#

𝐷

$

𝐷

%

!

&$

𝐷

%

!

𝐷

%

!

&#

𝐷

%

!

&!

𝑅

𝑅

𝑅

𝑅

𝑅

𝑅

W

2

TN

(b)

two-dimensional

coordinates

1

2

3

4

5

6

7

8

1

2

5

6

17

18

21

22

3

4

7

8

19

20

23

24

9

10

13

14

25

26

29

30

11

12

15

16

27

28

31

32

33

34

37

38

49

50

53

54

35

36

39

40

51

52

55

56

41

42

45

46

57

58

61

62

43

44

47

48

59

60

63

64

1

2

3

4

5

6

7

8

⊗

N

2

n

=1

x

µ

-

1

(

n

)

,

W

2

TN

〉

µ

: [8]

×

[8]

→

[64]

axes indices

of

W

2

TN

𝐱

𝝁

"𝟏

(𝑵

𝟐

&𝟐)

𝐱

𝝁

"𝟏

(𝑵

𝟐

&𝟏)

𝐱

𝝁

"𝟏

(𝑵

𝟐

)

𝐱

𝝁

"𝟏

(𝟏)

𝐱

𝝁

"𝟏

(𝟐)

𝐱

𝝁

"𝟏

(𝟑)

𝐱

𝝁

"𝟏

(𝟒)

𝐱

𝝁

"𝟏

(𝑵

𝟐

&𝟑)

J.1 Low Entanglement Under Canonical Partitions Is Necessary and Sufficient for Fitting Tensor

We introduce the locally connected tensor network equivalent to a locally connected neural network that operates over P -dimensional data (Appendix J.1.1). Subsequently, we establish a necessary and sufficient condition required for it to fit a given tensor (Appendix J.1.2), generalizing the results of Section 3.2.

J.1.1 Tensor Network Equivalent to a Locally Connected Neural Network

For P -dimensional data, the locally connected tensor network we consider (defined in Section 3.1 for one-dimensional data) has an underlying perfect 2 P -ary tree graph of height L . We denote the tensor it generates by W P TN ∈ R D 1 ×···× D NP . Figure 6(a) provides its diagrammatic definition. As in the one-dimensional case, the lengths of axes corresponding to inner edges are taken to be R ∈ N , referred to as the width of the tensor network.

The axes of W P TN are associated with P -dimensional coordinates through a bijective function µ : [ N ] P → [ N P ] .

Definition 5. We say that a bijective function µ : [ N ] P → [ N P ] is compatible with the locally connected tensor network if, for any node in the tensor network, the coordinates mapped to indices of W P TN 's axes descendant to that node form a contiguous P -dimensional cubic block in [ N ] P ( e.g. , square block when P = 2 ) - see Figure 6(b) for an illustration. With slight abuse of notation, for K ⊆ [ N ] P we denote µ ( K ) := { µ ( n 1 , . . . , n P ) : ( n 1 , . . . , n P ) ∈ K} ⊆ [ N P ] .

Contracting the locally connected tensor network with { x ( n 1 ,...,n P ) ∈ R D µ ( n 1 ,...,n P ) } n 1 ,...,n P ∈ [ N ] according to a compatible µ , as depicted in Figure 6(c), can be viewed as a forward pass of the data instance { x ( n 1 ,...,n P ) } n 1 ,...,n P ∈ [ N ] through a locally connected neural network (with polynomial non-linearity), which produces the scalar 〈⊗ N P n =1 x µ -1 ( n ) , W P TN 〉 (see, e.g. , [12, 10, 35, 49]).

𝑙 = 1

𝑙 = 𝐿

Canonical Partitions of

𝑁

!

𝑙 = 0

K

∈

∈

K

c

J.1.2 Necessary and Sufficient Condition for Fitting Tensor

The ability of the locally connected tensor network, defined in Appendix J.1.1, to fit ( i.e. represent) a tensor is determined by the entanglements that the tensor admits under partitions of its axes, induced by the following canonical partitions of [ N ] P .

Definition 6. The canonical partitions of [ N ] P , illustrated in Figure 7 for P = 2 , are: 10

With the definition of canonical partitions for P -dimensional data in place, Theorem 3 generalizes Theorem 1. In particular, suppose that W P TN -the tensor generated by the locally connected tensor network - well-approximates A ∈ R D 1 ×···× D NP . Then, given a compatible µ : [ N ] P → [ N P ] (Definition 5), Theorem 3 establishes that the entanglement of A with respect to ( µ ( K ) , µ ( K ) c ) , where ( K , K c ) ∈ C P N , cannot be much larger than ln( R ) , whereas the expected entanglement attained by a random tensor with respect to ( µ ( K ) , µ ( K ) c ) is on the order of min {|K| , |K c |} (which is linear in N P for some canonical partitions).

In the other direction, Theorem 4 implies that low entanglement under partitions of axes induced by canonical partitions of [ N ] P is not only necessary for a tensor to be fit by the locally connected tensor network, but also sufficient.

Theorem 3. Let W P TN ∈ R D 1 ×···× D NP be a tensor generated by the locally connected tensor network defined in Appendix J.1.1, and µ : [ N ] P → [ N P ] be a compatible map from P -dimensional coordinates to axes indices of W P TN (Definition 5). For any A ∈ R D 1 ×···× D NP and glyph[epsilon1] ∈ [0 , ‖A‖ / 4] , if ‖W P TN -A‖ ≤ glyph[epsilon1] , then for all canonical partitions ( K , K c ) ∈ C P N (Definition 6):

where D µ ( K ) := min { ∏ n ∈ µ ( K ) D n , ∏ n ∈ µ ( K ) c D n } . In contrast, a random A ′ ∈ R D 1 ×···× D NP , drawn according to the uniform distribution over the set of unit norm tensors, satisfies for all canonical partitions ( K , K c ) ∈ C P N :

Proof sketch (proof in Appendix M.7). The proof is analogous to that of Theorem 1.

Theorem 4. Let A ∈ R D 1 ×···× D NP and glyph[epsilon1] > 0 . Suppose that for all canonical partitions ( K , K c ) ∈ C P N (Definition 6) it holds that QE ( A ; µ ( K )) ≤ glyph[epsilon1] 2 (2 N P -3) ‖A‖ 2 · ln( R ) , where µ : [ N ] P → [ N P ] is compatible with the locally connected tensor network (Definition 5). Then, there exists an assignment for the tensors constituting the locally connected tensor network (defined in Appendix J.1.1) such that it generates W P TN ∈ R D 1 ×···× D NP satisfying:

Proof sketch (proof in Appendix M.8). The claim follows through a reduction from the locally connected tensor network for P -dimensional data to that for one-dimensional data (defined in Section 3.1), i.e. from perfect 2 P -ary to perfect binary tree tensor networks. Specifically, we consider a modified locally connected tensor network for one-dimensional data, where axes corresponding to different inner edges can vary in length (as opposed to all having length R ). We then show, by arguments analogous to those in the proof of Theorem 2, that it can approximate A while having certain inner axes, related to the canonical partitions of [ N ] P , of lengths at most R . The proof concludes by establishing that, any tensor represented by such a locally connected tensor network for one-dimensional data can be represented via the locally connected tensor network for P -dimensional data (where the length of each axis corresponding to an inner edge is R ).

J.2 Low Entanglement Under Canonical Partitions Is Necessary and Sufficient for Accurate Prediction

In this appendix, we consider the locally connected tensor network from Appendix J.1.1 in a machine learning setting, and extend the results and experiments of Section 4 from one-dimensional to P -dimensional models and data.

J.2.1 Accurate Prediction Is Equivalent to Fitting Data Tensor

The locally connected tensor network generating W P TN ∈ R D 1 ×···× D NP is equivalent to a locally connected neural network operating over P -dimensional data (see Appendix J.1.1). Specifically, a forward pass of the latter over { x ( n 1 ,...,n P ) ∈ R D µ ( n 1 ,...,n P ) } n 1 ,...,n P ∈ [ N ] yields 〈⊗ N P n =1 x µ -1 ( n ) , W P TN 〉 , for a compatible µ : [ N ] P → [ N P ] (Definition 5). Suppose we are given a training set of labeled instances { { x (( n 1 ,...,n P ) ,m ) } n 1 ,...,n P ∈ [ N ] , y ( m ) } M m =1 drawn i.i.d. from some distribution, where y ( m ) ∈ { 1 , -1 } for m ∈ [ M ] . Learning the parameters of the neural network through the soft-margin support vector machine (SVM) objective amounts to optimizing:

for a predetermined constant B > 0 . This objective generalizes Equation (1) from one-dimensional to P -dimensional model and data. Assume that instances are normalized, i.e. ‖ x (( n 1 ,...,n P ) ,m ) ‖ ≤ 1 for all n 1 , . . . , n P ∈ [ N ] , m ∈ [ M ] , and that B ≤ 1 . By a derivation analogous to that succeeding Equation (1) in Section 4.1, if follows that minimizing the SVM objective is equivalent to minimizing ∥ ∥ W P TN ‖W P TN ‖ -D P emp ‖D P emp ‖ ∥ ∥ , where:

extends the notion of empirical data tensor to P -dimensional data. In other words, the accuracy achievable over the training data is determined by the extent to which W P TN ‖W P TN ‖ can fit the normalized empirical data tensor D P emp ‖D P emp ‖ .

The same arguments apply to the population loss, in which case D P emp is replaced by the population data tensor :

The achievable accuracy over the population is therefore determined by the extent to which W P TN ‖W P TN ‖

P

can fit the normalized population data tensor D pop ‖D P pop ‖ . Accordingly, we refer to the minimal distance from it as the supobtimality in achievable accuracy , generalizing Definition 3 from Section 4.1.

Definition 7. In the context of the classification setting above, the suboptimality in achievable accuracy is:

J.2.2 Necessary and Sufficient Condition for Accurate Prediction

In the classification setting of Appendix J.2.1, by invoking Theorems 3 and 4 from Appendix J.1.2, we conclude that the suboptimality in achievable accuracy is small if and only if the population (empirical) data tensor D P pop ( D P emp ) admits low entanglement under the canonical partitions of features (Definition 6). Specifically, we establish Corollary 3, Proposition 4 and Corollary 4, which generalize Corollary 1, Proposition 2 and Corollary 2 from Section 4.2, respectively, to P -dimensional model and data.

Corollary 3. Consider the classification setting of Appendix J.2.1, and let glyph[epsilon1] ∈ [0 , 1 / 4] . If there exists a canonical partition ( K , K c ) ∈ C P N (Definition 6) under which QE ( D P pop ; µ ( K ) ) > ln( R ) + 2 glyph[epsilon1] · ln( D µ ( K ) ) + 2 √ 2 glyph[epsilon1] , where D µ ( K ) := min { ∏ n ∈ µ ( K ) D n , ∏ n ∈ µ ( K ) c D n } , then: SubOpt P > glyph[epsilon1] .

Conversely, if for all ( K , K c ) ∈ C P N it holds that QE ( D P pop ; µ ( K ) ) ≤ glyph[epsilon1] 2 8 N P -12 · ln( R ) , then:

Proof. Implied by Theorems 3 and 4 after accounting for W P TN being normalized in the suboptimality in achievable accuracy, as done in the proof of Corollary 1.

Proposition 4. Consider the classification setting of Appendix J.2.1, and let δ ∈ (0 , 1) and γ > 0 . If the training set size M satisfies M ≥ 2 ln( 2 δ ) ‖D P pop ‖ 2 γ 2 , then with probability at least 1 -δ :

Proof. The claim is established by following steps identical to those in the proof of Proposition 2.

Second, if for all ( K , K c ) ∈ C P N it holds that QE ( D P emp ; µ ( K ) ) ≤ glyph[epsilon1] 2 32 N P -48 · ln( R ) , then:

Moreover, the conditions above on the entanglements of D P emp can be evaluated efficiently (in O ( DN P M 2 + N P M 3 ) time O ( DN P M + M 2 ) and memory, where D := max n ∈ [ N P ] D n ).

Proof. Implied from Corollary 3, Proposition 4 with γ = glyph[epsilon1] 2 and Algorithm 1 in Appendix F by following steps identical to those in the proof of Corollary 2.

(∐{}{]̂∐⌈(∑∐√√]√]}{√



〉{√∐{˜⌈˜⌉˜{√(⊎{̂˜√(







⋂˜√√(̂̂√√∐̂glyph[arrowbt]({⌋}



/

/

/

/

/

/

/









⊗⊗

∧⊔⊔

⊕}̂]⌈˜⊗˜√

J.2.3 Empirical Demonstration

Figure 8 extends the experiments reported by Figure 3 in Section 4.3 from one-dimensional (sequential) audio data to two-dimensional image data. Specifically, it demonstrates that the prediction accuracies of convolutional neural networks over a variant of CIFAR10 [33] is inversely correlated with the entanglements that the data admits under canonical partitions of features (Definition 6), i.e. with the entanglements of the empirical data tensor under partitions of its axes induced by canonical partitions.

J.3 Enhancing Suitability of Data to Locally Connected Neural Networks

We extend the preprocessing algorithm from Section 5, aimed to enhance the suitability of a data distribution to locally connected neural networks, from one-dimensional to P -dimensional models and data (Appendices J.3.1 and J.3.2). Empirical evaluations demonstrate that it significantly improves prediction accuracy of common architectures (Appendix J.3.3).

J.3.1 Search for Feature Arrangement With Low Entanglement Under Canonical Partitions

Suppose we have M ∈ N training instances { { x (( n 1 ,...,n P ) ,m ) } n 1 ,...,n P ∈ [ N ] , y ( m ) } M m =1 , where x (( n 1 ,...,n P ) ,m ) ∈ R D and y ( m ) ∈ { 1 , -1 } for n 1 , . . . , n P ∈ [ N ] , m ∈ [ M ] , with D ∈ N . For models intaking P -dimensional data, the recipe for enhancing the suitability of a data distribution to locally connected neural networks from Section 5.1 boils down to finding a permutation π : [ N ] P → [ N ] P , which when applied to feature coordinates leads the empirical data tensor D P emp (Equation (8)) to admit low entanglement under canonical partitions (Definition 6). 11

A greedy realization, analogous to that outlined in Section 5.1, is as follows. Initially, partition the features into 2 P equally sized disjoint sets {K 1 , ( k 1 ,...,k P ) ⊂ [ N ] P } k 1 ,...,k P ∈ [2] such that the average of D P emp 's entanglements induced by these sets is minimal. That is, find an element of:

where µ : [ N ] P → [ N P ] is a compatible map from coordinates in [ N ] P to axes indices (Definition 5). The permutation π will map each K 1 , ( k 1 ,...,k P ) to coordinates × P p =1 { N 2 · ( k p -1) + 1 , . . . , N 2 · k p } , for k 1 , . . . , k P ∈ [2] . Then, partition similarly each of {K 1 , ( k 1 ,...,k P ) } k 1 ,...,k P ∈ [2] into 2 P equally sized disjoint sets. Continuing in the same fashion, until we reach subsets {K L, ( k 1 ,...,k P ) } k 1 ,...,k P ∈ [ N ] consisting of a single feature coordinate each, fully specifies the permutation π .















Algorithm 3 Enhancing Suitability of P -Dimensional Data to Locally Connected Neural Networks

1: Input: X := { { x (( n 1 ,...,n P ) ,m ) } n 1 ,...,n P ∈ [ N ] } M m =1 -M ∈ N data instances comprising N P features each

2: Output: Permutation π : [ N ] P → [ N ] P to apply to feature coordinates

3: Let K 0 , (1 ,..., 1) := [ N ] P and denote L := log 2 ( N )

4: # We assume for simplicity that N is a power of two, otherwise one may add constant features

5: for l = 0 , . . . , L -1 , k 1 , . . . , k P = 1 , . . . , 2 l do

6: Using a reduction to a minimum balanced 2 P -cut problem (Proposition 5), find an approximate solution {K l +1 , (2 k 1 -i 1 ,..., 2 k P -i P ) ⊂ K l, ( k 1 ,...,k P ) } i 1 ,...,i P ∈{ 0 , 1 } for:

7: end for

8: # At this point, {K L, ( k 1 ,...,k P ) } k 1 ,...,k P ∈ [ N ] each contain a single feature coordinate

9: return π that maps ( n 1 , . . . , n P ) ∈ K L, ( k 1 ,...,k P ) to ( k 1 , . . . , k P ) , for every k 1 , . . . , k P ∈ [ N ]

As in the case of one-dimensional models and data (Section 5.1), the step lying at the heart of the above scheme - finding a balanced partition into 2 P sets that minimizes average entanglement - is computationally prohibitive. In the next supappendix we will see that replacing entanglement with surrogate entanglement brings forth a practical implementation.

J.3.2 Practical Algorithm via Surrogate for Entanglement

To efficiently implement the scheme from Appendix J.3.1, we replace entanglement with surrogate entanglement (Definition 4), which for P -dimensional data is straightforwardly defined as follows.

Definition 8. Given a set of M ∈ N instances X := { { x (( n 1 ,...,n P ) ,m ) ∈ R D } n 1 ,...,n P ∈ [ N ] } M m =1 , denote by p ( n 1 ,...,n P ) , ( n ′ 1 ,...,n ′ P ) the multivariate Pearson correlation between features ( n 1 , . . . , n P ) ∈ [ N ] P and ( n ′ 1 , . . . , n ′ P ) ∈ [ N ] P . For K ⊆ [ N ] P , the surrogate entanglement of X with respect to the partition ( K , K c ) , denoted SE ( X ; K ) , is the sum of Pearson correlation coefficients between pairs of features, the first belonging to K and the second to K c := [ N ] P \ K . That is:

Analogously to the case of one-dimensional data (Appendix G), Proposition 5 shows that replacing the entanglement with surrogate entanglement in the scheme from Appendix J.3.1 converts each search for a balanced partition minimizing average entanglement into a minimum balanced 2 P -cut problem . Although the minimum balanced 2 P -cut problem is NP-hard (see, e.g. , [23]), similarly to the minimum balanced ( 2 -) cut problem, it enjoys a wide array of well-established approximation implementations, particularly ones designed for large scale [29, 57]. We therefore obtain a practical algorithm for enhancing the suitability of a data distribution with P -dimensional instances to locally connected neural networks - see Algorithm 3.

Proposition 5. For any s K ⊆ [ N ] P of size divisible by 2 P , the following optimization problem can be framed as a minimum balanced 2 P -cut problem over a complete graph with | s K| vertices:

Specifically, there exists a complete undirected weighted graph with vertices s K and edge weights w : s K × s K → R such that, for any partition of s K into equally sized dis-

joint sets {K k 1 ,...,k P } k 1 ,...,k P ∈ [2] , the weight of the 2 P -cut in the graph induced by them 1 2 ∑ k 1 ,...,k P ∈ [2] ∑ ( n 1 ,...,n P ) ∈K k 1 ,...,k P , ( n ′ 1 ,...,n ′ P ) ∈ s K\K k 1 ,...,k P w ( { ( n 1 , . . . , n P ) , ( n ′ 1 , . . . , n ′ P ) } ) -is equal, up to multiplicative and additive constants, to the term minimized in Equation (5) .

Proof. Consider the complete undirected graph whose vertices are s K and where the weight of an edge { ( n 1 , . . . , n P ) , ( n ′ 1 , . . . , n ′ P ) } ∈ s K× s K is:

(recall that p ( n 1 ,...,n P ) , ( n ′ 1 ,...,n ′ P ) stands for the multivariate Pearson correlation between features ( n 1 , . . . , n P ) and ( n ′ 1 , . . . , n ′ P ) in X ). For any partition of s K into equally sized disjoint sets {K k 1 ,...,k P } k 1 ,...,k P ∈ [2] , it holds that:

where 1 2 SE ( X ; s K ) does not depend on {K k 1 ,...,k P } k 1 ,...,k P ∈ [2] . This concludes the proof.

J.3.3 Experiments

We supplement the experiments from Section 5.2 for one-dimensional models and data, by empirically evaluating Algorithm 3 using two-dimensional convolutional neural networks over randomly permuted image data. Specifically, Table 3 presents experiments analogous to those of Table 1 from Section 5.2.1 over the CIFAR10 [33] image dataset. For brevity, we defer some implementation details to Appendix L.

K Further Experiments

Table 4 supplements the experiment of Table 1 from Section 5.2.1, demonstrating that Algorithm 2 can be applied to data with a large number of features ( e.g. with tens of thousands of features). In the experiment, instances of the minimum balanced cut problem solved as part of Algorithm 2 entail graphs with up to 25 · 10 8 edges. They are solved using the well-known edge sparsification algorithm of [57] that preserves weights of cuts, allowing for configurable compute and memory consumption (the more resources are consumed, the more accurate the solution will be). In contrast, this approach is not directly applicable for improving the efficiency of IGTD.

Table 5 supplements Table 2 from Section 5.2.2 by reporting results of experiments with an additional tabular benchmark - 'dna'.

L Further Implementation Details

We provide implementation details omitted from our experimental reports (Section 4.3, Section 5, Appendix J.2.3, Appendix J.3.3 and Appendix K). Source code for reproducing our results and figures, based on the PyTorch [44] framework, can be found at https://github.com/nmd95/data_ suitable_lc_nn_code . All experiments were run on a single Nvidia RTX A6000 GPU.

L.1 Empirical Demonstration of Theoretical Analysis (Figures 3, 5, and 8)

L.1.1 Figures 3 and 5

Dataset : The SpeechCommands dataset [64] contains raw audio segments of length up to 16000, split into 84843 train and 11005 test segments. We zero-padded all audio segments to have a length of 16000 and resampled them using sinc interpolation (default PyTorch implementation). We allocated 11005 audio segments from the train set for validation, i.e. the dataset was split into 73838 train, 11005 validation and 11005 test audio segments, and created a binary one-vs-all classification version of SpeechCommands by taking all audio segments labeled by the class '33' (corresponding to the word 'yes'), and sampling an equal amount of segments from the remaining classes (this process was done separately for the train, validation and test sets). The resulting balanced classification dataset had 5610 train, 846 validation and 838 test segments. Lastly, we resampled all audio segments in the dataset from 16000 to 4096 features using sinc interpolation.

Random feature swaps: Starting with the original order of features, we created increasingly 'shuffled' versions of the dataset by randomly swapping the position of features. For each number of random position swaps k ∈ { 0 , 250 , . . . , 4000 } , we created ten datasets, whose features were subject to k random position swaps between features, using different random seeds.

Quantum entanglement measurement : Each reported value in the plot is the average of entanglements with respect to canonical partitions (Definition 2) corresponding to levels l = 1 , 2 , 3 , 4 , 5 , 6 . We used Algorithm 1 described in Appendix F on two mini-batches of size 500, randomly sampled from the train set. As customary ( cf. [59]), every input feature x was embedded using the following sine-cosine scheme:

with θ = 0 . 085 .

Neural network architectures :

CNN : An adaptation of the M5 architecture from [17], which is designed for audio data. Our implementation is based on https://github.com/danielajisafe/Audio_WaveForm_ Paper_Implementation .

S4 : Official implementation of [26] with a hidden dimension of 128 and 4 layers.

Local-Attention : Adaptation of the local-attention model from [46], as implemented in https: //github.com/lucidrains/local-attention . We use a multi-layer perceptron (MLP) for mapping continuous (raw) inputs to embeddings, which are fed as input the the local-attention model. For classification, we collapse the spatial dimension of the network's output using mean-pooling and pass the result into an MLP classification head. The network had attention dimension 128 (with 2 heads of dimension 64), depth 8, hidden dimension of MLP blocks 341

(computed automatically by the library based on the attention dimension) and local-attention window size 10.

Training and evaluation: The binary cross-entropy loss was minimized via the Adam optimizer [32] with default β 1 , β 2 coefficients. Batch sizes were chosen to be the largest powers of two that fit in the GPU memory. The duration of optimization was 300 epochs for all models (number of epochs was taken large enough such that the train loss plateaued). After the last training epoch, the model which performed best on the validation set was chosen, and its average test accuracy is reported. Additional optimization hyperparameters are provided in Table 6. We note that for S4 [26], in accordance with its official implementation, we use a cosine annealing learning rate scheduler.

L.1.2 Figure 8

Dataset : We created one-vs-all binary classification datasets based on CIFAR10 [33] as follows. All images were converted to grayscale using the PyTorch default implementation. Then, we allocated 7469 images from the train set for validation, i.e. the dataset was split into 42531 train, 7469 validation and 1000 test images. We took all images labeled by the class '0' (corresponding to images of airplanes), and uniformly sampled an equal amount of images from the remaining classes (this process was done separately for the train, validation and test sets). The resulting balanced classification dataset had 8506 train, 1493 validation and 2001 test images.

Random feature swaps: We created increasingly 'shuffled' versions of the dataset according to the protocal described in Appendix L.1.1.

Quantum entanglement measurement : Each reported value in the plot is the average entanglements with respect to canonical partitions (Definition 6) corresponding to levels l = 1 , 2 .

Neural network architectures :

CNN : Same architecture used for the experiments of Table 2 (see Appendix L.2.3), but with one-dimensional convolutional layers replaced with two-dimensional convolutional layers. See Table 7 for the exact architectural hyperparameters.

VGG : Adaptation of the VGG16 architecture, as implemented in the PyTorch framework, for binary classification tasks on grayscale images. Specifically, the number of input channels is set to one and the number of output classes is set to two.

MobileNet : Adaptation of the MobileNet V2 architecture, as implemented in the PyTorch framework, for binary classification on grayscale images. Specifically, the number of input channels is set to one and the number of output classes is set to two.

Training and evaluation : The binary cross-entropy loss was minimized for 150 epochs via the Adam optimizer [32] with default β 1 , β 2 coefficients (number of epochs was taken large enough such that the train loss plateaued). Batch size was chosen to be the largest power of two that fit in the

GPU memory. After the last training epoch, the model which performed best on the validation set was chosen, and its average test accuracy is reported. Additional optimization hyperparameters are provided in Table 8.

L.2 Enhancing Suitability of Data to Locally Connected Neural Networks (Tables 1, 2, 3, 4, and 5)

L.2.1 Feature Rearrangement Algorithms

Algorithm 2, 3 and IGTD [67] were applied to the training set. Subsequently, the learned feature rearrangement was used for both the validation and test data. In instances where three-way crossvalidation was used, distinct rearrangements were learned for each separately.

Algorithm 2 and Algorithm 3 : Approximate solutions to the minimum balanced cut problems were obtained using the METIS graph partitioning algorithm [29], as implemented in https://github. com/networkx/networkx-metis .

IGTD [67] : The original IGTD implementation supports rearranging data only into two-dimensional images. We adapted its implementation to support one-dimensional (sequential) data for the experiments of Tables 1 and 2.

L.2.2 Randomly Permuted Audio Datasets (Table 1)

Dataset : To facilitate efficient experimentation, we downsampled all audio segments in SpeechCommands to have 2048 features. Furthermore, for the train, validation and test sets separately, we used 20% of the audio segments available for each class.

Neural network architectures :

CNN : Same architecture used for the experiments of Figure 3 (see Appendix L.1.1).

S4 : Same architecture used for the experiments of Figure 3 (see Appendix L.1.1).

Local-Attention : Same architecture used in the experiments of Figure 3, but with a depth 4 network (we reduced the depth to allow for more efficient experimentation).

Training and evaluation: The cross-entropy loss was minimized via the Adam optimizer [32] with default β 1 , β 2 coefficients. Batch sizes were chosen to be the largest powers of two that fit in the GPU memory. The duration of optimization was 200, 200 and 450 epochs for the CNN, S4 and Local-Attention models, respectively (number of epochs was taken large enough such that the train loss plateaued). After the last training epoch, the model which performed best on the validation set was chosen, and its average test accuracy is reported. Additional optimization hyperparameters are provided in Table 9. We note that for S4 [26], in accordance with its official implementation, we use a cosine annealing learning rate scheduler.

L.2.3 Tabular Datasets (Tables 2 and 5)

Datasets : The datasets 'dna', 'semeion' and 'isolet' are all from the OpenML repository [60]. For each dataset we split the samples intro three folds, which were used for evaluation according to a

standard three-way cross-validation protocol. That is, for each of the three folds, we used one third of the data as a test set and the remaining for train and validation. One third of the samples in the remaining folds (not used for testing) were allocated for the validation set.

Neural network architectures :

CNN : We used a ResNet adapted for tabular data. It consisted of residual blocks of the following form:

After applying a predetermined amount of residual blocks, a global average pooling and fully connected layers were used to output the prediction. The architectural hyperparameters are specified in Table 10.

S4 : Same architecture used for the experiments of Figure 3 (see Appendix L.1.1), but with a hidden dimension of 64.

Local-Attention : Same architecture used for the experiments of Figure 3 (see Appendix L.1.1), but with 4 attention heads of dimension 32 and a local-attention window size of 25.

Table 10: Architectural hyperparameters for the convolutional neural network used in the experiments of Tables 2 and 4.

Training and evaluation : The cross-entropy loss was minimized for 300 epochs via the Adam optimizer [32] with default β 1 , β 2 coefficients (number of epochs was taken large enough such that the train loss plateaued). Batch sizes were chosen to be the largest powers of two that fit in the GPU memory. After the last training epoch, the model which performed best according to the validation sets was chosen, and test accuracy was measured on the test set. The reported accuracy is the average over the three folds. Additional optimization hyperparameters are specified in Table 11. We note that for S4 [26], in accordance with its official implementation, we use a cosine annealing learning rate scheduler.

L.2.4 Randomly Permuted Image Datasets (Table 3)

Dataset : The data acquisition process followed the protocol described in Appendix L.1.2, except that the data was not converted into a binary one-vs-all classification dataset.

Neural network architectures :

CNN : Same architecture used for the experiments of Figure 8 (see Appendix L.1.2).

Training and evaluation : The cross-entropy loss was minimized for 500 epochs via the Adam optimizer [32] with default β 1 , β 2 coefficients (number of epochs was taken large enough such that the train loss plateaued). Batch size was chosen to be the largest power of two that fit in the GPU memory. After the last training epoch, the model which performed best on the validation set was chosen, and its average test accuracy is reported. Additional optimization hyperparameters are provided in Table 12.

L.2.5 Randomly Permuted Audio Datasets With a Large Number of Features (Table 4)

Dataset : The data acquisition process followed the protocol described in Appendix L.1.1, except that the data was not transformed into a binary one-vs-all classification dataset and the audio segments were upsampled from 16,000 to 50,000.

Edge Sparsification : To facilitate running the METIS graph partitioning algorithm over the minimum balanced cut problems encountered as part of Algorithm 2, we first removed edges from the graph using the spectral sparsification algorithm of [57]. Specifically, we used the official Julia implementation ( https://github.com/danspielman/Laplacians.jl ) with hyperparameter glyph[epsilon1] = 0 . 15 .

Neural network architectures :

CNN : Same architecture used for the experiments of Table 2 (see Appendix L.2.3).

S4 : Same architecture used for the experiments of Table 1 (see Appendix L.1.1), but with a hidden dimension of 32 (we reduced the hidden dimension due to GPU memory considerations).

Training and evaluation : The cross-entropy loss was minimized for 200 epochs via the Adam optimizer [32] with default β 1 , β 2 coefficients (number of epochs was taken large enough such that the train loss plateaued). Batch sizes were chosen to be the largest powers of two that fit in the GPU memory. After the last training epoch, the model which performed best on the validation set was chosen, and its average test accuracy is reported. Additional optimization hyperparameters are provided in Table 13. We note that for S4 [26], in accordance with its official implementation, we use a cosine annealing learning rate scheduler.

M Deferred Proofs

M.1 Useful Lemmas

Below we collect a few useful results, which we will use in our proofs.

Lemma 2. We denote the vector of singular values of a matrix X (arranged in decreasing order) by S ( X ) . For any A , B ∈ R D 1 × D 2 it holds that:

Lemma 3. Let P = { p 1 , ..., p N } , Q = { q 1 , ..., q N } be two probability distributions supported on [ N ] , and denote by TV ( P , Q ) := 1 2 ∑ N n =1 | p n -q n | their total variation distance . If for glyph[epsilon1] ∈ (0 , 1 / 2) it holds that TV ( P , Q ) ≤ glyph[epsilon1] , then:

where H ( P ) := -∑ N n =1 p n ln( p n ) is the entropy of P , and H b ( c ) := -c · ln( c ) -(1 -c ) · ln(1 -c ) is the binary entropy of a Bernoulli distribution parameterized by c ∈ [0 , 1] .

Lemma 4 (Hoeffding inequality in Hilbert space) . Let X 1 , .., X N be an i.i.d. sequence of random variables whose range is some separable Hilbert space H . Suppose that E [ X n ] = 0 and ‖ X n ‖ ≤ c for all n ∈ [ N ] . Then, for all t ≥ 0 :

where ‖·‖ refers to the Hilbert space norm.

Proof. See Section 2.4 of [51].

Lemma 5 (adapted from [35]) . Let G = ( V, E ) be the perfect binary tree graph, with vertices V and edges E , underlying the locally connected tensor network that generates W TN ∈ R D 1 ×···× D N (defined in Section 3.1). For K ⊆ [ N ] , let V K ⊆ V and V K c ⊆ V be the leaves in G corresponding to axes indices K and K c of W TN , respectively. Lastly, given a cut ( A,B ) of V , i.e. A ⊆ V and B ⊆ V are disjoint and A ∪ B = V , denote by C ( A,B ) := {{ u, v } ∈ E : u ∈ A,v ∈ B } the edge-cut set. Then:

In particular, if ( K , K c ) ∈ C N , then:

Proof. See Claim 1 in [35] for the upper bound on the rank of J W TN ; K K for any K ⊆ [ N ] . If ( K , K c ) ∈ C N , since there exists a cut ( A,B ) of V such that V K ⊆ A and V K c ⊆ B whose edge-cut set is of a singleton, we get that rank( J W TN ; K K ) ≤ R .

Lemma 6 (adapted from [35]) . Let G = ( V, E ) be the perfect 2 P -ary tree graph, with vertices V and edges E , underlying the locally connected tensor network that generates W P TN ∈ R D 1 ×···× D NP (defined in Appendix J.1.1). Furthermore, let µ : [ N ] P → [ N P ] be a compatible map from P -dimensional coordinates to axes indices of W P TN (Definition 5). For J ⊆ [ N P ] , let V J ⊆ V and V J c ⊆ V be the leaves in G corresponding to axes indices J and J c of W P TN , respectively. Lastly, given a cut ( A,B ) of V , i.e. A ⊆ V and B ⊆ V are disjoint and A ∪ B = V , denote by C ( A,B ) := {{ u, v } ∈ E : u ∈ A,v ∈ B } the edge-cut set. Then:

In particular, if ( K , K c ) ∈ C P N , then:

Proof. See Claim 1 in [35] for the upper bound on the rank of q W P TN ; J y for any J ⊆ [ N P ] . If ( K , K c ) ∈ C P N , since there exists a cut ( A,B ) of V such that V µ ( K ) ⊆ A and V µ ( K ) c ⊆ B whose edge-cut set is of a singleton, we get that rank( q W P TN ; µ ( K ) y ) ≤ R .

Lemma 7 (adapted from Theorem 3.18 in [25]) . Let A ∈ R D 1 × ... × D N and glyph[epsilon1] > 0 . For each canonical partition ( K , K c ) ∈ C N , let σ K , 1 ≥ ...σ K,D K be the singular values of J A ; K K , where D K := min { ∏ n ∈K D n , ∏ n ∈K c D n } . Let ( n K ) K∈C N ∈ N C N be an assignment of an integer to each K ∈ C N . For any such assignment, consider the set of tensors with Hierarchical Tucker (HT) rank at most ( n K ) K∈C N as follows:

Suppose that for all ( K , K c ) ∈ C N it holds that √ ∑ D K d = n K +1 σ 2 K ,d ≤ glyph[epsilon1] √ 2 N -3 . Then, there exists W ∈ HT (( n K ) K∈C N ) satisfying:

In particular, if for all ( K , K c ) ∈ C N it holds that √ ∑ D K d = R +1 σ 2 K ,d ≤ glyph[epsilon1] √ 2 N -3 , then there exists W TN ∈ R D 1 ×···× D N generated by the locally connected tensor network (defined in Section 3.1) satisfying:

Lemma 8 (adapted from Theorem 3.18 in [25]) . Let ( n K ) K∈C N ∈ N C N be an assignment of an integer to each K ∈ C N . For any such assignment, consider the set of tensors with Hierarchical Tucker (HT) rank at most ( n K ) K∈C N as follows:

Consider a locally connected tensor network with varying widths ( n K ) K∈C N , i.e. a tensor network conforming to a perfect binary tree graph in which the lengths of inner axes are as follows (in contrast to all being equal to R as in the locally connected tensor network defined in Section 3.1). An axis corresponding to an edge that connects a node with descendant leaves indexed by K to its parent is assigned the length n K . Then, every A ∈ HT (( n K ) K∈C N ) can be represented by said locally connected tensor network with varying widths, meaning there exists an assignment to the tensors of the tensor network such that it generates A . In particular, if n K = R for all K ∈ C N , then A can be generated by the locally connected tensor network with all inner axes being of length R .

Lemma 9. Let V , W ∈ R D 1 ×···× D N be tensors such that ‖V‖ = ‖W‖ = 1 and ‖V - W‖ < 1 2 . Then, for any ( K , K c ) ∈ C N it holds that:

where H b ( c ) := -( c · ln( c ) + (1 -c ) · ln(1 -c )) is the binary entropy of a Bernoulli distribution parameterized by c ∈ [0 , 1] , and D K := min { ∏ n ∈K D n , ∏ n ∈K c D n } .

Proof. For any matrix M ∈ R D 1 × D 2 with D := min { D 1 , D 2 } , let S ( M ) = ( σ M , 1 , ..., σ M ,D ) be the vector consisting of its singular values. First note that

J K J K Let v 1 ≥ · · · ≥ v D K and w 1 ≥ · · · ≥ w D K be the singular values of J V ; K K and J W ; K K , respectively. We have by the Cauchy-Schwarz inequality that

Now the first term is upper bounded by ‖V - W‖ 2 , and for the second we have where we use the fact that ‖V‖ = ‖W‖ = 1 , and again Cuachy-Schwarz. Overall we have:

Note that the left hand side of the inequality above equals twice the total variation distance between the distributions defined by { w 2 d } D K d =1 and { v 2 d } D K d =1 . Therefore by Lemma 3 we have:

Lemma10. Let P = { p ( x ) } x ∈ [ S ] , where S ∈ N , be a probability distribution, and denote its entropy by H ( P ) := E x ∼P [ln (1 /p ( x ))] . Then, for any 0 < a < 1 , there exists a subset T ⊆ [ S ] such that Pr x ∼P ( T c ) ≤ a and | T | ≤ e H ( P ) a .

Proof. By Markov's inequality we have for any 0 < a < 1 :

Let T := { x : e -H ( P ) a ≤ p ( x ) } ⊆ [ S ] . Note that

and so | T | ≤ e H ( P ) a and Pr x ∼P ( T c ) ≤ a , as required.

M.2 Proof of Theorem 1

If A = 0 the theorem is trivial, since then QE ( A ; K ) = 0 for all ( K , K c ) ∈ C N , so we can assume A glyph[negationslash] = 0 . We have:

Now, let ˆ A := A ‖A‖ and ˆ W TN = W TN ‖W TN ‖ be normalized versions of A and W TN , respectively, and let c = 2 glyph[epsilon1] ‖ A ‖ . Note that c < 2 ‖A‖ 4 1 ‖A‖ = 1 2 , and therefore by Lemma 9 we have:

By Lemma 5 we have that and therefore

Substituting c = 2 glyph[epsilon1] ‖ A ‖ and invoking the elementary inequality H b ( x ) ≤ 2 √ x we obtain as required.

As for the expected entanglements of a random tensor, let A ′ ∈ R D 1 ×···× D N be drawn according to the uniform distribution over the set of unit norm tensors. According to [52], for any canonical partition ( K , K c ) ∈ C N it holds that:

where D ′ K := max { ∏ n ∈K D n , ∏ n ∈K c D n } and recall that D K := min { ∏ n ∈K D n , ∏ n ∈K c D n } . By the elementary inequality of ∑ p k = q 1 /k ≥ ln( p ) -ln( q ) we therefore have that:

Since D K ≤ D ′ K and D K ≥ 1 , it holds that ( D K -1) / 2 D ′ K ≤ 1 / 2 and ln( D ′ K / ( D ′ K +1)) ≥ ln(1 / 2) . Thus:

M.3 Proof of Theorem 2

Consider, for each canonical partition ( K , K c ) ∈ C N , the distribution where σ K , 1 ≥ σ K , 2 ≥ ... ≥ σ K ,D K are the singular values of J A ; K K (note that 1 ‖A‖ 2 ∑ j σ 2 K ,j = ‖A‖ 2 ‖ A ‖ 2 = 1 so P K is indeed a probability distribution). Denoting by H ( P K ) := E i ∼P K [ln (1 /p K ( i ))] the entropy of P K , by assumption:

for all ( K , K c ) ∈ C N . Thus, taking a = glyph[epsilon1] 2 ‖A‖ 2 (2 N -3) we obtain by Lemma 10 that there exists a subset T K ⊆ [ D K ] such that and | T K | ≤ e H ( P K ) c = e ln( R ) = R . Note that

Since this holds for any subset of cardinality at most R . Taking complements we obtain so

We can now invoke Lemma 7, which implies that there exists some W TN ∈ R D 1 ×···× D N generated by the locally connected tensor network satisfying:

M.4 Proof of Corollary 1

Notice that the entanglements of a tensor are invariant to multiplication by a constant. In particular, QE ( D pop ; K ) = QE ( D pop / ‖D pop ‖ ; K ) for any ( K , K c ) ∈ C N . Hence, if there exists a canonical partition ( K , K c ) ∈ C N under which QE ( D pop ; K ) > ln( R ) + 2 glyph[epsilon1] · ln( D K ) + 2 √ 2 glyph[epsilon1] , then Theorem 1 implies that min W TN ‖W TN -D pop / ‖D pop ‖‖ > glyph[epsilon1] . Now, for any non-zero W ∈ R D 1 ×··· D N generated by the locally connected tensor network, one can also represent W / ‖W‖ by multiplying any of the tensors constituting the tensor network by 1 / ‖W‖ (contraction is a multilinear operation). Thus:

∥ ∥ ∥ ∥ which concludes the first part of the claim, i.e. the necessary condition for low suboptimality in achievable accuracy.

For the sufficient condition, if for all ( K , K c ) ∈ C N it holds that QE ( D pop ; K ) ≤ glyph[epsilon1] 2 8 N -12 · ln( R ) , then by Theorem 2 there exists an assignment for the locally connected tensor network such that ‖W TN -D pop / ‖D pop ‖‖ ≤ glyph[epsilon1]/ 2 . From the triangle inequality we obtain:

Since ‖W TN -D pop / ‖D pop ‖‖ ≤ glyph[epsilon1]/ 2 it holds that ‖W TN ‖ ≤ 1 + glyph[epsilon1]/ 2 . Combined with the fact that ∥ ∥ W TN -W TN ‖W TN ‖ ∥ ∥ = ‖W TN ‖ -1 , we get that ∥ ∥ W TN -W TN ‖W TN ‖ ∥ ∥ ≤ glyph[epsilon1]/ 2 . Plugging this into Equation (11) yields:

and so SubOpt := min W TN ∥ W TN ‖W TN ‖ -D pop ‖D pop ‖ ∥ ≤ glyph[epsilon1] . M.5 Proof of Proposition 2

We have the identity

By the triangle inequality the above is bounded by

For m ∈ [ M ] , let X ( m ) = y ( m ) · ⊗ n ∈ [ N ] x ( n,m ) - D pop . These are i.i.d. random variables with E [ X ( m ) ] = 0 and ‖X ( m ) ‖ ≤ 2 for all m ∈ [ M ] . Note that

∥ ∥ so by Lemma 4 with c = 2 , t = ‖D pop ‖ γ 2 , assuming M ≥ 2 ln( 2 δ ) ‖D pop ‖ 2 γ 2 we have with probability at least 1 -δ

and therefore

M.6 Proof of Corollary 2

First, by Proposition 2, for M ≥ 8 ln( 2 δ ) ‖D pop ‖ 2 glyph[epsilon1] 2 we have that with probability at least 1 -δ :

∥ ∥ Now, we establish the necessary condition on the entanglements of D emp . Assume that there exists a canonical partition ( K , K c ) ∈ C N under which

We may view D emp as the population data tensor for the uniform data distribution over the training instances {( ( x (1 ,m ) , . . . , x ( N,m ) ) , y ( m ) )} M m =1 . Thus, Corollary 1 implies that

∥ ∥ Using the triangle inequality after adding and subtracting D pop ‖D pop ‖ , it follows that

Hence, combined with Equation (12) this concludes the current part of the proof:

We turn our attention to the sufficient condition on the entanglements of D emp . Suppose that QE ( D emp ; K ) ≤ glyph[epsilon1] 2 32 N -48 · ln( R ) for all canonical partitions ( K , K c ) ∈ C N . Invoking Corollary 1 while viewing D emp as the population data tensor for the uniform data distribution over the training instances {( ( x (1 ,m ) , . . . , x ( N,m ) ) , y ( m ) )} M m =1 , we get that

Combined with Equation (12), and by the triangle inequality, we conclude:

Lastly, the fact that the entanglements of D emp can be evaluated efficiently is discussed in Appendix F.

M.7 Proof of Theorem 3

If A = 0 the theorem is trivial, since then QE ( A ; µ ( K )) = 0 for all ( K , K c ) ∈ C P N , so we can assume A glyph[negationslash] = 0 . We have:

Now, let ˆ A := A ‖A‖ and ˆ W P TN = W P TN ‖W P TN ‖ be normalized versions of A and W P TN , respectively, and let c = 2 glyph[epsilon1] ‖ A ‖ . Note that c < 2 ‖A‖ 4 1 ‖A‖ = 1 2 , and therefore by Lemma 9 we have:

and therefore

Substituting c = 2 glyph[epsilon1] ‖ A ‖ and invoking the elementary inequality H b ( x ) ≤ 2 √ x we obtain as required.

As for Equation (7), let A ′ ∈ R D 1 ×···× D NP be drawn according to the uniform distribution over the set of unit norm tensors. According to [52], for any canonical partition ( K , K c ) ∈ C P N it holds that:

where we denote D ′ µ ( K ) := max { ∏ n ∈ µ ( K ) D n , ∏ n ∈ µ ( K ) c D n } and recall that D µ ( K ) := min { ∏ n ∈ µ ( K ) D n , ∏ n ∈ µ ( K ) c D n } . By the elementary inequality of ∑ p k = q 1 /k ≥ ln( p ) -ln( q ) we therefore have that:

Since D µ ( K ) ≤ D ′ µ ( K ) and D µ ( K ) ≥ 1 , we have that:

Thus:

M.8 Proof of Theorem 4

Let C N P be the one-dimensional canonical partitions of [ N P ] (Definition 2). Note that µ ( C P N ) := { µ ( K ) : K ∈ C P N } ⊆ C N P . For an assignment ( n K ) K∈C NP ∈ N C NP of integers to one-dimensional canonical partitions K ∈ C N P , we consider the set of tensors whose matricization with respect to each K ∈ C N P has rank at most n K . This set is also known in the literature as the set of tensors with Hierarchical Tucker (HT) rank at most ( n K ) K∈C NP ( cf. [25]). Accordingly, we denote it by:

Now, define ( n ∗ K ) K∈C NP ∈ N C NP by:

where D K := min { ∏ n ∈K D n , ∏ n ∈K c D n } . We show that for any tensor A that satisfies for all K ∈ C P N :

there exists a tensor V ∈ HT ( ( n ∗ K ) K∈C NP ) such that ‖A - V‖ ≤ glyph[epsilon1] . Consider, for each canonical partition ( K , K c ) ∈ C P N , the distribution where σ K , 1 ≥ σ K , 2 ≥ ... ≥ σ K ,D K are the singular values of J A ; µ ( K ) K (note that 1 ‖A‖ 2 ∑ j σ 2 K ,j = ‖A‖ 2 ‖ A ‖ 2 = 1 so P K is indeed a probability distribution). Denoting by H ( P K ) := E i ∼P K [ln (1 /p K ( i ))] the entropy of P K , by assumption:

for all ( K , K c ) ∈ C P N . Thus, taking a = glyph[epsilon1] 2 ‖A‖ 2 (2 N P -3) we get by Lemma 10 that there exists a subset T K ⊆ [ D K ] such that and | T K | ≤ e H ( P K ) a = e ln( R ) = R . Note that

Since this holds for any subset of cardinality at most R . Taking complements we obtain so

We can now invoke Lemma 7 (note that if µ -1 ( K ) / ∈ C P N , then the requirements of Lemma 7 are trivially fullfiled with respect to the partition ( K , K c ) since n ∗ K = D K ), which implies that there exists some W ∈ HT ( ( n ∗ K ) K∈C NP ) satisfying:

as required.

To see why this is the case, note that by Lemma 8 any tensor W ∈ HT ( ( n ∗ K ) K∈C NP ) can be represented by a (one-dimensional) locally connected tensor network with varying widths ( n ∗ K ) K∈C NP , i.e. a tensor network conforming to a perfect binary tree graph in which the lengths of inner axes are as follows: an axis corresponding to an edge that connects a node with descendant leaves indexed by K to its parent is assigned the length n ∗ K . We can obtain an equivalent representation of any such tensor as a P -dimensional locally connected tensor network (described in Appendix J.1.1) via the following procedure. For each node at level l ∈ { 0 , P, 2 P, . . . , ( L -1) P } of the tree (recall N = 2 L for L ∈ N ), contract it with its descendant nodes at levels { l +1 , . . . , l +( P -1) } . 12 This results in a new tensor network whose underlying graph is a perfect 2 P -ary tree and the remaining edges all correspond to inner axes of lengths equal to n ∗ K = R for K ∈ C P N , i.e. in a representation of W as a P -dimensional locally connected tensor network.

The proof concludes by establishing that for any tensor W ∈ HT ( ( n ∗ K ) K∈C NP ) , there exists assignment for the tensors constituting the P -dimensional locally connected tensor network (defined in Figure 6) such that it generates W .

Peer reviews:
comments: SUMMARY:
By utilizing theoretical tools from quantum physics, the authors propose that a locally connected neural network can accurately predict data if and only if the data distribution exhibits low quantum entanglement under certain feature partitions. Based on this result, they develop a preprocessing method to enhance the compatibility between data distributions and locally connected neural networks. Experimental evaluations using various datasets and models validate their findings.

STRENGTHS:
It investigates what makes a data distribution suitable for machine learning from the theories of tensor network and quantum entanglement, which provides a new perspective and tools from other fields to explore the learning theory.

WEAKNESSES:
The numerical experiments are insufficient, it is only applied to randomly arranged data instead of original data which might be more convincing to support the argument.

QUESTIONS:
1. In corollary 1(line 223), does it still work under other partitions? and what's the motivation for choosing "canonical partition"? 
2. It demonstrates that the locally connected neural network is capable of accurate prediction over distribution if only if it admits low entanglements. In the experiments, it increases the entanglement of data via randomly swapping the feature and provides the comparison between a random permutation of features and proposed methods. Why the proposed method, in the numerics (Table 1, 2), is applied to random arrangement data instead of original data? After we manually randomly arrange the data feature, the data might be meaningless which is expected to get bad performance. Thus, is it appropriate to support the argument? If I miss some thing, please correct me.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates criterion that make data distributions suitable for being accurately fit by neural networks using tools from tensor networks. Specifically, it shows that some locally connected neural networks (with polynomial activations) fit a data distribution accurately if and only if the quantum entanglement of the data tensor is sufficiently low under all canonical partitions of the axes. The argument uses the previously established fact that a certain tensor network (locally connected tensor network) is equivalent to a locally connected neural network. The main theoretical contribution is to show that low quantum entanglement across canonical partitions is a necessary and sufficient condition for an arbitrary tensor $T$ to be approximated by the locally connected tensor network $\\mathcal{W}_\\mathrm{TN}$. The problem of a locally connected neural network fitting a distribution is then identified with the corresponding data tensor being well-approximated by $\\mathcal{W}_\\mathrm{TN}$, to establish the main claim.\n\nNumerical simulations are performed to show that under random swaps of the features of standard datasets (that are known to be well fit by local neural networks), more swaps results in more entanglement and this corresponds to the predictive performance being correspondingly lower. The authors also suggest a heuristic method for improving the compatibility of data to a local neural network architecture, by searching for permutations of the axes that result in a low entanglement.\n\nSTRENGTHS:\nThe main result is compelling, providing a clean and computationally efficient measure that may be used to quantify the suitability of a data distribution to certain local neural network architectures. Despite the minor limitations (such as polynomial activations) placed on the corresponding network, this still seems to be significant progress on an important and difficult question.  The potential of systematically searching for feature permutations to improve the predictive power of locally connected networks is also an intriguing possibility.\n\nThe theoretical results seem sound and correct to the best of my knowledge, although I did not check every detail.\n\nThe numerical results do provide evidence of an increase in predictive power from searching for low entanglement feature permutations.\n\nWEAKNESSES:\n1. The presentation of the paper is rather dense in some places, which is understandable due to the breadth of material covered, but makes the exposition hard to follow in places. It would be helpful to have some intuition about how the notions defined in the paper correspond to standard ones in deep learning. For example, the definition of canonical partitions seems unmotivated at first and it may be helpful to the reader if the connection to locality is highlighted. \n\nAs a minor comment, there are a few terms that are defined once and then used in later statements without context: for example, it would be helpful to mention in Definition 2 that $N = 2^L$, and to have a reminder about $R$ near Corollary 1.\n\n2. Section 5.2.2 only presents numerical results for the proposed preprocessing pipeline against a baseline of randomly permuted features. It is not clear to me whether these improvements manifest for real data ie. data already suited to a locally connected network, with the goal being to improve predictive accuracy. It is unlikely in practice that a preprocessing step would be used to improve the performance of a model with very low predictive accuracy.\n\nQUESTIONS:\nDoes the suggested pipeline lead to any accuracy improvements for natural datasets (without randomly permuted features) and local networks that already achieve good performance on them?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the problem that which data distribution is more learnable by locally-connected neural networks such as CNN, RNN, and local-attention. The paper introduces the notation of quantum entanglement, theoretically proves and empirically verifies that the network can achieve accurate predictions if and only if the data distribution exhibits low quantum entanglement under canonical partitions of features.\n\nSTRENGTHS:\n1. This paper introduces the notation of quantum entanglement from physics, which provides new insights into the problem of which data distribution is more learnable by a specific family of neural networks.\n\n2. The paper provides a comprehensive theoretical analysis of the above problem.\n\n3. Overall speaking, the paper is easy to read and follow.\n\nWEAKNESSES:\n1. The notion of entanglement is not very intuitive. Although the authors formally introduce the mathematical definition of quantum entanglement, it is still not clear what characteristics a data distribution would exhibit if the distribution has low entanglement. I suggest the authors illustrate this low-entanglement property on toy datasets and/or textual data. This intuitive illustration might provide valuable insights to the machine learning community.\n\n2. The authors are encouraged to conduct comprehensive experiments on more architectures and datasets. For example, for image classification, it is suggested to test on classical CNN architectures such as AlexNet, VGG, and ResNet. Besides, textual data should also be investigated.\n\nQUESTIONS:\n1. In Theorem 1, the authors derive an upper bound for quantum entanglement. However, it is not clear whether this upper bound truly ensures *low* entanglement. There are mainly two terms in the bound, $\\ln(R)$ and $\\ln(D_\\mathcal{K})$. The magnitude of the first term is already addressed by the authors, but it is not clear whether the second term can be extremely large. The authors are encouraged to have a discussion on this issue and show that this upper bound is indeed much smaller than the entanglement of a random tensor (as a baseline).\n\n2. How will the network architecture affect the main theoretical results in Section 3.1, 4.1, and 4.2? For example, I wonder if the results still hold if the neural network has skip connections and batch normalization operations.\n\n3. What does feature (re)arrangement mean in Section 5.1? Let us take textual data as an example and consider a sentence with N words. Does a feature rearrangement mean that the words (as well as their embeddings) in the sentence are randomly permuted? The authors are suggested to make this clearer, and better with examples.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the representation power of locally connected neural networks, a prevalent family of deep learning architectures, using tools from quantum physics. In particular, following the established equivalence between locally connected neural network and locally connected tensor network, the authors showed that the necessary and sufficient condition of a locally connected neural network fitting an objective tensor is the entanglements of the tensor admits under canonical partitions being small enough. As an application, the authors discuss the condition of making accurate predictions using locally connected neural networks, accompanied by empirical demonstrations. Notably, the findings suggest that the representation power of locally connected neural networks can potentially be enhanced by reorganizing the features to achieve reduced entanglements under canonical partitions.\n\nSTRENGTHS:\nThe authors successfully extend the connection between deep learning and quantum physics by revealing a compelling link between representation power and entanglement – two fundamental concepts in both domains. From my perspective, the results are not only theoretically elegant but also closely related to illustrations of practical situations.\n\nWEAKNESSES:\nThe paper's presentation could benefit from further improvement, especially in providing more qualitative discussions to enhance readers' intuition regarding the technical correspondence between representation power and entanglement. While the link between these two concepts is established, the paper lacks sufficient qualitative explanations to make the connection more accessible.\n\nFurthermore, the tensor network's width, denoted as $R$ in the paper, seems to be crucial in the derived bounds. However, the definition of this parameter remains somewhat unclear, so I'm not fully convinced on why it is usually small.\n\nFurthermore, the current results solely focus on the representation of tensors using locally connected neural networks. It would be good to include an explanation as to why other types of underlying functions for representation were not considered.\n\nIf these questions are addressed, I'm happy to further increase my evaluation.\n\nQUESTIONS:\nI don't have more questions other than the existing ones above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe fundamental question of what makes a data distribution suitable for deep learning is addressed in this study, focusing on locally connected neural networks. The study uses theoretical tools from quantum physics to tackle this problem. The main theoretical finding is that a specific type of locally connected neural network can accurately predict over a data distribution if, and only if, the data distribution shows low quantum entanglement under specific feature partitions. The study suggests that using quantum entanglement could inspire further use of physics tools to understand the relationship between deep learning and real-world data.\n\nSTRENGTHS:\n- The paper technically sounds correct and claims well supported by theoretical analysis and experimental results.\n- Related works and background knowledge are covered and discussed.\n- Experiments are conducted extensively in different locally connected neural networks (CNN, RNN, local self-attention), and experimental results are thoroughly discussed.\n\nWEAKNESSES:\n- Need more discussion on the motivation of using quantum entanglement.\n- Need more discussion on the limitations of this study.\n\nQUESTIONS:\n- The whole study targtes at locally connected neural networks. What happens to a NN with high connectivity? Does entanglement help the learning task?\n- Entanglement entropy is a typical metric. But what does entanglement mean in the dataset? If it is used to quantify the temporal/spatial non-locality in the data, why not use other metrics such as autocorrelation or other metrics from information theory? Why choose entanglement specifically?\n- Why use a tensor network as the equivalent model instead of directly evaluating on locally connected neural networks?\n- Will the code be available to reproduce the findings?\n\nLIMITATIONS:\nNot found.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}