SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition 4. Consider the classification setting of Appendix J.2.1, and let  $\delta \in (0,1)$  and  $\gamma > 0$ . If the training set size  $M$  satisfies  $M \geq \frac{2\ln\left(\frac{2}{\delta}\right)}{\|\mathcal{D}_{pop}^{P}\|^{2}\gamma^{2}}$ , then with probability at least  $1 - \delta$ :
$$
\left\| \frac {\mathcal {D} _ {p o p} ^ {P}}{\| \mathcal {D} _ {p o p} ^ {P} \|} - \frac {\mathcal {D} _ {e m p} ^ {P}}{\| \mathcal {D} _ {e m p} ^ {P} \|} \right\| \leq \gamma
$$
Proof. The claim is established by following steps identical to those in the proof of Proposition 2.  $\square$
Corollary 4. Consider the setting and notation of Corollary 3, with  $\epsilon \in (0,\frac{1}{6} ]$ . For  $\delta \in (0,1)$ , suppose that the training set size  $M$  satisfies  $M\geq \frac{8\ln(\frac{2}{\delta})}{\|\mathcal{D}_{pop}^{P}\|^{2}\epsilon^{2}}$ . Then, with probability at least  $1 - \delta$  the following hold. First, if there exists a canonical partition  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N^P$  (Definition 6) under which  $QE\bigl (\mathcal{D}_{emp}^{P};\mu (\mathcal{K})\bigr) > \ln (R) + 3\epsilon \cdot \ln (D_{\mu (\mathcal{K})}) + 2\sqrt{3\epsilon}$ , then:
Second, if for all  $(\mathcal{K},\mathcal{K}^c)\in \mathcal{C}_N^P$  it holds that  $QE\big(\mathcal{D}_{emp}^{P};\mu (\mathcal{K})\big)\leq \frac{\epsilon^{2}}{32N^{P} - 48}\cdot \ln (R),$  then:
Moreover, the conditions above on the entanglements of  $\mathcal{D}_{emp}^{P}$  can be evaluated efficiently (in  $\mathcal{O}(DN^PM^2 +N^PM^3)$  time  $\mathcal{O}(DN^PM + M^2)$  and memory, where  $D\coloneqq \max_{n\in [N^P]}D_n$
Proof. Implied from Corollary 3, Proposition 4 with  $\gamma = \frac{\epsilon}{2}$  and Algorithm 1 in Appendix F by following steps identical to those in the proof of Corollary 2.
Figure 8: The prediction accuracies of convolutional neural networks are inversely correlated with the entanglements of image data under canonical partitions of features, in compliance with our theory (Appendices J.1 and J.2). This figure is identical to Figure 3, except that the measurements were carried over a binary classification version of the CIFAR10 image dataset, as opposed to a one-dimensional (sequential) audio dataset, using three different convolutional neural network architectures. For further details see caption of Figure 3 as well as Appendix L.
J.2.3 Empirical Demonstration
Figure 8 extends the experiments reported by Figure 3 in Section 4.3 from one-dimensional (sequential) audio data to two-dimensional image data. Specifically, it demonstrates that the prediction accuracies of convolutional neural networks over a variant of CIFAR10 [33] is inversely correlated with the entanglements that the data admits under canonical partitions of features (Definition 6), i.e. with the entanglements of the empirical data tensor under partitions of its axes induced by canonical partitions.
J.3 Enhancing Suitability of Data to Locally Connected Neural Networks
We extend the preprocessing algorithm from Section 5, aimed to enhance the suitability of a data distribution to locally connected neural networks, from one-dimensional to  $P$ -dimensional models and data (Appendices J.3.1 and J.3.2). Empirical evaluations demonstrate that it significantly improves prediction accuracy of common architectures (Appendix J.3.3).
J.3.1 Search for Feature Arrangement With Low Entanglement Under Canonical Partitions
Suppose we have  $M \in \mathbb{N}$  training instances  $\{\{\mathbf{x}^{((n_1,\dots,n_P),m)}\}_{n_1,\dots,n_P \in [N]}, y^{(m)}\}_{m=1}^M$ , where  $\mathbf{x}^{((n_1,\dots,n_P),m)} \in \mathbb{R}^D$  and  $y^{(m)} \in \{1,-1\}$  for  $n_1,\ldots,n_P \in [N], m \in [M]$ , with  $D \in \mathbb{N}$ . For models intaking  $P$ -dimensional data, the recipe for enhancing the suitability of a data distribution to locally connected neural networks from Section 5.1 boils down to finding a permutation  $\pi: [N]^P \to [N]^P$ , which when applied to feature coordinates leads the empirical data tensor  $\mathcal{D}_{\mathrm{emp}}^P$  (Equation (8)) to admit low entanglement under canonical partitions (Definition 6).<sup>11</sup>
A greedy realization, analogous to that outlined in Section 5.1, is as follows. Initially, partition the features into  $2^{P}$  equally sized disjoint sets  $\{\mathcal{K}_{1,(k_1,\dots,k_P)}\subset [N]^P\}_{k_1,\dots,k_P\in [2]}$  such that the average of  $\mathcal{D}_{\mathrm{emp}}^P$  's entanglements induced by these sets is minimal. That is, find an element of:
$$
\operatorname *{argmin}_{\substack{\left\{\mathcal{K}^{\prime}_{k_{1},\ldots ,k_{P}}\subset [N]^{P}\right\}_{k_{1},\ldots ,k_{P}\in [2]}\\ \text{s.t. $\cup_{k_{1},\ldots ,k_{P}\in[2]}\mathcal{K}^{\prime}_{k_{1},\ldots ,k_{P}} = [N]^{P}$},\\ \forall k_{1},\ldots ,k_{P},k^{\prime}_{1},\ldots ,k^{\prime}_{P}\in [2]\mid \mathcal{K}^{\prime}_{k_{1},\ldots ,k_{P}}| = |\mathcal{K}^{\prime}_{k^{\prime}_{1},\ldots ,k^{\prime}_{P}}|}}\frac{1}{2^{P}}\sum_{k_{1},\ldots ,k_{P}\in [2]}QE\big(\mathcal{D}^{P}_{\mathrm{emp}};\mu (\mathcal{K}^{\prime}_{k_{1},\ldots ,k_{P}})\big)  ,
$$
where  $\mu :[N]^P\to [N^P ]$  is a compatible map from coordinates in  $[N]^P$  to axes indices (Definition 5). The permutation  $\pi$  will map each  $\mathcal{K}_{1,(k_1,\ldots ,k_P)}$  to coordinates  $\times_{p = 1}^{P}\{\frac{N}{2}\cdot (k_p - 1) + 1,\dots ,\frac{N}{2}\cdot k_p\}$ , for  $k_{1},\ldots ,k_{P}\in [2]$ . Then, partition similarly each of  $\{\mathcal{K}_{1,(k_1,\ldots ,k_P)}\}_{k_1,\ldots ,k_P\in [2]}$  into  $2^{P}$  equally sized disjoint sets. Continuing in the same fashion, until we reach subsets  $\{\mathcal{K}_{L,(k_1,\ldots ,k_P)}\}_{k_1,\ldots ,k_P\in [N]}$  consisting of a single feature coordinate each, fully specifies the permutation  $\pi$ .
Enhancing the suitability of a data distribution with instances of dimension different than  $P$  to  $P$ -dimensional models is possible by first arbitrarily mapping features to coordinates in  $[N]^P$ , and then following the scheme for rearranging  $P$ -dimensional data.
Algorithm 3 Enhancing Suitability of  $P$ -Dimensional Data to Locally Connected Neural Networks
1: Input:  $\mathcal{X} := \left\{\left\{\mathbf{x}^{((n_1,\dots,n_P),m)}\right\}_{n_1,\dots,n_P\in [N]}\right\}_{m=1}^M - M \in \mathbb{N}$  data instances comprising  $N^P$  features each
2: Output: Permutation  $\pi :[N]^P\to [N]^P$  to apply to feature coordinates
3: Let  $\mathcal{K}_{0,(1,\dots,1)}\coloneqq [N]^P$  and denote  $L\coloneqq \log_2(N)$
4: # We assume for simplicity that  $N$  is a power of two, otherwise one may add constant features
5: for  $l = 0, \dots, L - 1$ ,  $k_{1}, \dots, k_{P} = 1, \dots, 2^{l}$  do
6: Using a reduction to a minimum balanced  $2^{P}$ -cut problem (Proposition 5), find an approximate solution  $\{\mathcal{K}_{l+1, (2k_{1} - i_{1}, \ldots, 2k_{P} - i_{P})} \subset \mathcal{K}_{l, (k_{1}, \ldots, k_{P})}\}_{i_{1}, \ldots, i_{P} \in \{0, 1\}}$  for:
$$
\min_{\substack{\left\{\mathcal{K}_{k_{1}^{\prime}},\ldots ,k_{P}^{\prime}\subset \mathcal{K}_{l,(k_{1},\ldots ,k_{P})}\right\}_{k_{1}^{\prime}},\ldots ,k_{P}^{\prime}\in [2]\\ \text{s.t.}\cup_{k_{1}^{\prime},\ldots ,k_{P}^{\prime}\in [2]}\mathcal{K}_{k_{1}^{\prime},\ldots ,k_{P}^{\prime}} = \mathcal{K}_{l,(k_{1},\ldots ,k_{P})},\\ \forall k_{1}^{\prime},\ldots ,k_{P}^{\prime},\hat{k}_{1},\ldots ,\hat{k}_{P}\in [2]\mid \mathcal{K}_{k_{1}^{\prime},\ldots ,k_{P}^{\prime}}| = |\mathcal{K}_{\hat{k}_{1},\ldots ,\hat{k}_{P}}|}}\frac{1}{2^{P}}\sum_{k_{1}^{\prime},\ldots ,k_{P}^{\prime}\in [2]}SE\big(\mathcal{X};\mathcal{K}_{k_{1}^{\prime},\ldots ,k_{P}^{\prime}}\big)
$$
8: # At this point,  $\{\mathcal{K}_{L,(k_1,\dots,k_P)}\}_{k_1,\dots,k_P\in [N]}$  each contain a single feature coordinate
9: return  $\pi$  that maps  $(n_{1},\ldots ,n_{P})\in \mathcal{K}_{L,(k_{1},\ldots ,k_{P})}$  to  $(k_{1},\dots ,k_{P})$  , for every  $k_{1},\ldots ,k_{P}\in [N]$
As in the case of one-dimensional models and data (Section 5.1), the step lying at the heart of the above scheme — finding a balanced partition into  $2^{P}$  sets that minimizes average entanglement — is computationally prohibitive. In the next supappendix we will see that replacing entanglement with surrogate entanglement brings forth a practical implementation.
J.3.2 Practical Algorithm via Surrogate for Entanglement
To efficiently implement the scheme from Appendix J.3.1, we replace entanglement with surrogate entanglement (Definition 4), which for  $P$ -dimensional data is straightforwardly defined as follows.
Definition 8. Given a set of  $M \in \mathbb{N}$  instances  $\mathcal{X} := \left\{\left\{\mathbf{x}^{((n_1, \dots, n_P), m)} \in \mathbb{R}^D\right\}_{n_1, \dots, n_P \in [N]}\right\}_{m=1}^M$ , denote by  $p_{(n_1, \dots, n_P), (n_1', \dots, n_P')}^{}$  the multivariate Pearson correlation between features  $(n_1, \dots, n_P) \in [N]^P$  and  $(n_1', \dots, n_P') \in [N]^P$ . For  $\mathcal{K} \subseteq [N]^P$ , the surrogate entanglement of  $\mathcal{X}$  with respect to the partition  $(\mathcal{K}, \mathcal{K}^c)$ , denoted  $SE(\mathcal{X}; \mathcal{K})$ , is the sum of Pearson correlation coefficients between pairs of features, the first belonging to  $\mathcal{K}$  and the second to  $\mathcal{K}^c := [N]^P \setminus \mathcal{K}$ . That is:
$$
S E \big (\mathcal {X}; \mathcal {K} \big) = \sum_ {(n _ {1}, \dots , n _ {P}) \in \mathcal {K}, (n _ {1} ^ {\prime}, \dots , n _ {P} ^ {\prime}) \in \mathcal {K} ^ {c}} p _ {(n _ {1}, \dots , n _ {P}), (n _ {1} ^ {\prime}, \dots , n _ {P} ^ {\prime})}.
$$
Analogously to the case of one-dimensional data (Appendix G), Proposition 5 shows that replacing the entanglement with surrogate entanglement in the scheme from Appendix J.3.1 converts each search for a balanced partition minimizing average entanglement into a minimum balanced  $2^{P}$ -cut problem. Although the minimum balanced  $2^{P}$ -cut problem is NP-hard (see, e.g., [23]), similarly to the minimum balanced (2-) cut problem, it enjoys a wide array of well-established approximation implementations, particularly ones designed for large scale [29, 57]. We therefore obtain a practical algorithm for enhancing the suitability of a data distribution with  $P$ -dimensional instances to locally connected neural networks — see Algorithm 3.
Proposition 5. For any  $\bar{\mathcal{K}}\subseteq [N]^P$  of size divisible by  $2^{P}$ , the following optimization problem can be framed as a minimum balanced  $2^{P}$ -cut problem over a complete graph with  $|\bar{\mathcal{K}}|$  vertices:
$$
\min  _ {\left\{\mathcal {K} _ {k _ {1}, \dots , k _ {P}} \subset \bar {\mathcal {K}} \right\} _ {k _ {1}, \dots , k _ {P} \in [ 2 ]}} \frac {1}{2 ^ {P}} \sum_ {k _ {1}, \dots , k _ {P} \in [ 2 ]} S E \left(\mathcal {X}; \mathcal {K} _ {k _ {1}, \dots , k _ {P}}\right). \tag {10}
$$
Specifically, there exists a complete undirected weighted graph with vertices  $\bar{\mathcal{K}}$  and edge weights  $w: \bar{\mathcal{K}} \times \bar{\mathcal{K}} \to \mathbb{R}$  such that, for any partition of  $\bar{\mathcal{K}}$  into equally sized dis
Table 3: Arranging features of randomly permuted image data via Algorithm 3 significantly improves the prediction accuracies of convolutional neural networks. Reported are the results of experiments analogous to those of Table 1, carried out over the CIFAR10 image dataset (as opposed to an audio dataset) using Algorithm 3. For further details see caption of Table 1 as well as Appendix L.
Randomly Permuted	Algorithm 3	IGTD
CNN	35.1 ± 0.5	38.2 ± 0.4	36.2 ± 0.7
Table 4: Our feature rearrangement method (Algorithm 2 in Appendix G) can be efficiently applied to data with a large number of features. Reported are the results of an experiment identical to that of Table 1, but over a version of the Speech Commands [64] dataset in which every instance has 50,000 features. Instances of the minimum balanced cut problem solved as part of Algorithm 2 entail graphs with up to  $25 \cdot 10^{8}$  edges. They are solved using the well-known edge sparsification algorithm of [57] that preserves weights of cuts, allowing for configurable compute and memory consumption (the more resources are consumed, the more accurate the solution will be). As can be seen, Algorithm 2 leads to significant improvements in prediction accuracies. See Appendix L for further implementation details.
Randomly Permuted	Our Method
CNN	15.0 ± 1.6	65.6 ± 1.1
S4	18.2 ± 0.5	82.2 ± 0.4
joint sets  $\{\mathcal{K}_{k_1,\dots,k_P}\}_{k_1,\dots,k_P\in [2]}$ , the weight of the  $2^P$ -cut in the graph induced by them —  $\frac{1}{2}\sum_{k_1,\dots,k_P\in[2]}\sum_{(n_1,\dots,n_P)\in\mathcal{K}_{k_1,\dots,k_P}, (n'_1,\dots,n'_P)\in\bar{\mathcal{K}}\setminus\mathcal{K}_{k_1,\dots,k_P}}w((n_1,\dots,n_P),(n'_1,\dots,n'_P))$  — is equal, up to multiplicative and additive constants, to the term minimized in Equation (5).
Proof. Consider the complete undirected graph whose vertices are  $\bar{\mathcal{K}}$  and where the weight of an edge  $\{(n_1, \ldots, n_P), (n'_1, \ldots, n'_P)\} \in \bar{\mathcal{K}} \times \bar{\mathcal{K}}$  is:
$$
w \big (\{(n _ {1}, \ldots , n _ {P}), (n _ {1} ^ {\prime}, \ldots , n _ {P} ^ {\prime}) \} \big) = p _ {(n _ {1}, \ldots , n _ {P}), (n _ {1} ^ {\prime}, \ldots , n _ {P} ^ {\prime})}
$$
(recall that  $p_{(n_1,\dots,n_P),(n_1',\dots,n_P')}$  stands for the multivariate Pearson correlation between features  $(n_1,\ldots ,n_P)$  and  $(n_1',\ldots ,n_P')$  in  $\mathcal{X}$ ). For any partition of  $\bar{\mathcal{K}}$  into equally sized disjoint sets  $\{\mathcal{K}_{k_1,\dots,k_P}\}_{k_1,\dots,k_P\in [2]}$ , it holds that:
$$
\begin{array}{l} \frac {1}{2} \sum_ {k _ {1}, \dots , k _ {P} \in [ 2 ]} \sum_ {(n _ {1}, \dots , n _ {P}) \in \mathcal {K} _ {k _ {1}, \dots , k _ {P}}, (n _ {1} ^ {\prime}, \dots , n _ {P} ^ {\prime}) \in \bar {\mathcal {K}} \backslash \mathcal {K} _ {k _ {1}, \dots , k _ {P}}} w (\{(n _ {1}, \dots , n _ {P}), (n _ {1} ^ {\prime}, \dots , n _ {P} ^ {\prime}) \}) \\ = \frac {1}{2} \sum_ {k _ {1}, \dots , k _ {P} \in [ 2 ]} S E (\mathcal {X}; \mathcal {K} _ {k _ {1}, \dots , k _ {P}}) - \frac {1}{2} S E \big (\mathcal {X}; \bar {\mathcal {K}} \big), \\ \end{array}
$$
where  $\frac{1}{2} SE\big(\mathcal{X};\bar{\mathcal{K}}\big)$  does not depend on  $\{\mathcal{K}_{k_1,\ldots ,k_P}\}_{k_1,\ldots ,k_P\in [2]}$ . This concludes the proof.
J.3.3 Experiments
We supplement the experiments from Section 5.2 for one-dimensional models and data, by empirically evaluating Algorithm 3 using two-dimensional convolutional neural networks over randomly permuted image data. Specifically, Table 3 presents experiments analogous to those of Table 1 from Section 5.2.1 over the CIFAR10 [33] image dataset. For brevity, we defer some implementation details to Appendix L.
K Further Experiments
Table 4 supplements the experiment of Table 1 from Section 5.2.1, demonstrating that Algorithm 2 can be applied to data with a large number of features (e.g. with tens of thousands of features). In the experiment, instances of the minimum balanced cut problem solved as part of Algorithm 2 entail graphs with up to  $25 \cdot 10^{8}$  edges. They are solved using the well-known edge sparsification algorithm of [57] that preserves weights of cuts, allowing for configurable compute and memory consumption (the more resources are consumed, the more accurate the solution will be). In contrast, this approach is not directly applicable for improving the efficiency of IGTD.
Table 5: Arranging features of tabular datasets via our method (detailed in Appendix G) significantly improves the prediction accuracies of locally connected neural networks. Reported are results of experiments identical to those of Table 2, but over the "dna" tabular classification dataset [60]. For further details, see caption of Table 2 and Appendix L.
Dataset: dna
Baseline	Our Method	IGTD
CNN	82.5 ± 1.7	91.2 ± 1.1	87.4 ± 1.1
S4	86.4 ± 1.7	89.1 ± 3.7	89.9 ± 1.1
Local-Attention	79.2 ± 4.0	85.7 ± 4.5	82.7 ± 3.2
Table 5 supplements Table 2 from Section 5.2.2 by reporting results of experiments with an additional tabular benchmark — "dna".
L Further Implementation Details
We provide implementation details omitted from our experimental reports (Section 4.3, Section 5, Appendix J.2.3, Appendix J.3.3 and Appendix K). Source code for reproducing our results and figures, based on the PyTorch [44] framework, can be found at https://github.com/nmd95/data Suitable_gc_nn_code. All experiments were run on a single Nvidia RTX A6000 GPU.
L.1 Empirical Demonstration of Theoretical Analysis (Figures 3, 5, and 8)
L.1.1 Figures 3 and 5
Dataset: The SpeechCommands dataset [64] contains raw audio segments of length up to 16000, split into 84843 train and 11005 test segments. We zero-padded all audio segments to have a length of 16000 and resampled them using sinc interpolation (default PyTorch implementation). We allocated 11005 audio segments from the train set for validation, i.e. the dataset was split into 73838 train, 11005 validation and 11005 test audio segments, and created a binary one-vs-all classification version of SpeechCommands by taking all audio segments labeled by the class "33" (corresponding to the word "yes"), and sampling an equal amount of segments from the remaining classes (this process was done separately for the train, validation and test sets). The resulting balanced classification dataset had 5610 train, 846 validation and 838 test segments. Lastly, we resampled all audio segments in the dataset from 16000 to 4096 features using sinc interpolation.
Random feature swaps: Starting with the original order of features, we created increasingly "shuffled" versions of the dataset by randomly swapping the position of features. For each number of random position swaps  $k \in \{0, 250, \dots, 4000\}$ , we created ten datasets, whose features were subject to  $k$  random position swaps between features, using different random seeds.
Quantum entanglement measurement: Each reported value in the plot is the average of entanglements with respect to canonical partitions (Definition 2) corresponding to levels  $l = 1, 2, 3, 4, 5, 6$ . We used Algorithm 1 described in Appendix F on two mini-batches of size 500, randomly sampled from the train set. As customary (cf. [59]), every input feature  $x$  was embedded using the following sine-cosine scheme:
$$
\phi (x) := (\sin (\pi \theta x), \cos (\pi \theta x)) \in \mathbb {R} ^ {2},
$$
with  $\theta = 0.085$
Neural network architectures:
- CNN: An adaptation of the M5 architecture from [17], which is designed for audio data. Our implementation is based on https://github.com/danielajisafe/Audio_WaveForm_Paper_Implementation.
- S4: Official implementation of [26] with a hidden dimension of 128 and 4 layers.
- Local-Attention: Adaptation of the local-attention model from [46], as implemented in https://github.com/lucidrains/local-attention. We use a multi-layer perceptron (MLP) for mapping continuous (raw) inputs to embeddings, which are fed as input the the local-attention model. For classification, we collapse the spatial dimension of the network's output using mean-pooling and pass the result into an MLP classification head. The network had attention dimension 128 (with 2 heads of dimension 64), depth 8, hidden dimension of MLP blocks 341
(computed automatically by the library based on the attention dimension) and local-attention window size 10.
Training and evaluation: The binary cross-entropy loss was minimized via the Adam optimizer [32] with default  $\beta_{1},\beta_{2}$  coefficients. Batch sizes were chosen to be the largest powers of two that fit in the GPU memory. The duration of optimization was 300 epochs for all models (number of epochs was taken large enough such that the train loss plateaued). After the last training epoch, the model which performed best on the validation set was chosen, and its average test accuracy is reported. Additional optimization hyperparameters are provided in Table 6. We note that for S4 [26], in accordance with its official implementation, we use a cosine annealing learning rate scheduler.
Table 6: Optimization hyperparameters for the experiments of Figure 3.
Model	Optimizer	Learning Rate	Weight Decay	Batch Size
CNN	Adam	0.001	0.0001	128
S4	AdamW	0.001	0.01	64
Local-Attention	Adam	0.0005	0	32
L.1.2 Figure 8
Dataset: We created one-vs-all binary classification datasets based on CIFAR10 [33] as follows. All images were converted to grayscale using the PyTorch default implementation. Then, we allocated 7469 images from the train set for validation, i.e. the dataset was split into 42531 train, 7469 validation and 1000 test images. We took all images labeled by the class "0" (corresponding to images of airplanes), and uniformly sampled an equal amount of images from the remaining classes (this process was done separately for the train, validation and test sets). The resulting balanced classification dataset had 8506 train, 1493 validation and 2001 test images.
Random feature swaps: We created increasingly "shuffled" versions of the dataset according to the protocol described in Appendix L.1.1.
Quantum entanglement measurement: Each reported value in the plot is the average entanglements with respect to canonical partitions (Definition 6) corresponding to levels  $l = 1,2$ .
- CNN: Same architecture used for the experiments of Table 2 (see Appendix L.2.3), but with one-dimensional convolutional layers replaced with two-dimensional convolutional layers. See Table 7 for the exact architectural hyperparameters.
- VGG: Adaptation of the VGG16 architecture, as implemented in the PyTorch framework, for binary classification tasks on grayscale images. Specifically, the number of input channels is set to one and the number of output classes is set to two.
- MobileNet: Adaptation of the MobileNet V2 architecture, as implemented in the PyTorch framework, for binary classification on grayscale images. Specifically, the number of input channels is set to one and the number of output classes is set to two.
Table 7: Architectural hyperparameters for the convolutional neural network (referred to as "CNN") used in the experiments of Figure 8 and Table 3.
Hyperparameter	Value
Stride	(3, 3)
Kernel size	(3, 3)
Pooling window size	(3, 3)
Number of blocks	8
Hidden dimension	32
Training and evaluation: The binary cross-entropy loss was minimized for 150 epochs via the Adam optimizer [32] with default  $\beta_{1},\beta_{2}$  coefficients (number of epochs was taken large enough such that the train loss plateaued). Batch size was chosen to be the largest power of two that fit in the
GPU memory. After the last training epoch, the model which performed best on the validation set was chosen, and its average test accuracy is reported. Additional optimization hyperparameters are provided in Table 8.
Table 8: Optimization hyperparameters for the experiments of Figure 8.
Model	Optimizer	Learning Rate	Weight Decay	Batch Size
CNN	Adam	0.001	0.0001	128
VGG	Adam	0.0001	0.0001	64
MobileNet	Adam	0.001	0.0001	64
L.2 Enhancing Suitability of Data to Locally Connected Neural Networks (Tables 1, 2, 3, 4, and 5)
L.2.1 Feature Rearrangement Algorithms
Algorithm 2, 3 and IGTD [67] were applied to the training set. Subsequently, the learned feature rearrangement was used for both the validation and test data. In instances where three-way cross-validation was used, distinct rearrangements were learned for each separately.
Algorithm 2 and Algorithm 3: Approximate solutions to the minimum balanced cut problems were obtained using the METIS graph partitioning algorithm [29], as implemented in https://github. com/networkx/networkx-metis.
IGTD [67]: The original IGTD implementation supports rearranging data only into two-dimensional images. We adapted its implementation to support one-dimensional (sequential) data for the experiments of Tables 1 and 2.
L.2.2 Randomly Permuted Audio Datasets (Table 1)
Dataset: To facilitate efficient experimentation, we downsampled all audio segments in SpeechCommands to have 2048 features. Furthermore, for the train, validation and test sets separately, we used  $20\%$  of the audio segments available for each class.