SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Replace the conventional feedforward, fully-connected layers with the so called Spectral layers, where the links are parameterized as in Equation (4).
- Initialize  $\lambda^{(k)in}$  to 0 (and leave it untrained), set  $\lambda^{(k)out}$  to 1, and initialize  $\phi^{(k)}$  by using the Glorot distribution [24].
- Apply  $L_{2}$  regularization to  $\lambda^{(k)}out$  and  $\phi^{(k)}$ , and proceed with the optimization.
- Examine the entries  $\mathcal{L}_i$  for each node  $i$  from Equation (10); a larger value indicates higher node relevance.
The implementation of the Spectral layer and the Structural pruning functions can be found in https://github.com/Jamba15/SpectralTools and Supplementary Material.
In order to assess the effectiveness of the introduced parametrization and regularization, the dimension of the hidden layer  $h$  has been varied within the interval  $\mathcal{I} = [10, 20, 40, 60, 100, 200, 500, 700, 1000]$ . For each  $h \in \mathcal{I}$ , a total of 30 trials have been conducted, and train and test losses have been recorded for statistical purposes. This evaluation has been performed for both  $S$  and  $S_{\lambda}$ , which we will now denote as  $S(h)$  and  $S_{\lambda}(h)$  respectively, with  $h \in \mathcal{I}$ . The teacher function  $T(x)$  is fixed as described above and the student gets trained over 2000 epochs, to ensure a consistent and proper convergence. The average test loss, evaluated on a different set of 1000 samples distributed according to the same  $p_{data}(x,y)$ , is computed. For all  $h > 20$ , the average mean squared error (MSE) of  $S(h)$  is  $\langle \mathrm{MSE}(S(h)) \rangle = 9 \pm 1 \times 10^{-3}$  (with a one-sigma error), and the average MSE of the predictions of  $S_{\lambda}(h)$  reads  $\langle \mathrm{MSE}(S_{\lambda}(h)) \rangle = 9 \pm 3 \times 10^{-3}$ . Note that averages here are taken over the 30 repetitions of the experiment. It is worth emphasizing that these values stay stable across all choices of  $h$  above 20. This implies that both models are operated in a properly converging regime, where the two different parametrizations prove equivalent in terms of test error. Building upon this, we can now examine the properties of the network after training in light of the two employed parametrizations. To assess feature localization within the network, we generated aggregated histograms of the scalars (Eq. 9) and (Eq. 10). This aggregation was performed across all trials for a fixed hidden layer size,  $h$ . The range scanned by the aforementioned scalar entries was rescaled in such a way that both  $\mathcal{W}$  and  $\mathcal{L}$  lie in the same range [0, 1]. This enables for a meaningful comparison between the two distributions. Figure 1 demonstrates a stark difference between the empirical distributions of the two computed scalars. The blue histograms stands for (Eq. 10), which refer to the spectral attribute  $\mathcal{L}$ , while the orange histograms represent the feature norm for a conventional regularized training scheme. The latter exhibits a prototypical behavior, with the distribution slowly shifting towards zero as the size of the hidden layer increases. On the other hand, the former displays a more marked dependence on the imposed size  $h$  of the variable hidden layer. In the overparameterized  $(h > 20)$  student regime, we observe the emergence of a peak in the first bin, which corresponds to values of the feature parameter close to zero. This observation qualifies as a significant change as compared to the underparameterized regime, where the distribution only contains non-zero values of  $\mathcal{L}_i$ . On the other hand, the behaviour of  $\mathcal{W}$  remains qualitatively similar across different sizes of system being trained. The significance of the zero-peaked  $\mathcal{L}$  values
Figure 1: Histogram for the quantities (Eq. 10) (in blue) and (Eq. 9) (in orange) for the first hidden layer of the student. The scalars entries are divided by the maximum of the sample so that the two distributions lie in the interval [0, 1].
is straightforward: these are the features (i.e. the neurons) that can be safely discarded without affecting the generalization properties of the network  $S_{\lambda}(h)$ . To support this interpretation, we plot the average dimension size of the first hidden layer, considering only the non-zero features, in the left panel of Figure 2. Strikingly enough, we observe the emergence of an invariant processing core within the student, the dimension of which is closely connected to that of the teacher (in this case,  $h = 20$ ). Despite achieving the same test accuracy, the two networks ( $S$  and  $S_{\lambda}$ ) exhibit distinct topologies and levels of interpretability. The former demonstrates a sparse structure that distributes information across the entire set of connections, while the latter, obtained as a byproduct of the spectral training complemented with a suitable regularization, drives the information processing towards a minimal and identifiable subnetwork core. The additional feature weighting parameter, which can be interpreted as a particular case of a Spectral parametrization of the transfer operator, could be effectively utilized, along with regularization, to identify the Winning Lottery Ticket features within the network. We speculate that an iterative approach, similar to the one described by Frankle and Carbin [25] but focused on nodes rather than links, may be feasible, but we leave this deepening for future investigation.
Figure 2: (Left) Average dimension of the first hidden layer after the nodes associated to zero values of  $\mathcal{W}$  or  $\mathcal{L}$  have been removed. The average is taken across the 30 trials and the shaded region refers to one standard deviation. (Right) The deviation of the mean squared error (MSE) is plotted against  $n_{\lambda} - n_{T}$ . Here  $n_{\lambda}$  stands for the number of neurons that are left after pruning, while  $n_{T}$  refers to the actual size of the teacher. A phase transition-like behaviour occurs when the size of the first hidden layer of the student matches that of the teacher network. The vertical scale is logarithmic.
To ensure that the achieved conclusions are not influenced by the size of the second hidden layer, various student configurations have been tested, yielding equivalent results. Specifically, the size of the second layer has been modulated within a wide range. The size of the preceding layer and that of the corresponding layer in the teacher network are included within this range. In every case, the size of the retrieved computational core remains constant, across different initial choices of the variable hidden layer size. The residual size reflects the inherent complexity of the teacher. As such, it could be thus different from the size of the teacher's homologous layer.
To provide a complementary view, we also estimate the deviation of the mean squared error (MSE) computed after pruning (i.e. upon removing unessential nodes as ranked via  $\mathcal{L}$ ) from the corresponding value recovered at the end of training, for the full network. This latter quantity is plotted in the right panel of Figure 2 against  $n_{\lambda} - n_{T}$  where  $n_{\lambda}$  represents the number of residual neurons and  $n_{T}$  refers to the number of nodes in the teacher (here  $h_{T} = 20$ ). The invariant core exhibits remarkable robustness when neurons corresponding to low  $\mathcal{L}$  values are removed. However, as soon as the minimal bulk is perturbed, the MSE deviation starts growing regardless of the initial size of the layer, thereby revealing a universal scaling profile with respect to  $h$ . More specifically, the deviation follows an approximately exponential decrease, and the curve  $\Delta_{MSE}$  vs.  $n_{\lambda} - h_{T}$  exhibits a critical point at  $n_{\lambda} - h_{T} = 0$ , where a sudden change in the generalization performance of  $S_{\lambda}$  is observed. Our results have been confirmed with reference to other more realistic datasets. The same conclusions have been reached, both in terms of the emergence of the invariant structure and the phase-transition like behaviour. In particular we tested our framework by using (i) Shuffled Fashion MNIST (ii) Shuffled MNIST (iii) two hidden layer teacher but with heterogeneous hidden size (20-40-20-1) (iv) California Housing (v) CIFAR-100 with a pretrained ResNet50 backbone enhanced with three Spectral Layers and Batch Normalization. The corresponding results can be found in Figure 3
It is important to emphasize that the structure of the conventionally parametrized network is fundamentally different, and due to its sparsity, it is impossible to conduct a similar analysis for extracting a compact invariant subnetwork. After applying the node removal strategy to  $S_{\lambda}$ , we can further analyze the relationship between the pruned student network and the teacher network. Specifically, we investigate whether any properties of the teacher, other than its layer size, can be inferred from the analysis of the invariant core within the student network  $S_{\lambda}$ . We focus on a natural property given the feedforward structure of both networks: the path magnitude. To examine the magnitude of each path in the compacted network from a given input node to a specific neuron in the second hidden layer, we construct the expression:
$$
\tilde {w} _ {i _ {0}, i _ {1}} ^ {(1)} \cdot \tilde {w} _ {i _ {1}, i _ {2}} ^ {(2)} = \Gamma_ {i _ {0}, i _ {2}} ^ {i _ {1}}, \quad \forall i _ {0} \in 1 \dots N _ {0} = 1 0, i _ {1} \in 1 \dots h, i _ {2} \in 1 \dots N _ {2} = 2 0 \tag {11}
$$
Here,  $\tilde{w}_{i_0,i_1}^{(1)}$  represents the weight between input node  $i_0$  and hidden layer neuron  $i_1$ , and  $\tilde{w}_{i_1,i_2}^{(2)}$  represents the weight between hidden layer neuron  $i_1$  and second hidden layer neuron  $i_2$ .  $\Gamma_{i_0,i_2}^{i_1}$  then corresponds to the path from  $i_0$  to  $i_2$  passing through neuron  $i_1$ . To account for the large amount of permutation invariance in the problem, we transform the tensor  $\Gamma_{i_0,i_2}^{i_1}$  into a vector  $\vec{\Gamma}$  of size  $N_{tot} = N_0 \cdot N_1 + N_1 \cdot N_2$ , where the components  $\Gamma_i$  range from 0 to  $N_{tot}$ .
Next, we sort the components of the vector in ascending order and plot them for the teacher network in orange, and for the student network in blue and green (Figure 4). The values are plotted with respect to fraction of the vector components obtained by mapping the index of the sorted vector between zero and one, enabling a meaningful comparison between the networks (due to different number of nodes).
Upon visual inspection of panel (a) in Figure 4, it is evident that the combined effect of spectral parametrization and regularization yields the emergence of an invariant core within the network. This core can be easily retrieved and exhibits a path structure that closely aligns with that of the teacher network.
In contrast, when analyzing the sparse structure of the conventionally trained and  $L_{2}$  regularized network, a significantly different path structure is seen, as depicted in panel (b) of Figure 4. Specifically, the edges of the path magnitudes differ, as does the overall distribution.
4 Limitations and future work
While our study provides valuable insights into the regularization and pruning of neural networks, there are some limitations that must be considered. First, the experiments so far conducted employ a straightforward student network architecture. Although this choice allows for a cleaner interpretation of the underlying principles, one should repeat the analysis assuming more complex architectures to draw general conclusions. We are in the process of extending our parametrization techniques to more advanced structures so as to address this limitation. Second, our regularization and pruning scheme have only been validated on relatively simple datasets. While this demonstrates the effectiveness of our approach within a controlled setting, it leaves open the question of how well the method
Figure 3: Average dimension of the hidden layer (a-e) and effect on the  $\Delta_{MSE}$  (f-j) using the node removal strategy mentioned in the paper. The dataset presented are: Shuffled Fashion MNIST (panels  $a$  and  $f$ ), Shuffled MNIST ( $b$  and  $g$ ), two hidden layer teacher but with heterogeneous hidden size (20-40-20-1) (panels  $c$  and  $h$ ), and California Housing (on panels  $d$  and  $i$ ) and CIFAR-100 with a ResNet50 pretrained backbone (panels  $e$  and  $j$ ). In this case no reference dimension is shown. Regarding the Average dimension of the hidden layer the accuracy and loss of the Spectral and Classical students are the same for each size (within the statistical error, namely the one standard deviation variance within the 30 trials done for each 'Units' value).
generalizes to more complex, real-world datasets. Future work will involve testing the proposed techniques on larger and more reach datasets once we move to more advanced architectures. Third, in the current work, we solely focused on ranking the nodes belonging to the hidden layers with a combined optimization of eigenvalues and eigenvectors. An extension of this approach amounts to also including in the analysis neurons associated to the input space. This latter could provide valuable insights on key relevant features, particularly when dealing with high-dimensional or complex data.
(a) Path comparison between compressed (Pruned) Spectral student (blue) and the teacher (orange).
(b) Path comparison between the full Direct space trained (green) and the teacher (orange).
Figure 4: Comparison of the path magnitude between the input and second hidden layer of the teacher, displayed in orange, and those obtained for every other case study here considered. The first hidden layer in the student network is set to 200 but equivalent results hold for every other  $h \in \mathcal{I}$ . In panel (a) the first hidden Spectral layer is compressed by removing the nodes that populate with the peak of the normalized distribution of  $\mathcal{L}$  reaching  $n_{\lambda} = 20$  neurons. In panel (b) no compression is done. As can be seen the correct distribution cannot be retrieved via the usual Direct approach.
In summary, our study illustrates the advantages of a novel approach to Deep Neural Networks (DNNs) training which is anchored on the spectral domain. The eigenvector and eigenvalues of the underlying linear transfer operators are the target of the optimization. The proposed method weights feature relevance in each layer through a dedicated indicator which scales as the eigenvalues. When regularized, this latter quantity enables us to identify a minimal, efficient subnetwork within the original structure that retains full network performance upon pruning. Tested using the teacher-student paradigm, we found that this reduced network recovers the same topological features as the original teacher network. This approach outperforms conventional weight decay and classical parametrization, opening up new possibilities for the optimization and scalability of DNNs especially when working in the overparametrized regime.
[23] B. Aubin, A. Maillard, J. Barbier, F. Krzakala, N. Macris, and L. Zdeborova, “The committee machine: computational to statistical gaps in learning a two-layers neural network,” Journal of Statistical Mechanics: Theory and Experiment, vol. 2019, p. 124023, Dec 2019.
[24] X. Glorot and Y. Bengio, "Understanding the difficulty of training deep feedforward neural networks," PMLR, p. 249-256, Mar 2010.
[25] J. Frankle and M. Carbin, "The lottery ticket hypothesis: Finding sparse, trainable neural networks," ICLR Conference, Dec 2018.
Supplementary Material - How a Student becomes a Teacher: learning and forgetting through Spectral methods
A Spectral Reparametrization
We will give here some more details about the spectral parametrization, with specific regards to the construction of the underlying adjacency matrices. To this end, we begin by noting that layer  $k$  of a fully connected neural network, as noted in the main text, can be viewed as a bipartite directed graph connecting the nodes in layer  $k - 1$  to those in layer  $k$ . This graph can be fully described in terms of an adjacency matrix  $A^{(k)}$ , which encodes all the relevant information on existing connections.  $A^{(k)}$  is a  $N_{k-1} + N_k \times N_{k-1} + N_k$  matrix. If we label the neurons belonging to layer  $k - 1$  from 1 to  $N_{k-1}$  and those in layer  $k$  from  $N_k + 1$  to  $N_{k-1} + N_k$ , the elements  $A_{lm}^{(k)}$  exemplify the weighted connection from node  $m$  to node  $l$ , with  $l, m \in 1 \ldots N_{k-1} + N_k$ .
Due to the directed nature of the graph, the structure of  $A^{(k)}$  is lower-block diagonal and can be represented as:
$$
A ^ {(k)} = \left( \begin{array}{c c c c c} 0 & & \dots & & 0 \\ \vdots & & & & \\ 0 & & \dots & 0 & 0 \\ w _ {1 1} ^ {(k)} & \dots & w _ {1 N _ {k - 1}} ^ {(k)} & \vdots & \vdots \\ \vdots & & \vdots & & \\ w _ {N _ {k} 1} ^ {(k)} & \dots & w _ {N _ {k} N _ {k - 1}} ^ {(k)} & 0 & \dots 0 \end{array} \right) \tag {A.1}
$$
In this framework, the activity of nodes across the graph can be described through a vector, termed  $v$  for clarity, made of  $N_{k-1} + N_k$  components, where  $v_m$  points to the activity of node  $m$ . This will refer to a neuron in layer  $k-1$  if  $m \leq N_{k-1}$ , or, conversely, to layer  $k$  if  $N_{k-1} + 1 \leq m \leq N_k + N_{k-1}$ . The structure of such vector for activities localized on layer  $k-1$ , and before the linear transfer gets applied, is shown in Eq.(A.2).
The feedforward transfer from layer  $k - 1$  to layer  $k$  can be hence described as the action of the square matrix  $A^{(k)}$  on the vector  $v$ . The resulting vector components, which are connected to the transferred activity  $\mathbf{z}^{(k)}$ , will be identically equal to zero for  $m \leq N_{k-1}$ . This can be expressed mathematically as:
$$
v = \left( \begin{array}{c} x _ {1} ^ {(k - 1)} \\ \vdots \\ x _ {N _ {k - 1}} ^ {(k - 1)} \\ 0 \\ \vdots \\ 0 \end{array} \right) \quad \Rightarrow \quad \left( \begin{array}{c} 0 \\ \vdots \\ 0 \\ z _ {1} ^ {(k - 1)} \\ \vdots \\ z _ {N _ {k}} ^ {(k - 1)} \end{array} \right) = A ^ {(k)} v = A ^ {(k)} \left( \begin{array}{c} x _ {1} ^ {(k - 1)} \\ \vdots \\ x _ {N _ {k - 1}} ^ {(k - 1)} \\ 0 \\ \vdots \\ 0 \end{array} \right) \tag {A.2}
$$
As explained in the main text, the above graph-oriented way of describing the activity paves the way to a different parametrization of the linear transfer between layers.
The spectral parametrization builds on the observation that another class of adjacency matrices can be written, whose action on vector  $v$  is equivalent to that illustrated above. This latter matrix can be assembled starting from two other matrices: a diagonal eigenvalue matrix and one lower-block triangular eigenvector matrix.
This reparametrization is made possible by the fact that the feedforward action is solely encoded in the lower-block diagonal elements, and will always operate on vectors whose structure is depicted in
Eq.(A.2). Take then two matrices, denoted respectively  $\Phi^{(k)}$  and  $\Lambda^{(k)}$ , with the following structure:
$$
\Phi^ {(k)} = \left( \begin{array}{c c c c c} 1 & & \dots & & 0 \\ \vdots & \ddots & & & \\ 0 & & 1 & 0 & 0 \\ \phi_ {1 1} ^ {(k)} & \dots & \phi_ {1 N _ {k - 1}} ^ {(k)} & 1 & \vdots \\ \vdots & & \vdots & 0 & \ddots & 0 \\ \phi_ {N _ {k} 1} ^ {(k)} & \dots & \phi_ {N _ {k} N _ {k - 1}} ^ {(k)} & 0 & \dots & 1 \end{array} \right) \quad \Lambda^ {(k)} = \left( \begin{array}{c c c c c} \lambda_ {1} ^ {(k) i n} & 0 & & \dots & 0 \\ 0 & \ddots & & & \\ & & \lambda_ {N _ {k - 1}} ^ {(k) i n} & 0 & 0 \\ \vdots & & & \lambda_ {1} ^ {(k) o u t} & \vdots \\ & & & 0 & \ddots \\ 0 & & \dots & 0 & \dots \\ & & & & \lambda_ {N _ {k}} ^ {(k) o u t} \end{array} \right)
$$
the block structure of matrix  $\Phi^{(k)}$  is the one responsible for the feedforward arrangement of the resulting adjacency matrix. Specifically, this structure enables activity localized in the  $k - 1$  layer (i.e., vectors whose first  $N_{k - 1}$  components are non-zero) to be transferred to the  $k$ -th layer, resulting in a vector whose last  $N_{k}$  components are non-zero. Additionally, it is worth mentioning that the inverse of  $\Phi^{(k)}$  can be computed analytically and it is equal to  $(\Phi^{(k)})^{-1} = 2\mathbb{I}_{N_{k - 1}\times N_k} - \Phi^{(k)}$ .
Using this property, we can explicitly express the result of the spectral composition  $\tilde{A}^{(k)} = \Phi^{(k)}\Lambda^{(k)}(\Phi^{(k)})^{-1}$  as a lower-triangular matrix, shown in Eq.(A.4).
$$
\tilde {A} ^ {(k)} = \left( \begin{array}{c c c c c} \lambda_ {1} ^ {(k) i n} & & \dots & & 0 \\ \vdots & \ddots & & & \\ 0 & & \lambda_ {N _ {k - 1}} ^ {(k) i n} & 0 & 0 \\ \tilde {w} _ {1 1} ^ {(k)} & \dots & \tilde {w} _ {1 N _ {k - 1}} ^ {(k)} & \lambda_ {1} ^ {(k) o u t} & \vdots \\ \vdots & & \vdots & 0 & \ddots \\ \tilde {w} _ {N _ {k} 1} ^ {(k)} & \dots & \tilde {w} _ {N _ {k} N _ {k - 1}} ^ {(k)} & 0 & \dots \end{array} \right) \tag {A.4}
$$
We can the conclude that the action of this matrix on vector  $v$  is analogous to that exemplified in Eq.(A.1) provided the weights  $\tilde{w}_{ij}^{(k)}$  are parametrized according to the spectral recipe as introduced in the main text.
B Generalization to Convolutions
In the following, we will show how the spectral parametrization can be extended to a Convolutional Neural Network (CNN) Layer. In order to do that we shall first describe the CNN action as a classical vector matrix multiplication. Working with CNNs, once the size of the filter is fixed, one needs to specify (i) the  $x,y$  striding  $(s_{x,y})$ , i.e. how much each filter is shifted in each direction during the convolution, and (ii) the  $x,y$  padding  $(p_{x,y})$  i.e. how many zeros are added at each side of the input so that the convolution starts and ends with a given "offset". More specifically focus on the notation introduced Figure B.1. We are aware of the fact that we are showing a simplified case of the general convolutions, where also different channels are present. However the underlying math is basically the same and, for a matter of clarity, only this simplified yet general case is presented. The same operation can be framed in terms of a matrix-vector multiplication as soon as every parameter is fixed. To this end fix the padding to 0 and the striding to 1, without loss of generality. The convolution of the input  $x^0$  with a filter  $w$  can be written as the action of a matrix  $\mathcal{M}(w)$  on the input written with an equivalent vector notation.  $\mathcal{M}(w)$  is a matrix with a peculiar structure called Toeplitz matrix. A paradigmatic example is shown for the aforementioned choice of parameters in Figure B.2 We point out that this formulation is nothing but the fully connected representation of a Convolutional layer. In the following, we will show how, the above representation opens up the perspective to a spectral interpretation of the Convolutional layer.
B.1 Convolution parameters are eigenvalues
The first thing to point out is that convolutions are already formulated in what we could call 'reciprocal space'. Let us consider the transformation in Figure B.2. First of all, by a proper duplication of
Figure B.1: Setting the notation for dealing with CNN. In yellow the input rectangular matrix and in purple the filter structure.
Figure B.2: The figure shows how a convolution operation of Figure B.1 can be translated into a matrix-vector product via an opportune remapping of the filter into a Toeplitz matrix and a rearrangement of the rectangular image into a column vector. This is equivalent to represent the single Convolutional layer (with a single filter) as a Feed-forward fully connected one, the weight of the latter given by  $\mathcal{M}(w)$ .
the input components the structure of matrix  $\mathcal{M}(w)$  can be simplified into one with only a single non-zero value per column.