SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lorenzo Giambagli $^{1,2}$ , Lorenzo Buffoni $^{1}$ , Lorenzo Chicchi $^{1}$ , Duccio Fanelli $^{1}$
$^{1}$ Department of Physics, University of Florence
$^{2}$ Department of Mathematics, University of Namur
{lorenzo.giambagli, lorenzo缓冲oni, lorenzo.chicchi,
duccio.fanelli}@unifi.it
In theoretical Machine Learning, the teacher-student paradigm is often employed as an effective metaphor for real-life tuition. A student network is trained on data generated by a fixed teacher network until it matches the instructor's ability to cope with the assigned task. The above scheme proves particularly relevant when the student network is overparameterized (namely, when larger layer sizes are employed) as compared to the underlying teacher network. Under these operating conditions, it is tempting to speculate that the student ability to handle the given task could be eventually stored in a sub-partition of the whole network. This latter should be to some extent reminiscent of the frozen teacher structure, according to suitable metrics, while being approximately invariant across different architectures of the student candidate network. Unfortunately, state-of-the-art conventional learning techniques could not help in identifying the existence of such an invariant subnetwork, due to the inherent degree of non-convexity that characterizes the examined problem. In this work, we take a decisive leap forward by proposing a radically different optimization scheme which builds on a spectral representation of the linear transfer of information between layers. The gradient is hence calculated with respect to both eigenvalues and eigenvectors with negligible increase in terms of computational and complexity load, as compared to standard training algorithms. Working in this framework, we could isolate a stable student substructure, that mirrors the true complexity of the teacher in terms of computing neurons, path distribution and topological attributes. When pruning unimportant nodes of the trained student, as follows a ranking that reflects the optimized eigenvalues, no degradation in the recorded performance is seen above a threshold that corresponds to the effective teacher size. The observed behavior can be pictured as a genuine second-order phase transition that bears universality traits. Code available at: https://github.com/Jamba15/Spectral-regularization-teacher-student/tree/master.
Machine learning (ML) technologies nowadays play a pivotal role in systematizing complex data by elucidating how different elements relate to each other. To this end, ML models learn from data by identifying distinctive features, which form their basis for decision-making. This task is accomplished by solving an optimisation problem that seeks to minimise a suitably defined loss function which compares expected and actual output. When operating with deep learning architectures with a feedforward structure, data supplied as input are processed via a repeated sequence of linear and non-linear transformations. The linear transformation, indeed, maps the signal from any given layer
to its immediate analogue, following the assigned neural network arrangement. Customarily, the matrix elements of the linear transformation (weights) are the target of the optimization (i.e. loss minimization). In [1] a radically new approach to this learning scheme, which anchors the process to reciprocal space, was first proposed. The training there seeks to modify the eigenvectors and eigenvalues of transfer operators in direct space. By acting on the spectral coordinates no increase in terms of computational and complexity load is produced, as compared to conventional algorithms. Remarkably, the eigenvalues yield a reliable ranking of the nodes, in terms of their contribution to the network performance. Indeed, the absolute value of the eigenvalues proves to be an excellent hallmark of the node's significance to the overall functioning of the trained model. This observation paves the way for an innovative ex-post pruning strategy to cut out unessential neurons and eventually yield smaller models, with almost identical classification abilities and thus more suitable for possible hardware deployment. Starting from these premises, here we will first deepen the understanding of the foundations of spectral learning, by focusing on the key aspects that relate to the underlying optimization protocol. We will prove that the spectral learning method - complemented with an appositely tailored regularization that favors the selection of eigenvalues with reduced magnitude - enables one to isolate an invariant trained subnetwork where the acquired ability to perform the target task is stored using a minimal amount of neurons. This latter is stably retrieved also when starting with an over-parametrized neural network, an outcome that cannot be reached when working with conventional learning algorithms formulated in direct space. To come to this conclusion, we will work under the teacher-student scheme. Concretely, we will prove that the student's ability to mimic a quenched teacher hides in a sub-portion of the whole network which consistently emerges across different architectures of the student candidate network. Remarkably, the student mirrors the teacher in terms of computing neurons, path distribution and topological attributes. As a side remark, we will also generalize the spectral scheme to deal with convoluti<onal models and show that an empirical method for structural pruning, previously proposed in [2], can be rationalized as a simplifying variant of the spectral method above.
Neural Network pruning is a relevant topic in Deep Learning [3, 4, 5]. The idea of our method is to a posteriori identify the nodes of the trained network which prove unessential for a proper functioning of the device and cut them out from the ensemble made of active units and related examples can be found in [6]. In this latter work a neuron importance score is calculated as function of the average activation and the related weights. At variance with the method that we propose, the relevance is calculated after training and by using training set related statistical quantities. As we will see no further post-training elaboration are necessary: all the information is stored in the magnitude of the optimized eigenvalues. Another quite effective approach to feature localization in the original space was proposed in [7] utilizing Group Lasso regularization. In [8], following the pioneering approach of [9], the relevance of nodes is assessed by using a Taylor expansion up to order two. It is also worth mentioning that, following the canonical pipeline Training  $\rightarrow$  Extracting Relevance Indicators  $\rightarrow$  Pruning, not always yields better performances when compared with model, of the same size of their pruned counterparts, initialized with random weights [10]. The aim of this work is not to conduct a rigorous benchmark against pre-existing pruning methods. Rather, we will prove that the spectral attributes of the neural network enable one to extract the relevant computational core for the task under inspection.
2 Spectral Parametrization of Feedforward Fully-connected layer
To establish a common understanding of the formalism, let us start by describing the linear action of a fully connected layer. Introduce  $x_{j}^{(k - 1)}$ , with  $j\in 1\dots N_{k - 1}$  and  $k\in 1\dots \ell$ , to represent the activity at the  $j$ -th neuron of layer  $k - 1$ . The linear action of the layer is expressed as a vector-matrix multiplication:  $z_{i}^{(k)} = \sum_{j}w_{ij}^{(k)}x_{j}^{(k - 1)}$  where the matrix components  $w_{ij}^{(k)}$  set the weight of the connections from node  $j$  to  $i$ . In vector notation yields:  $\mathbf{z}^{(k)} = \mathbf{w}^{(k)}\cdot \mathbf{x}^{(k - 1)}$ . We then apply a chosen element-wise non-linearity to the vector  $\mathbf{z}^{(k)}$  to obtain  $\mathbf{x}^{(k)}$  and repeat this operation for all the layers  $k\in 1\dots \ell$  to build the neural network architecture of our choice.
It is a well known interpretation [11] that this vector-matrix multiplication results in the projection of  $\mathbf{x}^{(k - 1)}$  along each feature  $\mathbf{w}_i^{(k)}$ , which corresponds to the linear activation of the corresponding
neuron (ignoring the bias addition). In this framework, one can speculate that the relevance of each feature can be assessed through a further set of scalar parameters, associated to the destination node, that modify the underlying optimization process. These latter parameters, denoted  $\lambda_i^{(k)}$  for reasons that will become clear in the following section, get initialized to one and leverage the projection along each feature  $\mathbf{w}_i^{(k)}$ . The rescaled linear transfer of information then becomes:
$$
z _ {i} ^ {(k)} = \lambda_ {i} ^ {(k)} \sum_ {j} w _ {i j} ^ {(k)} x _ {j} ^ {(k - 1)} \tag {1}
$$
or equivalently, in vector notation:
$$
\mathbf {z} ^ {(k)} = \lambda^ {(k)} \odot \left(\mathbf {w} ^ {(k)} \cdot \mathbf {x} ^ {(k - 1)}\right) \tag {2}
$$
where  $\odot$  stands for the Hadamard (or element-wise) product. As we shall prove in the following, this seemingly simple reparametrization holds unexpected capabilities and profound connections with graph theory.
To this end, we begin by noting that layer  $k$  can be viewed as a bipartite directed graph connecting the nodes in layer  $k - 1$  to those in layer  $k$ . This graph can be fully described in terms of an adjacency matrix  $A^{(k)}$ , which encodes all the relevant information on existing connections.  $A^{(k)}$  is a  $(N_{k - 1} + N_k) \times (N_{k - 1} + N_k)$  matrix. If we label the neurons belonging to layer  $k - 1$  from 1 to  $N_{k - 1}$  and those in layer  $k$  from  $N_{k} + 1$  to  $N_{k - 1} + N_{k}$ , the elements  $A_{lm}^{(k)}$  exemplify the weighted connection from node  $m$  to node  $l$ , with  $l, m \in 1 \dots N_{k - 1} + N_k$ . Due to the directed nature of the graph, the structure of  $A^{(k)}$  is lower-block diagonal and contains the values  $w_{ij}^{(k)}$  of the network layer in its nonzero terms. For a more detailed insight into the actual matrix structure, we refer to the extended discussion in the Supplementary Material. In this framework, the activity of nodes across the graph can be described through a vector, termed  $v$  for clarity, made of  $N_{k - 1} + N_k$  components, where  $v_m$  points to the activity of node  $m$ . This will refer to a neuron in layer  $k - 1$  if  $m \leq N_{k - 1}$ , or, conversely, to layer  $k$  if  $N_{k - 1} + 1 \leq m \leq N_k + N_{k - 1}$ . The structure of such vector for activities localized on layer  $k - 1$ , and before the linear transfer gets applied, is shown in Eq.(3).
The feedforward transfer from layer  $k - 1$  to layer  $k$  can be hence described as the action of the square matrix  $A^{(k)}$  on the vector  $v$ . The resulting vector components, which are connected to the transferred activity  $\mathbf{z}^{(k)}$ , will be equal to 0 for  $m \leq N_{k-1}$ . This can be expressed mathematically as:
$$
v = \left( \begin{array}{c} x _ {1} ^ {(k - 1)} \\ \vdots \\ x _ {N _ {k - 1}} ^ {(k - 1)} \\ 0 \\ \vdots \\ 0 \end{array} \right) \quad \Rightarrow \quad \left( \begin{array}{c} 0 \\ \vdots \\ 0 \\ z _ {1} ^ {(k - 1)} \\ \vdots \\ z _ {N _ {k}} ^ {(k - 1)} \end{array} \right) = A ^ {(k)} v = A ^ {(k)} \left( \begin{array}{c} x _ {1} ^ {(k - 1)} \\ \vdots \\ x _ {N _ {k - 1}} ^ {(k - 1)} \\ 0 \\ \vdots \\ 0 \end{array} \right) \tag {3}
$$
The above graph-oriented way of describing the activity paves the way to a different parametrization of the linear transfer between layers, of which Eq.(1) represents a special case. The spectral parametrization builds on the observation that another class of adjacency matrices can be written, whose action on vector  $v$  is equivalent to that illustrated above. This latter matrix  $\tilde{A}$  can be assembled starting from two other matrices: a diagonal eigenvalues matrix with entries  $\lambda_j^{(k)}$  in for the first  $N_{k-1}$  elements and  $\lambda_i^{(k)}$  out for the last  $N_k$  and one lower-block triangular eigenvectors matrix with nonzero elements  $\phi_{ij}^{(k)}$  and unitary diagonal.
This reparametrization is made possible by the fact that the feedforward action is solely encoded in the lower-block diagonal elements, and will always operate on vectors whose structure is depicted in Eq.(3). The block structure of the matrix  $\Phi^{(k)}$  is the one responsible for the feedforward behaviour of the resulting adjacency matrix. Specifically, this structure enables activity localized in the  $k - 1$  layer (i.e., vectors whose first  $N_{k - 1}$  components are non-zero) to be transferred to the  $k$ -th layer, resulting in a vector whose last  $N_{k}$  components are non-zero. Additionally, it is worth mentioning that the inverse of  $\Phi^{(k)}$  can be computed analytically and it is equal to  $(\Phi^{(k)})^{-1} = 2\mathbb{I}_{N_{k - 1} + N_k} - \Phi^{(k)}$ .
Using this property, we can explicitly express the result of the spectral composition  $\tilde{A}^{(k)} =$
$\Phi^{(k)}\Lambda^{(k)}(\Phi^{(k)})^{-1}$  as a lower-triangular matrix, whose element realizing the linear transfer of information as in Eq.(3) can be called  $\tilde{w}_{ij}$  (see Supplementary Material for more details). Previous research [1, 12, 13] has demonstrated that we can write the elements  $\tilde{w}_{ij}$  in terms of the off-diagonal elements of the eigenvector matrix  $\Phi^{(k)}$  and the diagonal eigenvalue matrix  $\Lambda^{(k)}$ , specifically:
$$
\tilde {w} _ {i j} ^ {(k)} = \left(\lambda_ {j} ^ {(k) i n} - \lambda_ {i} ^ {(k) o u t}\right) \phi_ {i j} ^ {(k)} \tag {4}
$$
The matrix  $\tilde{A}$  acts as an equivalent feedforward network, transferring  $k - 1$ -layer localized activity to the following layer through a simple vector-matrix multiplication. By substituting  $\tilde{A}$  for  $A$  in Eq.(3), we can express the linear activity of neurons in the  $k$ -th layer,  $z_{i}^{(k)}$ , in terms of the spectral parameters introduced earlier and the  $k - 1$  layer activity  $x_{j}^{(k - 1)}$ , namely:
$$
z _ {i} ^ {(k)} = \sum_ {j = 1} ^ {N _ {k - 1}} \left(\lambda_ {j} ^ {(k) i n} - \lambda_ {i} ^ {(k) o u t}\right) \phi_ {i j} ^ {(k)} x _ {j} ^ {(k - 1)} \quad \forall i \in 1 \dots N _ {k} \tag {5}
$$
or, in vector notation
$$
\mathbf {z} ^ {(k)} = \phi^ {(k)} \cdot \left(\boldsymbol {\lambda} ^ {(k) i n} \odot \mathbf {x} ^ {(k - 1)}\right) - \boldsymbol {\lambda} ^ {(k) o u t} \odot \left(\phi^ {(k)} \cdot \mathbf {x} ^ {(k - 1)}\right) \quad \forall i \in 1 \dots N _ {k} \tag {6}
$$
This simple decomposition allows for an intuitive understanding of the role of eigenvalues and eigenvectors, keeping the computational cost almost invariant, since the inverse of the eigenvector matrix  $\Phi^{(k)}$  is given analytically. It is now immediate to see that, by setting  $\lambda_j^{(k)in} = 0$  for all  $j\in 1\dots N_{k - 1}$ , we recover Eq.(1). The features can now be seen as components of the eigenvectors of the adjacency matrix describing the sub-network made by layers  $k$  and  $k - 1$ . Their relative weight, instead, can be interpreted as the magnitude of the eigenvalue  $\lambda_j^{(k)out}$ . Since the feedforward fully-connected nature of the transfer from layer  $k - 1$  to  $k$  is encoded in the structure of  $\Phi^{(k)}$ , the entries of this latter matrix enable one to resolve the relationship between different eigenvector structures and dive into the obtained network topology. In Supplementary Material, we will show how a Convolutional Neural Network (CNN) scheme can be conceptualized as a genuine eigenvalue training for a peculiar structure of  $\Phi^{(k)}$ . Here the eigenvalues gauge the relative importance of the employed convolutional filters.
As already shown in the literature (see [14]), computing the gradient with respect to the eigenvalues and eigenvectors yields a node-related set of scalars (the magnitude of the optimized eigenvalues) that can be exploited for structural pruning. In this work, we extend this idea by introducing a spectral regularization term in the loss function. Using the simplified spectral formulation of Eq.(2), the gradient will be computed with respect to the parameters  $\{\lambda^{(k)},\phi^{(k)}\}_{k\in 1\dots \ell}$  and feature localization forced by adding a  $L_{2}$  penalty in the loss function - in light of the above interpretation that identifies in  $\lambda$  a feature weight. We will show that this learning strategy yields a non-trivial effect in that it enables to consistently identify an invariant information bulk stored inside the trained network. The invariance is assessed with respect to the initial complexity of the network employed. The characteristic of this minimal computing core will be explored working in a controlled teacher-student framework where the complexity of the teacher, namely the target function, is set to be lower than the student, the optimized network.
Remarkably several topological properties of the teacher can be inferred via post-training student analysis if spectral parametrization and regularization are employed. This is at variance with the conventional training scheme, where a  $L_{2}$  regularization that aims at inter-layers connections produces sparsity effects but yields no invariant core nor topological similarities with the teacher.
3 Definition of the experimental framework
The teacher-student framework is a widely recognized concept in the field of theoretical deep learning, and has gained significant attention due to its effectiveness in knowledge distillation [15, 16, 17], and theoretical insight [18, 19, 20]. Introduced by Hinton et al.[21], it employs a teacher model to guide the learning of a student model. By using a softened probability distribution from the teacher model as labels, the student model is encouraged to imitate the teacher's behaviour.
In this work, we will use this particular setting and focus on three different learning regimes. These latter can be hierarchically ranked depending on whether the complexity of the function that the
student network learns is greater than, less than, or equal to the complexity of the teacher network function that the student is attempting to regress.
The teacher network is characterized by specific topological features that reflect its inherent structure and path distribution. As a follow-up of the analysis, we will compare these characteristics to those recovered by the student network operated under the spectral paradigm. The degree to which they are preserved in the latter will be one of the metrics that we shall use to determine the effectiveness of the learning process. To ensure simplicity and generality in our analysis, we will adopt a teacher network with two hidden layers and a scalar output. The weights that refer to the final layer are all set to one. The proposed formulation is hence inspired by the theory of soft committee machines [22, 23], but with an additional layer to increase the problem's complexity.
The dataset was generated by choosing the probability distribution of the Instance space  $p(x)$  to be a Standardized Gaussian in  $\mathbb{R}^{10}$ , namely  $p(x) = \mathcal{N}(x; \mu = 0, \Sigma = \mathbb{I})$ . In this framework, the teacher network  $T(x)$  is used as a Supervisor defining a conditioned probability distribution  $g(y|x) = \delta(y - T(x))$ . The dataset is thus composed by the collection to tuples  $\mathcal{D} = \{(x^{(i)}, y^{(i)})_{i \in 1 \dots |D|} \mid (x^{(i)}, y^{(i)}) \sim p_{data}(x,y) = p(x)g(y|x)\}$  and has a number of elements  $|D| = 1.3 \cdot 10^4$ .
The dimensions of the teacher layers are respectively set to  $10 - 20 - 20 - 1$  and are initialized according to a standard Glorot Uniform distribution [24]. The student network  $S(x)$ , on the other hand, is formed by a two hidden layers deep neural network with dimension  $10 - h - 20 - 1$  where  $h$  is a variable integer that will be tuned across experiments. The second hidden layer is taken to be a standard fully connected layer (the gradient will be computed with respect to the connections). The first hidden layer, whose dimension  $h$  can be changed at will, can be either parametrized as a fully-connected layer or as its spectral analogue (the eigenvalues  $\lambda^{(1)}$  and eigenvectors components  $\phi^{(1)}$  are thus the target of the optimization), depending on the specific aims of the analysis. Accordingly, the loss function will be different depending on the layer employed. Namely, we will use
$$
L = \frac {1}{| D |} \sum_ {i \in 1} ^ {| D |} \left(T \left(x ^ {(i)}\right) - S \left(x ^ {(i)}\right)\right) ^ {2} + \alpha_ {w} \sum_ {k \in 1, 2} L _ {2} \left(\boldsymbol {w} ^ {(k)}\right) \tag {7}
$$
for the student parametrized in the standard way, and
$$
L = \frac {1}{| D |} \sum_ {i \in 1} ^ {| D |} \left(T \left(x ^ {(i)}\right) - S _ {\lambda} \left(x ^ {(i)}\right)\right) ^ {2} + \alpha_ {\lambda} L _ {2} \left(\boldsymbol {\lambda} ^ {(1)}\right) + \alpha_ {\phi} L _ {2} \left(\boldsymbol {\phi} ^ {(1)}\right) + \alpha_ {w} L _ {2} \left(\boldsymbol {w} ^ {(2)}\right) \tag {8}
$$
for the student which bears a spectral parametrization of the first hidden layer, here denoted by  $S_{\lambda}$  for clarity. Note the  $L_{2}$  penalty term that, as discussed above, acts as a regularization.
To focus on the impact of the spectral parametrization of the layer, we will initialize  $S$  and  $S_{\lambda}$  such that the set of links is identical but still randomized. We accomplish this by setting  $\lambda_i^{(1)} = 1$  for  $i \in 1 \dots h$  and  $\phi_{ij}^{(1)} = -w_{ij}^{(1)}$  for all  $i \in 1 \dots h$  and  $j \in 1 \dots 10$ . The  $w_{ij}^{(1)}$  are sampled from a Glorot distribution, ensuring that there is no bias in the optimization due to differences in the initialization of the inter-nodes connections. For each run, we will use the minibatch stochastic gradient descent algorithm Adam, with a batch size of 300 and 500 for  $S_{\lambda}$  and  $S$ , respectively, and a total of 2000 epochs. The relevance of the features extracted by  $S$  and  $S_{\lambda}$  will be computed as the norm of the vectors that modulate the information to be conveyed at the destination nodes on the layer of variable size  $h$ . More specifically, we can extract from the conventionally parametrized student  $S$ , the following indicator bound to node  $i$ :
$$
\mathcal {W} _ {i} = \left(\sum_ {j = 1} ^ {h} w _ {i j} ^ {2}\right) ^ {1 / 2} \tag {9}
$$
For the network  $S_{\lambda}$ , on the other hand, we will employ the following expression:
$$
\mathcal {L} _ {i} = \lambda_ {i} \left(\sum_ {j = 1} ^ {h} \phi_ {i j} ^ {2}\right) ^ {1 / 2} \tag {10}
$$
which scales proportionally to the eigenvalue entry  $\lambda_{i}$ . It is clear that these quantities hold for every  $i\in 1\dots h$ , and both support the idea that the larger the  $L_{2}$  magnitude of the connections, the
more relevant the associated processing node. Notice, however, that this latter quantity needs to be appropriately rescaled for the corresponding eigenvalue's magnitude for fair comparison when operating under spectral parametrization. We speculate that by imposing a node-wise localization of the features when working under the usual parametrization results in a cumbersome exercise which cannot be easily performed. On the other hand, this goal can be instead accomplished when working under the spectral parametrization. We also speculate that the same effect could be induced on the input features when using all the spectral components, i.e. when the  $\lambda^{in}$  in Eq.(4) are not left untrained.
Formulation of the algorithm
The results presented in this paper are obtained by using an algorithm that can be summarized as follows: