How a Student becomes a Teacher: learning and forgetting through Spectral methods
Authors: Lorenzo Giambagli
Lorenzo Buffoni
Lorenzo Chicchi
Duccio Fanelli
Venue: NeurIPS 2023 poster (2023)
Keywords: Network Slimming
Spectral Analysis
Node Pruning
Teacher-Student
TL;DR: In a specific task, the computational core of a feed-forward neural network can be identified using L2-regularized weight parameterization, based on the eigenvalues and eigenvectors of the network's adjacency matrix.

Abstract: In theoretical Machine Learning, the teacher-student paradigm is often employed as an effective metaphor for real-life tuition.  A student network is trained on data generated by a fixed teacher network until it matches the instructor’s ability to cope with the assigned task. The above scheme proves particularly relevant when the student network is overparameterized (namely, when larger layer sizes are employed) as compared to the underlying teacher network. Under these operating conditions, it is tempting to speculate that the student ability to handle the given task could be eventually stored in a sub-portion of the whole network. This latter should be to some extent reminiscent of the frozen teacher structure, according to suitable metrics, while being approximately invariant across different architectures of the student candidate network. Unfortunately, state-of-the-art conventional learning techniques could not help in identifying the existence of such an invariant subnetwork, due to the inherent degree of non-convexity that characterizes the examined problem. In this work, we take a decisive leap forward by proposing a radically different optimization scheme which builds on a spectral representation of the linear transfer of information between layers. The gradient is hence calculated with respect to both eigenvalues and eigenvectors with negligible increase in terms of computational and complexity load, as compared to standard training algorithms. Working in this framework, we could isolate a stable student substructure, that mirrors the true complexity of the teacher in terms of computing neurons, path distribution and topological attributes. When pruning unimportant nodes of the trained student, as follows a ranking that reflects the optimized eigenvalues, no degradation in the recorded performance is seen above a threshold that corresponds to the effective teacher size. The observed behavior can be pictured as a genuine second-order phase transition that bears universality traits.

How a student becomes a teacher: learning and forgetting through Spectral methods
Lorenzo Giambagli 1.2 Lorenzo Buffoni", Lorenzo Chicchi', Duccio Fanelli
Department Physics, University of Florence Department of Mathematics, University of Namur (lorenzo. gimbbgli, lorenzo buffoni, lorenzo chicchi,
faccio..ameeli)@uee
Abstract
In theoretical Machine Learning, the teacher-student paradigm is often employed as an effective metaphor for real-life tuition. A student network is trained on data generated by fixed teacher network until it matches the instructor's 's ability cope with the assigned task. The above scheme proves particularly relevant when the student network is overparameterized (namely, when larger layer sizes are employed) as compared to the underlying teacher network. Under these op- erating conditions, it is tempting to speculate that the student ability to handle the given task could be eventually stored in sub-portion of the whole network. This latter should be some extent reminiscent of the frozen teacher structure, according to suitable metrics, while being approximately invariant across different architectures of the student candidate network. Unfortunately, state-of-the-arr conventional learning techniques could not help in identifying the existence such invariant subnetwork, due to the inherent degree of non-convexity that characterizes the examined problem. In this work, we take a decisive leap for- ward by proposing radically different optimization scheme which builds spectral representation of the linear transfer of information between layers. The gradient is hence calculated with respect both eigenvalues and eigenvectors with negligible increase in terms of computational and complexity load, as compared standard training algorithms. Working in this framework, we could isolate a stable student substructure, that mirrors the true complexity of the teacher in terms of computing neurons, path distribution and topological attributes. When pruning unimportant nodes of the trained student, as follows a ranking that re- flects the optimized eigenvalues, no degradation in the recorded performance seen above threshold that corresponds to the effective teacher size. The ob- served behavior can be pictured as genuine second-order phase transition that bears universality traits. Code available at: /ttts::/github. com hammJammb11//// Spectral -aeeral-reeulle www.rrreuuaarraraeeeeeee tewchertteeherreeee sttpeeer-sstuddtttteee///eeee
1 Introduction
Machine learning (ML. technologies nowadays play a pivotal role in systematizing complex data by elucidating how different elements relate each other. To this end, MI models learn from data by identifying distinctive features, which form their basis for aaeision-makinn.. This task is accomplished by solving optimisation problem that seeks to minimise suitably defined loss function which compares expected and actual output. When operating with deep learning architectures with feedforward structure, data supplied input are processed via a repeated sequence of linear and non-linear transformations. The linear transformation, indeed, maps the signal from any given layer
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

to its immediate analogue, following the assigned neural network arrangement. Customarily, the matrix elements of the linear transformation (weights) are the target of the optimization (i.e. loss minimization). In []] radically new approach to this learning scheme, which anchors the process to reciprocal space, was first proposed. The training there seeks to modify the eigenvectors and eigenvalues of transfer operators in direct space. By acting on the spectral coordinates no increase in terms of computational and complexity load is produced, as compared to conventional algorithms. Remarkably, the eigenvalues yield a reliable ranking of the nodes, in terms of their contribution to the network performance. Indeed, the absolute value of the eigenvalues proves to be an excellent hallmark of the node's significance to the overall functioning of the trained model. This observation paves the way for an innovative ex-post pruning strategy to cut out unessential neurons and eventually yield smaller models, with almost identical classification abilities and thus more suitable for possible hardware deployment. Starting from these premises, here we will first deepen the understanding of the foundations of spectral learning, by focusing the key aspects that relate the underlying optimization protocol. We will prove that the spectral learning method complemented with an appositely tailored regularization that favors the selection of eigenvalues with reduced magnitude enables one to isolate an invariant trained subnetwork where the acquired ability to perform the target task is stored using a minimal amount of neurons. This latter is stably retrieved also when starting with an over-parametrized neural network, an outcome that cannot be reached when working with conventional learning algorithms formulated in direct space. To come to this conclusion, we will work under the teacher-student scheme. Concretely, we will prove that the student's ability to mimic a quenched teacher hides in a sub- sub-portion of the whole network which consistently emerges across different architectures of the student candidate network. Remarkably, the student mirrors the teacher in terms of computing neurons, path distribution and topological attributes. As a side remark, we will also generalize the spectral scheme to deal with convoluticonal models and show that an empirical method for structural pruning, previously proposed in 2, can be rationalized as a simplifying variant of the spectral method above.
1.1 Related Works
The idea of our method is to posteriori identify the nodes of the trained network which prove unessential for a proper functioning of the device and cut them out from the ensemble made of active units and related examples can be found in [6] In this latter work a neuron importance score is calculated as function of the average activation and the related weights. At variance with the method that we propose, the relevance is calculated after training and by using training set related statistical quantities. As we will see no further post-training elaboration are necessary: all the information is stored in the magnitude of the optimized eigenvalues. Another quite effective approach to feature localization in the original space was proposed in 7 utilizing Group Lasso regularization. In (8) following the pioneering approach of [9] the relevance of nodes is assessed by using a Taylor expansion up to order two. It is also worth mentioning that, following the canonical pipeline Training + Extracting Relevance Indicators - Pruning. not always yields better performances when compared with model, of the same size of their pruned counterparts, initialized with random weights [0O]. The aim of this work is not to conduct a rigorous benchmark against pre-existing pruning methods. Rather, we will prove that the spectral attributes of the neural network enable one to extract the relevant computational core for the task under inspection.
Neural Network pruning is a relevant topic in Deep Learning 3 4 5].
2 Spectral Parametrization of Feedforward Fully-connected layer
To establish a common understanding of the formalism, let us start by describing the linear action of a fully connected layer. Introduce 2 (k-1)) with j E 1... 1...NA-1 and A E 1...6, L, to represent the activity at the j-th neuron of layer k- - 1. The linear action of the layer is expressed as a vector-matrix multiplication: 2²(²) -, w.y x
(h) set the weight of the (k) (k-1) where the matrix components wiy
connections from node j to i. In vector notation yields: (k) = (k). (x)+1) We then apply a chosen element-wise non- cineariy to the vector (()) to obtain (k) and repeat this operation for all the layers & 1...& E to build the neural network architecture of our choice.
is a well known interpretation [I] that this vector- matrix multiplication results in the projection
(k) of x (k-1) along each feature w; which corresponds to the linear activation of the corresponding

---

neuron (ignoring the bias addition). In this framework, one can speculate that the relevance of each feature can be assessed through further set of scalar parameters, associated to the destination node, that modify the underlying optimization process. These latter parameters, denoted (k) for reasons that will become clear in the following section, get initialized one and leverage the projection along each feature w; The rescaled linear transfer of information then becomes:
(k)
(k)
W
3j
or equivalently, in vector notation:
(k) = (k)
@ (w
x 1))
(2)
where @ stands for the Hadamard (or element-wist product. As we shall prove in the following, this seemingly simple reparametrization holds unexpected capabilities and profound connections with graph theory.
To this end, we begin by noting that layer is can be viewed as a bipartite directed graph connecting the nodes in layer l - 1 to those in layer k. This graph can be fully described in terms of an adjacency matrix (k), which encodes all the relevant information on existing connections. A (k) is (N N₂) x (Nx-1* + N&) matrix. If we label the neurons belonging to layer i from 1 to N and those in layer & from N₂ + 1 to N + N2. the elements A exemplify the weighted connection from node m to node 1, with 1, m E l... N + N₁. Due to the directed nature of the graph, the structure of (R) is lower-block diagonal and contains the values wij (1)) of the network layer in nonzero terms. For more detailed insight into the actual matrix structure, we refer to the extended discussion in the Supplementary Material. In this framework, the activity of nodes across the graph can be described through a vector, termed v for clarity, made of N + Ng components, where um points to the activity of node m. This will refer to neuron in layer k - 1 if m S Nk-1- or, conversely, to layer if N + S m S N₂ + - The structure of such vector for activities localized on layer k - 1, and before the linear transfer gets applied, is shown in (3.
The feedforward transfer from layer h - 1 to layer can hence described as the action of the square matrix A (A) on the vector v. The resulting vector components, which are connected to the transferred activity (8),, will be equal to 0 for m - This can be expressed mathematically as:
(k
0
1
N
(k-1)
(k) U (k)
N
(3)
21
1)
PN
The above graph-oriented way of describing the activity paves the way to different parametriza- tion of the linear transfer between layers, of which [I. represents a special case. The spectral parametrization builds on the observation that another class of adjacency matrices can be written, whose action on vector v is equivalent that illustrated above. This latter matrix Á can be assembled starting from two other matrices: a diagonal eigenvalues matrix with entries 2/hh im for the first N
elements and X for the last N₂ and one lower-block triangular eigenvectors matrix with nonzero
(k) elements Øi and unitary diagonal.
This reparametrization is made possible by the fact that the feedforward action is solely encoded in the lower-block diagonal elements, and will always operate on vectors whose structure depicted in Eq. 3 The block structure of the matrix ((k) is the one responsible for the feedforward behaviour of the resulting adjacency matrix. Specifically, this structure enables activity localized the k layer (i.e., vectors whose first Nk-1 components are non-zero) to be transferred the k-th layer, resulting in a vector whose last N₂ components are non-zero. Additionally, it is worth mentioning that the inverse of (k) can be computed analytically and is equal to (Ø (8))-_ 2I, - -()).
Using this property, we explicitly express the result of the spectral composition A(k)
3

---

( ( as lower- lower-triangee triangular matrix, whose element realizing the linear transfer of in- formation as in Eg 3 can be called Wy (see Supplementary Material for more details). Previous research [] 12 R has demonstrated that we can write the elements Way in terms of the off-diagonal elements of the eigenvector matrix ((k) and the diagonal eigenvalue matrix A(k), specifically:
i kk 6j = (1 (k)im
(A)out) 1oi
The matrix À acts as an equivalent feedforward network, transferring k - 1-layer localized activity to the following layer through a simple vector-matrix multiplication. By substituting À for A in 3). we can express the linear activity of neurons in the k-th layer, 2!",, in terms of the spectral parameters introduced earlier and the k-
1 layer activity z (k-1), namely:
N
= Σ
(1)
(8)mn
X (()ouut 30 Ij (k-1)
EE.....
(5)
j=1
or, in vector notation
z = (A))
(A (k)in 1)) X (A)out 2 ((--1)) Ei..... Nz
(6)
This simple decomposition allows for an intuitive understanding of the role of eigenvalues and cigenvectors, keeping the computational cost almost invariant, since the inverse of the eigenvector matrix (8) is given analytically. It is now immediate to see that, by setting { 0 for all j E 1.... N we recover Eq.([). The features can now be seen as components of the eigenvectors of the adjacency matrix describing the sub-network made by layers k and k
Their relative
weight, instead, can be interpreted as the magnitude of the eigenvalue 2, (R)out Since the feedforward fully-connected nature of the transfer from layer k to k is encoded in the structure of ((A), the entries of this latter matrix enable one to resolve the relationship between different eigenvector structures and dive into the obtained network topology. In Supplementary Material, we will show how a Convolutional Neural Network (CNN) scheme can be conceptualized as a genuine eigenvalue training for a peculiar structure of (+), Here the eigenvalues gauge the relative importance of the employed convolutional filters.
As already shown in the literature (see 14] ), computing the gradient with respect to the eigenvalues
and eigenvectors yields node-related set of scalars (the magnitude of the optimized eigenvalues) that can be exploited for structural pruning. In this work, we extend this idea by introducing a spectral regularization term in the loss function. Using the simplified spectral formulation of Eq. (). the
gradient will be computed with respect to the parameters (R) (())) ) kEL.... and feature localization forced by adding L2 penalty in the loss function in light of the above interpretation that identifies in À a feature weight. We will show that this learning strategy yields a non-trivial effect in that it enables to consistently identify an invariant information bulk stored inside the trained network. The invariance is assessed with respect to the initial complexity of the network employed. The characteristic of this minimal computing core will be explored working in controlled teacher-student framework where the complexity of the teacher, namely the target function, is set to be lower than the student, the optimized network.
Remarkably several topological properties of the teacher can be inferred via post-training student analysis if spectral parametrization and regularization employed. This is at variance with the conventional training scheme, where a L2 regularization that aims at inter-layers connections produces sparsity effects but yields invariant core topological similarities with the teacher.
3 Definition of the experimental framework
The teacher-studed framework is a widely recognized concept in the field of theoretical deep learning. and has gained significant attention due its effectiveness in knowledge distillation [15] 16 17. and theoretical insight [18] 19, 20). Introduced by Hinton et al. [2]],it employs teacher model to guide the learning of a student model. By using a softened probability distribution from the teacher model as labels, the student model is encouraged to imitate the teacher's behaviour.
In this work, we will use this particular setting and focus on three different learning regimes. These latter can be hierarchically ranked depending on whether the complexity of the function that the

---

student network learns is greater than, less than, or equal to the complexity of the teacher network function that the student attempting to regress.
The teacher network is characterized by specific topological features that reflect its inherent structure and path distribution. As a follow-up of the analysis, we will compare these characteristics those recovered by the student network operated under the spectral paradigm. The degree to which they are preserved in the latter will be one of the metrics that we shall use to determine the effectiveness of the learning process. To ensure simplicity and generality in analysis, we will adopt teacher network with two hidden layers and a scalar output. The weights that refer to the final layer are all set to one. The proposed formulation is hence inspired by the theory of soft committee machines [22 231, but with an additional layer to increase the problem's 's complexity.
The dataset was generated by choosing the probability distribution of the Instance space p(æ) to be Standardized Gaussian in IR namely p(x) = Nxx; = 0, T). In this frame- work, the teacher network ((x)) is used a Supervisor defining conditioned probability dis- tribution g(ya)) = 8(y T())). The dataset is thus composed by the collection to tuples
D = ((x .y (e....p) I (xx1), .0)) ~ Paata y) = p(z)g(ylz)) and has a number of ele-
ments [D] 1.3 10
The dimensions of the teacher layers are respectively set to 10 20 20 1 and are initialized
according standard Glorot Uniform distribution (24). The student network S(x), the other hand, is formed by a two hidden layers deep neural network with dimension 10 h 20 - - where h is variable integer that will be tuned across experiments. The second hidden layer taken to be standard fully connected layer (the gradient will be computed with respect to the connections). The first hidden layer, whose dimension can be changed at will, can be either parametrized as a fully- connected layer or as its spectral analogue (the eigenvalues A (1) and eigenvectors components (1) are thus the target of the optimization), depending on the specific aims of the analysis. Accordingly, the loss function will be different depending on the layer employed. Namely, we will use
DDI
E
(T(x - ()x(1))) + aw Σ (2(
kE1,2
for the student parametrized in the standard way, and
D D
1
- - (x)2(( + a2I2(11 + a (2)
L=
(8)
EE1
for the student which bears a spectral parametrization of the first hidden layer, here denoted by Sx for clarity. Note the T22 penalty term that, as discussed above, acts as regularization. To focus on the impact of the spectral parametrization of the layer, we will initialize S and S) such
that the set of links is identical but still randomized. We accomplish this by setting x(") 1 for (1) for all E 1... and j 1... 10. The wif) are sampled from a Glorot distribution, ensuring that there is no bias in the optimization due to differences in the initialization of the inter-nodes connections. For each run, we will use the minibatch stochastic gradient descent algorithm Adam, with batch size of 300 and 500 for Sx and S, respectively, and a total of 2000 epochs. The relevance of the features extracted by S and SA will be computed as the norm of the vectors that modulate the information to be conveyed at the destination nodes the layer of variable size h. More specifically, we can extract from the conventionally parametrized student S, the following indicator bound to node é
i 1.... h and of ((())
1/2
h
W
w
(9)
For the network Sx, on the other hand, we will employ the following expression:
1/2
2- A
(10)
which scales proportionally to the eigenvalue entry It. is clear that these quantities hold for every i E 1... h. and both support the idea that the larger the Le magnitude of the connections, the
5

---

more relevant the associated processing node. Notice, however, that this latter quantity needs to be appropriately rescaled for the corresponding eigenvalue's 's magnitude for fair comparison when operating under spectral parametrization. We speculate that by imposing node-wise localization of the features when working under the usual parametrization results in a cumbersome excereise which cannot be easily performed. On the other hand, this goal can be instead accomplished when working under the spectral parametrization. We also speculate that the same effect could be induced on the input features when using all the spectral components, i.e. when the A in Eq. are not left untrained.
Formulation of the algorithm
The results presented in this paper are obtained by using an algorithm that can be summarized as follows:
Replace the conventional feedforward, fully-connected layers with the so called Spectral layers, where the links are parameterized as in Equation
Initialize (A)in to 0 (and leave it untrained), set X (k)ouu to 1, and initialize by using the
Glorot distribution [4
Apply L2 regularization to (()ouut and σf and proceed with the optimization.
Examine the entries L; for each node i from Equation (T0); larger value indicates higher node relevance.
The implementation of the Spectral layer and the Structural pruming functions can be found in pettps://github.oo. httpt//..w.a.aaaaaattttttttttrta and Supplementary Material.
Results
In order to assess the effectiveness of the introduced parametrization and regulariza- tion, the dimension of the hidden layer h has been varied within the interval I = [10, 20, 40, 60, 100, 200, 500, 700, 1000]. For each E I, a total of 30 trials have been conducted, and train and test losses have been recorded for statistical purposes. This evaluation has been performed for both and Sx. which we will now denote as ((h) and (h) respectively, with h E I. The teacher function T(x) is fixed as described above and the student gets trained over 2000 epochs, to ensure a consistent and proper convergence. The average test loss, evaluated on a dif- ferent set of 1000 samples distributed according to the same Pdota (I. y). is computed. For all h >>>>>>>>>>> 20, the average mean squared error (MSE) of (hh) (MSE(S(1)) = 1 x 10-3 (with one-sigma error), and the average MSE of the predictions of Sx (h) reads (h))) = 9 x 10-3 Note that averages here are taken over the 30 repetitions of the experiment. It is worth emphasizing that these values stay stable across all choices of h above 20. This implies that both models are operated in properly converging regime, where the two different parametrizations prove equivalent in terms of test error. Building upon this, light of the two employed parametrizations. To assess feature localization within the network, we generated aggregated histograms of the scalars (Eq. (9) and (Eq, TO). This aggregation was performed across all trials for a fixed hidden layer size, h. The range scanned by the aforementioned scalar entries rescaled in such way that both W and L lie in the same range [0, 1). This enables for a meaningful comparison between the two distributions. Figure T demonstrates a stark difference between the empirical distributions of the two computed scalars. The blue histograms stands for (Eq. 10). which refer to the spectral attribute C. while the orange histograms represent the feature norm for a conventional regularized training scheme. The latter exhibits a prototypical behavior, with the distribution slowly shifting towards zero as the size of the hidden layer increases. On the other hand, the former displays a more marked dependence on the imposed size h of the variable hidden layer. In the overparameterized (h > 20) student regime, we observe the emergence of a peak in the first bin, which corresponds to values of the feature parameter close to zero. This observation qualifies as a significant change as compared to the underparameterized regime, where the distribution only contains non-zero values of L. On the other hand, the behaviour of W remains qualitatively similar across different sizes of system being trained. The significance of the zero pero-peaked L values
now examine the properties of the network after training in

---

units: 10
units: 40
units: 500
100
Percentage
75
25
0.0
0.5
1.0
0.0
0.5
1.0 0.0
0.5
1.0
Normalized w,
Figure I: Histogram for the quantities (Eq. I0] (in blue) and (Eq, (in orange) for the first hidden layer of the student. The scalars entries are divided by the maximum of the sample so that the two distributions lies in the interval [0, I].
straightforward: these are the features (i.e. the neurons) that can be safely discarded without affecting the generalization properties of the network Sh))). To support this interpretation, we plot the average dimension size of the first hidden layer, considering only the non-zero features, in the left panel of Figure Strikingly enough, we observe the emergence of an invariant processing core within the student, the dimension of which is closely connected to that of the teacher (in this case, 20). Despite achieving the same test accuracy, the two networks (S and Sx) exhibit distinct topologies and levels of interpretability. The former demonstrates a sparse structure that distributes information across the entire set of connections, while the latter, obtained as byproduct of the spectral training complemented with suitable regularization, drives the information processing towards minimal and identifialle subnetwork core. The additional feature weighting parameter, which can be interpreted as particular case of Spectral parametrization of the transfer operator, could be effectively utilized, along with regularization, to identify the Winning Lottery Ticket features within the network. We speculate that iterative approach, similar to the one described by Frankle and Carbin [25] but focused on nodes rather than links, may be feasible, but we leave this deepening for future investigation.
Average Dimension
MSI
500
Drvet
Spectral
10
- 20
Average Dimenslon
400
Teecher Sow
10F
200
10-1
400
AwsE
800
100
20
Units
flx hr
Figure 2: (Left) Average dimension of the first hidden layer after the nodes associated to zero values of W or C have been removed. The average is taken across the 30 trials and the shaded region refer to one standard deviation. (Right) The deviation of the mean squared error (MSE) is plotted against na nr. Here y stands for the number of neurons that are left after pruning, while ny refer to the actual size of the teacher. A phase transition-liks behaviour occurs when the size of the first hidden layer of the student matches that of the teacher network. The vertical scale is logarithmic.
To ensure that the achieved conclusions are not influenced by the size of the second hidden layer, various student configurations have been tested, yielding equivalent results. Specifically, the size of the second layer has been modulated within wide range. The size of the preceding layer and that of the corresponding layer the teacher network are included within this range. every case, the size of the retrieved computational core remains constant, across different initial choices of the variable hidden layer size. The residual size reflects the inherent complexity of the teacher. As such, could be thus different from the size of the teacher's homologous layer.
7

---

To provide a complementary view, we also estimate the deviation of the mean squared error (MSE) computed after pruning (i.e. upon removing unessential nodes as ranked via C) from the corresponding value recovered at the end of training. for the full network. This latter quantity is plotted in the right panel of 2 against nı ür where n represents the number of residual neurons and nr refers to the number of nodes in the teacher (here hr 20). The invariant core exhibits remarkable robustness when neurons corresponding to low & values are removed. However, as soon as the minimal bulk is perturbed, the MSE deviation starts growing regardless of the initial size of the layer, thereby revealing a universal scaling profile with respect to h. More specifically, the deviation follows an approximately exponential decrease, and the curve AMSE vs. n hr exhibits critical point at n hr = O, where a sudden change in the generalization performance of S is observed. Our results have been confirmed with reference to other more realistic datasets. The same conclusions have been reached, both in terms of the emegerence of the invariant structure and the phase-transition like behaviour. In particular we tested our framework by using (i) Shuffied Fashion MNIST (ii) Shuffled MNIST (iii) two hidden layer teacher but with heterogeneous hidden size (20-40-20-1) (iv) California Housing (v) CIFAR- 100 with pretrained ResNet50 backbone enhanced with three Spectral Layers and Batch Normalization. The corresponding results can be found in Figure 3
It is important to emphasize that the structure of the conventionally parametrized network is funda- mentally different, and due to its sparsity, it is impossible to conduct a similar analysis for extracting compact invariant subnetwork. After applying the node removal strategy to Sx. we further analyze the relationship between the pruned student network and the teacher network. Specifically, investigate whether any properties of the teacher, other than its layer size, can be inferred from the analysis of the invariant core within the student network Sx. We focus on a natural property given the feedforward structure of both networks: the path magnitude. To examine the magnitude of each path in the compacted network from given input node to a specific neuron in the second hidden layer, we construct the expression:
(1)
u (2)
T
10.22 Vio L. No = 10, 1 1.... E... = 20
(11)
31.22
Here, w. 1o.4 represents the weight between input node in and hidden layer neuron i,, and (?)
$1.22
represents the weight between hidden layer neuron in and second hidden layer neuron ig. rhe to then corresponds to the path from in to i2 passing through neuron is To account for the large amount of permutation invariance in the problem, we transform the tensor 1 1g.12 into a vector I' of size Noor = No N1 + N₁ N2, where the components H, range from 0 to Neat-
Next, we sort the components of the vector in ascending order and plot them for the teacher network in orange, and for the student network in blue and green (Figure 4 The values are plotted with respect to fraction of the vector components obtained by mapping the index of the sorted vector between zero and one, enabling a meaningful comparison between the networks (due to different number of nodes).
Upon visual inspection of panel (a) in Figure 4 it is evident that the combined effect of spectral parametrization and regularization yields the emergence of an invariant core within the network. This core can be easily retrieved and exhibits a path structure that closely aligns with that of the teacher network.
In contrast, when analyzing the sparse structure of the conventionally trained and L-2 regularized net- work, a significantly different path structure is seen, as depicted in panel (b) of Figure Specifically, the edges of the path magnitudes differ, as does the overall distribution.
Limitations and future work
While our study provides valuable insights into the regularization and pruning of neural networks, there are some limitations that must be considered. First, the experiments so far conducted employ a straightforward student network architecture. Although this choice allows for a cleaner interpretation of the underlying principles, one should repeat the analysis assuming more complex architectures to draw general conclusions. We are in the process of extending our parametrization techniques to more advanced structures so as to address this limitation. Second, our regularization and pruning scheme have only been validated on relatively simple datasets. While this demonstrates the effectiveness of our approach within a controlled setting, it leaves open the question of how well the method
8

---

Average Dimension
Network Compression effect (AMS))
10a1
Special
50
80
10-2
100
(a)
10-3
400
800
10
20
20
10a
Spectial
140
10-2-
(g)
100
100
200
10
400
800
60-
10
10-2
(c)
10aa
80
(h)
100
10-5
400
10
150
Spectral
10
25
(d)
100
10a
50
80
10*
200
101
109
Spectral Disect
20
30
80
60
300
400
500
Units
10-1
200
300
800
10 20
90
50
n
hidden layer (a-e) and effect on the AMSE (f-i) using the node removal strategy mentioned in the paper. The dataset presented are: Shuffled Fashion MNIST (panels a and f). Shuffled MNIST (b and g). two hidden layer teacher but with heterogeneous hidden size (20-40-20-1) (panels and h), and California Housing (on panels d and i) and CIFAR- 100 with a ResNet50 pretrained backbone (panels e and j). In this case no reference dimension shown. Regarding the Average dimension of the hidden layer the accuracy and loss of the Spectral and Classical students are the same for each size (within the statistical error, namely the one standard deviation variance within the 30 trials done for each 'Units" value).
Figure 3: Average dimension
generalizes more complex, real-world datasets. Future work will involve testing the proposed techniques on larger and more reach datasets once we move to more advanced architectures. Third, the current work, we solely focused on ranking nodes belonging to the hidden layers with combined optimization of eigenvalues and eigenvectors. An extension of this approach amounts to also including in the analysis neurons associated the input space. This latter could provide valuable insights on key relevant features, particularly when dealing with high figg--iiensionall complex data.

---

Path Analysis (n, 20)
Path Analysis (n) 200)
0.2
0.15
Magnitude
0.1
Magnitudde
0.10
0.00
-0.1
-0.10
0.15
0.2
0.6
1.0
0.2
1.0
A.U.
A.U.
(a) Path comparison between compressed (Pruned) (b) Path comparison between the full Direct space Spectral student (blue) the teacher (orange).
trained (green) and the teacher (orange).
Figure 4: Comparison of the path magnitude between the input and second hidden layer of the teacher, displayed in orange, and those obtained for every other case study here considered. The first hidden layer in the student network is set to 200 but equivalent results hold for every other h I. In panel (a) the first hidden Spectral layer is compressed by removing the nodes that populate with the peak of the normalized distribution of L reaching n 20 neurons. In panel (b) compression is done. As can be seen the correct distribution cannot be retrieved via the usual Direct approach.
5 Conclusions
In summary, our study illustrates the advantages of a novel approach to Deep Neural Networks (DNNs) training which is anchored on the spectral domain. The eigenvector and eigenvalues of the underlying linear transfer operators are the target of the optimization. The proposed method weights feature relevance each layer through a dedicated indicator which scales as the eigenvalues. When regularized, this latter quantity enables to identify minimal, efficient subnetwork within the original structure that retains full network performance upon pruning. Tested using the teacher-student paradigm, we found that this reduced network recovers the same topological features the original teacher network. This approach outperforms conventional weight decay and classical parametrization, opening up new possibilities for the optimization and scalability of DNNs especially when working the overparametrized regime.
10

---

References
[1] L. Giambagli, L. Buffoni, Carletti, W. Nocentini, and D. Fanelli, "Machine learning in spectral domain," Nature Communications, vol. 12, Feb. 2021.
[2] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang, "Leeariig efficient convolutional networks through network slimming," in ICCV, 2017.
[3] T. Liang, J. Glossner, L.. Wang, S. Shi, and X. Zhang, "Pruning and quantization for deep neural network acceleration: survey," Neurocomputing, vol. 461, p. 370-403, Oct 2021.
[4] D. Elizondo and E. Fiesler, "A survey of partially connected neural networks," International Journal of Neural Systems, vol. 08, p. 535-5555 Oct 1997.
[5] Gou, B. Yu, S. J. Maybank, and D. Tao, "Knowledge distillation: survey," Int. J. Comput. Vision, vol. 129, p. 1789-1819, jun 2021.
[6] R. Yu, A. Li, C.-F. Chen, J. Lai, V. I. Morariu, X. Han, M. Gao, C.-Y. Lin, and L. S. Davis, "Nisp: Pruning networks using neuron importance score propagation." Thecvf.com, 9203, 2018.
[7] S. Scardapane, D. Comminiello, A. Hussain, and A. Uncini, "Group sparse regularization for deep neural networks," " Neurocomputiig, vol. 241, p. 81-89, Jun 2017.
[8] P. Molchanov, A. Mallya, S. Tyree, I. Frosio, and J. Kautz, "Importance estimation for neural network pruning," Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[9] Y. LeCunn, J. Denker, and S. Solla, "Optimal brain damage," Advances in Neural Information Processing Systems, 1989.
[10] Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell, "Rethinking the value of network pruning." " in International Conference Learning Representations, 2019.
[11] I. J. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Cambridge, MA, USA: MIT Press, 2016. http://wwxz dwwppeerpennnece www.weerrningbooo.....ooooo org
[12] L. Chicchi, L. Giambagli. L. Buffoni, T. Carletti, M. Ciavarella, and D. Fanelli, "Training of sparse and dense deep neural networks: Fewer parameters, same performance," Phys. Rev. E, vol. 104, 054312, Nov 2021.
[13] L. Chicchi, D. Fanelli, L. Giambagli, L. Buffoni, and T. Carletti, "Recurrent spectral network (rsn): Shaping a discrete map to reach automated classification," Chaos, Solitons & Fractals, vol. 168, p. 113128-1133
Mar 2023.
[14] L. Buffoni, E. Civitelli, L. Giambagli, L. Chicchi, and D. Fanelli, "Spectral pruning of fully connected layers." Scientific Reports, vol. 12, July 2022.
[15] N. Komodakis and S. Zagoruyko, "Paying more attention to attention: improving the perfor- mance of convolutional neural networks via attention transfer," Hal.science, Jun 2017.
[16] W. Park, D. Kim, Y. Lu, and M. Cho, "Relational knowledge distillation," Thecyf.com, p. 3967-3976, 2019.
[17] I.. Beyer, X. Zhai, A. Royer, L.. Markeeva, R. Anil, and A. Kolesnikov, Knowledge distillation: A good teacher is patient and consistent.
[18] S. Goldt, M. S. Advani, A. M. Saxe, F. Krzakala, and L. Zdeborová, "Dynamics of stochastic gradient descent for two-layer neural networks in the reacher-studend setup*." Journal of Statistical Mechanics: Theory and Experiment, vol. 2020, P. 124010, Dec 2020.
[19] H. S. Seung. H. Sompolinsky, and N. Tishby, "Statistical mechanics of learning from examples," vol. 45, p. 6056-6091 Apr 1992.
[20] F. Krzakala, F. Ricci- Tersenghrs,,,,, L. Zdeborová, R. Zecchina, E. W. Tramel, and L. F. Cugliandolo, Statistical Physics, Optimization, Inference, and Message-Passinnn Algorithms. Dec 2015.
[21] G. Hinton, O. Vinyals, and Dean, "Distilling the knowledge in a neural network," 2015.
[22] D. Saad and A. Solla, "On-line learning soft committee machines," " Physical Review E, vol. 52, p. 4225-4243, Oct 1995.

---

[23] B. Aubin, A. Maillard, J. Barbier, F. Krzakala, N. Macris, and L. Zdeborová, "The committee machine: computational statistical gaps in learning two-layers neural network," Journal of Statistical Mechanics: Theory and Experiment, vol. 2019, p. 124023, Dec 2019.
[24] X. Glorot and Y. Bengio, "Understanding the difficulty of training deep feedforward neural networks," PMER, p. 249-256, Mar 2010.
[25] J. Frankle and M. Carbin, "The lottery ticket hypothesis: Finding sparse, trainable neural networks," ICLR Conference, Dec 2018.
12

---

Supplementary Material How a Student becomes a Teacher: learning and forgetting through Spectral methods
A Spectral Reparametrization
We will give here some more details about the spectral parametrization, with specific regards the construction of the underlying adjacency matrices. To this end, we begin by noting that layer &s of fully connected neural network, as noted in the main text, can be viewed connecting the nodes in layer k - 1 to those in layer k. This graph can be fully described in terms of an adjacency matrix (k),, which encodes all the relevant information on existing connections. A Nk + N& N 1 N₂ matrix. If we label the neurons belonging layer k from to - 1 and those in layer from N& + to N + Nk. the elements (1A) exemplify the weighted connection from node m node with E 1...... N NR-
bipartite directed graph
Due to the directed nature of the graph, the structure of ( is lower-block diagonal and can be represented as:
0
(R)
(8:)
(K)
:
(A.1)
w
IN
(k)
(R)
0
N2-1
In this framework, the activity of nodes across the graph can be described through a vector, termed e for clarity, made of - components, where um points the activity of node m. This will refer to neuron in layer k - if m S. NE-1, or, conversely, to layer A if N 1 + < m S N₃ + N The structure of such vector for activities localized layer k 1, and before the linear transfer gets applied, shown in (A.2).
The feedforward transfer from layer &s 1 to layer can be hence described the action of the square matrix (k) the vector The resulting vector components, which are connected to the
transferred activity (k), will be identically equal to zero for m < Nx-1. This can be expressed mathematically as:
1
(k-1) 1)
No
(A--l
1)
A(h) v (1k)
N
(A.2)
:
(k-11 1)
0
As explained in the main text, the above graph-oriented way of describing the activity paves the way to different parametrization of the linear transfer between layers.
The spectral parametrization builds on the observation that another class of adjacency matrices can be written, whose action on vector is equivalent to that illustrated above. This latter matrix can be assembled starting from two other matrices: a diagonal eigenvalue matrix and one lower-block triangular eigenvecior matrix.
This reparametrization made possible by the fact that the feedforward action is solely encoded in the lower-block diagonal elements, and will always operate vectors whose structure is depicted in
13

---

Eq. A.2
Take then two matrices, denoted respectively ((x) and A with the following structure:
(k) sn
1
0
0
:
0
0
1
0
0
A No
(R) in
0
(())
A
Φ11
(k)
(k)
NR
1
:
A (k)
out
Ne1 N
0
0
0
oue
N2
(A.3)
the block structure of matrix ((() is the one responsible for the feedforward arrangement of the resulting adjaceney matrix. Specifically, this structure enables activity localized in the k layer (i.e., vectors whose first Nx-1 components are non-zero) to be transferred to the k-th layer, resulting in vector whose last N& components are non-zero. Additionally, it is worth mentioning that the inverse of (k) can be computed analytically and is equal to ($(1)) ( 2I 1x $(R).
Using this property, we can explicitly express the result of the spectral composition A(k) = g (8) ((b())) as a lower-triangun triangular matrix, shown in Eq.(A.4)
(k) in
Nx-1
A(A)
(A.4)
W11
(k)
INN
(k)
A
(A)
out
(k)
(k)
(k) out
Ng1
No Ne
N
We can the conclude that the action of this matrix on vector v is analogous to that exemplified in
Eq.(A.I) provided the weights w(fk) are parametrized according the spectral recipe as introduced in the main text.
B Generalization to Convolutions
In the following, we will show how the spectral parametrization can be extended to Convolutional Neural Network (CNN) Layer. In order to do that we shall first describe the CNN action as a classical vector matrix multiplication. Working with CNNs, once the size of the filter is fixed, one needs to specify (i) the æ, striding (s,.y), i.c how much each filter is shifted in each direction during the convolution, and (ii) the x, y padding (Pr.y) i.e. how many zeros are added at each side of the input so that the convolution starts and ends with given "offset". More specifically focus on the notation introduced Figure B.T. We are aware of the fact that we are showing simplified case of the general convolutions, where also different channels are present. However the underling math is basically same and, for a matter of clarity, only this simplified yet general case is presented. The same operation can be framed in terms of matrix-vector multiplication as soon as every parameter is fixed. To this end fix the padding to 0 and the striding to 1, without loss of generality. The convolution of the input x0 with a filter can be written the action of a matrix M(w) on the input written with an equivalent vector notation. M(w) a matrix with a peculiar structure called Toeplitz matrix. A paradigmatic example is shown for the aforementioned choice of parameters in Figure B.2 We point out that this formulation is nothing but the fully connected representation of a Convolutional layer. In the following, will show how, the above representation opens up the perspective spectral interpretation of the Convolutional layer.
B.1 Convolution parameters eigenvalues
The first thing to point out is that convolutions are already formulated in what we could call 'reciprocal space'. Let us consider the transformation in Figure B.2 First of all, by proper duplication of
14

---

M
M-m+P= +xx
Fg Ad 1 N
Wt
We W0's W2
N
wy Wy 10g
X
Figure B.I: Setting the notation for dealing with CNN. In yellow the input rectangular matrix and in purple the filter structure.
M
W3
0
0
0
X00
W2
Xon
R7
00g
W.X W
M(z))
Figure B.2: The figure shows how convolution operation of Figure B.1 can be translated into matrix-vector product via an opportune remapping of the filter into a Toeplitz matrix and rearrangement of the rectangular image into column vector. This is equivalent to represent the single Convolutional layer (with a single filter) as a Feed-forward fully connected one, the weight of the latter given by M(w).
the input components the structure of matrix M(w) can be simplified into with only single non-zero value per column.
By taking the structure of the matrix á as the binarization version of such simplified Toeplitz matrix, the am of the notation introduced in 12] play the exact same role of the filters degree of freedom in the CNN layer.
We recall, indeed, that the weights strength of a feedforward fully connected layer can written in terms of eigenvalues and cigenvectors of the equivalent directed bipartite graph follows:
(A, (6x)
(oat) )oi
Wij
(B.5)
The effect of (tn) can be therefore considered as column-wise product whereas the X as row-wise one. By setting (out) = each (in) can adjusted to the same value of the convolutional weight that is solely present in given column. This operation is graphically presented the bottom part of Figure B.3
B.2 Filter relevance as eigenvalue magnitude: a bridge with existing literature
Due to equation B.5) another correspondence can be elaborated upon: the matrix M(w) can be constructed using as degree of freedom the eigenvectors components diy as shown in Figure B.4 By setting A (in)
the filters action could accounted for by imposing an apposite structure of matrix Φ. The one shown in the figure is representative example of convolution with non overlapping filters after the input has been adjusted. The same conclusion can also be achieved for M(w) structure like the one depicted in Figure B.2
The remaining degree of freedom of the ((out) can the be used as measure for the filter relevance if every (out) Vi.
We posit, therefore, that the the regularization proposed in [2] is equivalent to a spectral regularization for the case of CNN with an imposed parametrization that goes as follows: every (in) = 0, Toeplitz
structure of o and X (out) X, (out) Vi,j. j.
15

---

W2
W5
o...
Xoo
10
122
X01
0
X02
X01
X02
Xg3
M(w)
(A1
A2
da
A2
Aa
Aa Ag
,g
...
1
1
0
0
1 1 1
0.. 0 ...1
0
0
1
0
0
M(
Figure B.3: In the upper part feedforward layer equivalent CNN one is shown. Input components have been duplicated in order to have single non zero component per column. In the bottom part same weight matrix is rephrased in terms of the spectral decomposition (B.5). The equivalence
can be achieved setting 21.2.... 21,2.3...
W6 O... 0
X00 W2
2000
11
12
10g
0
0
0
3002
X000
X01
X02
M(wa)
Φ
Φ1
Φ2 Φ3
O...
...99
A Az
41
Anna
M)
Figure B.4: The weight matrix rephrased in terms of the spectral parameters. this case the eigenvalues are exploited as a filter relevance proxy.
C Code availability
The python code used conduct the analysis is fully available the following GitHub reposi- tory //ttps://github. com/ Jamba16/Spectral-- reg.aaaiarrrrualaiiiiiiiiioo teachere student/tree/ master! The code allows both the retrieval of the full set of results presented in this work and the run of new experiments with different settings.
The implementation of the spectral layer and the Structural Pruning functions be found in bttps://github com/ Jambaa// hamm/Jmmaa1//Specaat /0t.///accctraoooss...... The repository, at the moment of publication, contains TensorFlow and Py Torch implementation with documentation.
16


Reviews:
[{"review_id": "BkXJre1jpS", "reviewer": "Reviewer_uE1J", "summary": "This paper tried to understand and extend a new parametrization, spectral parametrization, for fully connected networks. They empirically show that in the teacher student setup, even when the student network is highly over parametrized, the student network under that parametrization will converge to a somehow \"sparse\" network that can be compressed, using standard optimizers. And they show that standard parametrization for the student network cannot do that.", "questions": "No.", "limitations": "I don't think they discuss their limitations adequately. Here are my suggestions for improvement.\n\nSince this phenomenon on simple network structures is actually almost well understood, if you really want to show this kind of reparametrization works, you should do some larger scale experiments. If there are some larger scale experiments to show that that kind of parameterization really works in practice and helps practice problems, then I think it's a good paper.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "No.", "weaknesses": "1. Lack novelty. Why this parametrization can lead to sparsity is almost well-understood in the literature. For instance, it's well known that for such model $h_{\\theta}(x)=h(x;u\\odot v)$, if we initialize $u,v$ to be small and use standard gradient based algorithms, we will have a gradual rank increase for $u,v$, which leads to sparsity more easily. For some reference we can look at Abbe's paper https://arxiv.org/abs/2306.07042 (This is the most recent reference) or Jason's paper https://arxiv.org/abs/2207.04036. Or we can simply compute GD dynamics for using diagonal linear network to learn a linear target with small initialization. In a word, this result is not surprising to me. I feel like I already know this/ expect that happens.\n\n2.For the empirical result, the input dimension in the experiments is too low. It's only 10. We have high dimensional input in practice. Also, the experiments are insufficient. Like you can try different optimizers, try different hyperparameters (width, initialization scheme), etc.\n\n3.The authors didn't discuss the related works sufficiently. This phenomenon is clearly related to training dynamics of fully connected networks and there are many theory papers discussing the same things (even similar implication/conclusion).\n\nUpdate: since the authors update the experiments, I have changed my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the referee for pointing us to the insightful reference concerning implicit regularization resulting from the commuting nature of our parametrization. Unfortunately, the other mentioned reference came out after the submission of our work to the conference. However, we contend that the effect we describe extends beyond such implicit regularization. Specifically, we do not initialize our spectral weights to small values; instead, they are uniformly set to one in every trial. This suggests that rather than imposing a gradual increase in rank, we observe more of a decrease. While our parametrization may indeed exhibit an implicit bias towards sparse representation, it alone cannot account for the observed distribution of $\\\\lambda_i$ and consequently, $\\\\mathcal{L}_i$. \n\nIn the provided PDF, we present the distribution of $\\\\mathcal{L}_i$ (very similar to the one of $\\\\lambda_i^{(out)}$) for Shuffled Fashion MNIST. Notably, the after training distribution of the variable with explicit $L_2$ regularization is very different from the one we get whithout, relying solely on the implicit bias. The visual inspection of the image points in the direction that is the eigenvalues' sparsification, leading to feature-centric sparsification (involving node elimination rather than link elimination), which is considerably more pronounced when explicit bias is applied.\n\nRegarding the dataset's simplicity, we concur with the referee's observation and have undertaken additional experiments that we are willing to include in the manuscript. Specifically, in Figure 1 of the PDF, we showcase our analysis on four distinct datasets: Shuffled MNIST, Shuffled Fashion MNIST, California Housing, and a more intricate Teacher structure. For this Teacher configuration, the size of the first hidden layer differs from the second. We maintained a consistent student structure, and performance metrics (in terms of accuracy and loss function) for both the spectral Student and the conventional student are consistent across all datasets. We point outh that, in this case, we are not able to set an obvious complexity threshold of the teacher.\n\nThe results presented align with the simplified scenario depicted in the original version of our paper and we hope that, in light of those new findings, the referee will be keen on reconsidering the given 'Rating'."}, {"review_id": "QsVTaTP7SK", "reviewer": "Reviewer_EH9K", "summary": "This work focuses on the teacher-student paradigm in theoretical machine learning and shows that, for a unique optimization scheme that involves directly optimizing on the eigenvalues/eigenvectors of the data, a stable subnetwork in the student can be identified that can mirror the complexity of the teacher network. The area of this work is outside of my domain, so I am unable to comment further on the contributions. ", "questions": "How can the spectral method be scaled up for large datasets, like images?", "limitations": "I did not see a section dedicated to limitations of the authors' work. ", "rating": 6, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The approach seems theoretically-motivated and the result seems interesting. The area of this work is outside of my domain, so I am unable to comment further on the contributions. ", "weaknesses": "The experiments seem limited, but the area of this work is outside of my domain, so I am unable to discern what level of experimentation is normal for this kind of work. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the referee's feedback and recognize that the work may not fall directly within their domain of expertise. Regarding the implementation with other data, we agree with the referee and therefore have extended our analysis to four more complex and realistic datasets. Delving deeper, our refined analysis covers: Shuffled MNIST, Shuffled Fashion MNIST, California Housing, and an intricate Teacher structure. Within this Teacher design, the size of the first hidden layer differs from that of the second. We've maintained a consistent student framework, the only deviation being a second hidden layer size of 50. For both the spectral Student and the conventional student, performance metrics, encompassing accuracy and loss function, are uniform across all datasets.\n\nRegarding the implementation of a large image dataset in the main paper we show how the spectral parametrization can be extended to a Convolutional Layer. In this setting, the eigenvalues can be mapped into weights that leverage the relevance of every filter resulting in an already analyzed algorithm of proven effectiveness.\n\nWe are, moreover, willing to insert a dedicated Limitation section where we recognize that due to computational limitations, we haven't tested more complex models where outcomes might differ. Nevertheless, we aspire to explore such models in forthcoming research. Furthermore, we haven't fully tapped into the capabilities of spectral decomposition. Indeed, a feature relevance analysis can also be conducted in the input space.  By setting $w_{ij}=(\\lambda_j^{in}-\\lambda_i^{out})\\phi_{ij}$ in the initial layer, the eigenvalues $ \\lambda^{in} $ could highlight the significance of the input's $j$-th component.\n\nWe hope that, given these supplemental results which we believe bolster the paper's strength, the referee might re-evaluate the manuscript's current rating which we believe is a little unfair considering their assessment of inexperience with the domain."}, {"review_id": "Ehu4TjfjVS", "reviewer": "Reviewer_Cz6u", "summary": "The authors consider a knowledge distillation setting involving a teacher and a student neural network.\nThe authors exploit a few interesting tricks -especially the use of a spectral parameterization of the network, and special regularizers- to show that it is possible to enforce learning a submodule within a student network (as long as it is larger than the original teacher), that implements the teacher behavior with a minimal number of neurons, and which can thus be used to estimate the effective size of the teacher.\n\nThe authors present thorough theoretical work, along with experimental verification; the results are convincing, but the topic is outside my area of expertise.", "questions": "\nN/A\n", "limitations": "- Evaluation is performed only on small networks and on toy data. It remains to be seen how well the approach would work on complex, modern datasets and network architectures.\n\n", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is interesting and well written.\n- The results are convincing.\n", "weaknesses": "- The empirical experiments could be more extensive.\n- The dataset used is extremely simple, and it is not clear whether the method would work in more typical scenarios.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the referee for their feedback. We acknowledge the need for more comprehensive results. In response, we've conducted four additional more complex scenarios, with the outcomes presented in Figure 1 of the accompanying PDF file. The experimental framework remains consistent with the original paper, and the outcomes corroborate the preliminary findings we described. In more detail, our expanded analysis encompasses four unique datasets: Shuffled MNIST, Shuffled Fashion MNIST, California Housing, and a nuanced Teacher structure. In this Teacher model, the first hidden layer's size is distinct from the second and from the fixed one of the student. We've adhered to a uniform student structure, with the only variation being the second hidden layer's size set to 50. At variance with the showcase of the paper, however, we are not able to set a proper complexity threshold in the plots. The performance indicators, both in accuracy and loss function, for the spectral Student and the classical Student, remain consistent across all datasets. \nWe would like to stress that with those datasets that are much closer to the typical scenarios in terms of complexity and input dimension, the capability of finding the computational core of the student (panels $a-d$) and the phase transition-like behavior of the MSE (panels $e-h$) are both validated.\n\nWe trust that in light of these additional findings, which in our view enhance the manuscript's robustness, the referee might reconsider the manuscript's rating.\n\nRegarding the network architecture, we intend to address this in the _Limitations_ section of the paper and are keen to delve into more sophisticated architectures, such as Residual Networks and Transformers, in forthcoming publications."}, {"review_id": "N9NTkgeaHY", "reviewer": "Reviewer_dofn", "summary": "This paper analyses the performance of a spectral parameterisation/regularisation scheme for neural networks. After first introducing the spectral approach, the authors describe student-teacher experiments where they attempt to distil a fixed teacher network’s behaviour into a student network. The authors show that the spectral parameterisation gives equivalently good predictive performance, but that the structure of the trained network is considerably different. In particular, they show that the spectral network shows significant sparsity when it is over-parameterised with respect to the teacher. Intriguingly, they show a way to measure the “size” of a dense computational core that seems to be invariant with respect to changing the size of the student network.", "questions": "I have two non-trivial questions, and then a lot of minor comments. The two questions:\n\nI found it surprising that the number of “active” nodes in the spectrally parameterised student networks corresponded exactly with the number of hidden nodes in the teacher network. I’m not sure how to understand this. It in some sense suggests that the data emitted from the teacher network in some sense “maxes out” the computational capacity of the network. It’s not obvious to me at all that this would happen with a randomly initialised network. In fact, it’s not really obvious what I even mean by “computational capacity” of a network. It may just be that I’m not at all an expert in these matters, but I found this confusing. Either way, I think it would be very helpful for the authors to make some comment on why such a perfect numerical correspondence might be expected, and how it is to be interpreted.\n\nThe other question, as noted above is that I didn’t understand the path analysis section. In particular I wasn’t able to understand the transformation described in line 279-281, which I think then made it difficult to understand the rest of this section. Perhaps define the transformation explicitly in terms of the components?\n\nSome minor comments. As noted above some of these are simply English-language suggestions that I think might make the paper a little easier to read:\n\nLine 9 (and 55): I’m not familiar with the term “quenched teacher structure” and searching the internet didn’t help me. This might be a specific technical term that I’m not familiar with, but if not I wonder whether it might be more natural to use “frozen teacher” which I think would be the more usual English idiomatic form for a teacher whose weights are not modified during the experiment.\n\nLine 28: “self-consistent elucidation of intertwined data correlations” sounds a bit confusing, I wonder if the authors could state more plainly what they mean here?\n\nLine 33: “spatial sorting of the chosen reservoir of computing neurons” I’m not sure what this means … what does “spatial mean” in this context?\n\nLine 46: ​​”thus more suitable for possible hardware deployment”. This is an interesting claim. I wouldn’t have (naively) guessed that a sparse computation in eigenvalue/eigenvector space would necessarily map well to the constraints of current ML accelerator hardware. I don’t think this point needs to be expanded upon in the paper, but perhaps the authors could add a suitable reference (if one exists) that the interested reader could follow up on?\n\nLine 75: I think I’m to read equation 1 as a simple reparameterisation. In which case it’s a little bit confusing as the w_{ij}^{(k)} here is not the same as the one introduced above. Maybe it would be clearer if the initial weight matrix were given a slightly different symbol?\n\nLine 105: notation a bit confusing, it looks like a lower-case phi has turned into an upper-case phi. This isn’t consistent with how matrices and matrix elements are handled previously in the manuscript. It then becomes lower-case bold in equation 6 which is consistent with the previous notation.\n\nLine 202-204: I didn’t understand what the “alternative scenario” here was. Perhaps the authors can reword this part to be more clear?\n\nLine 215: “mean squared error (MSE) of S(h)” -> “mean squared error (MSE) of the predictions of S(h)”?\n\nLine 226: the identification between the histogram colour and the equation here is the opposite of that in the caption of Figure 1! I think the text is the correct one.\n\nLine 224: I’ve never heard “vehiculate” used before, and the dictionary doesn’t suggest a definition that quite fits here. Can I suggest that “drives” might be a more standard English term for this?\n\nFigure 3 caption: describe what the colours correspond to.\n", "limitations": "I think the only considerable limitation is what I mentioned in the questions section about the weakness of evaluating with only one source of data.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is on the whole well-written and easy to understand. The authors have done a very good job of taking a theoretical topic and making it accessible and understandable.\n\nThe idea they present is simple at its core, and the results are very interesting: both that their parameterisation/regularisation scheme leads to marked sparsity and the presence of a “core” with invariant size.\n\nWhile I think the results are somewhat limited (see below) the results that are presented are clearly described and care has been taken with the experimentation, leading to a convincing presentation. EDIT: the authors have fully addressed my concern around limited evaluation.\n\nI am unable to judge the significance or novelty of these results, as I am not an expert in the field of this paper. I would say that from the perspective of a general Neurips participant that I found the results interesting and exciting. So while I would defer to experts to place the work within the literature, I will comment that I think the results may be of interest to the general Neurips audience.\n", "weaknesses": "I think the only real weakness of the paper is that the experiments are quite limited in scope. The authors use a single source of “teacher” data, which is a fully-connected network with a certain shape. I think the paper would be considerably stronger if the authors were to repeat their analysis with further sources of data. In particular, I think it would be an interesting complement to use a non-synthetic source of data as a training objective for the network, if a suitable source could be found. While this would limit the ability of the authors in making the exact correspondence between the size of the teacher and the effective size of the student, I think seeing the same behaviours - enhanced sparsity as compared to the direct parameterisation, and a “phase transition” like behaviour showing a “computational core” - would considerably strengthen the results of the paper. I think without some strengthening of the results it is difficult to accept the paper, but I think further results would change this assessment. EDIT: the authors have fully addressed my concern around limited evaluation.\n\nWhile on the whole the paper displays good clarity, there are some places where the use of language leads to confusion. I have called out areas below in the questions section where I think there was particular confusion, and I think if these are addressed the paper will be sufficiently clear.\n\nI didn’t understand the “path analysis” section, or what the significance of this result was. Perhaps being more precise with the description of the analysis would have helped (see questions).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the referee for the insightful feedback and in the following we will address all of their raised points.\n\n**TWO MAJOR QUESTIONS (number of active nodes and path analysis)**:\n\nIn the presented results, we propose that the data originates from a 10-dimensional space and is subsequently projected in a complex manner into a 20-dimensional space, corresponding to the second and third layers. This projection can be further understood as the union of a linear function and a non-linearity (specifically, the ReLu function), particularly when biases are turned off as in our setup. Thus, we posit that at least a 20-dimensional space is necessary to capture the complete variability of the function at this stage.\n\nFor clarity, the dimensions of the second student network's hidden layer is kept fixed an set at 20. Given this design (with a lot of dimension that are all equal to 20), we believe it's logical to anticipate that a well-optimized first hidden layer of the student, also of dimension 20, would be competent in identifying the salient features from the teacher's first and second embedding layers. Our rationale is further substantiated by the paper *Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup (Goldt et al.)*. This work, which influenced ours, points in the direction that in a teacher-student setup, the student's dimensionality should at least match that of the teacher for effective learning.\n\nIn this teacher-student framework, both networks share identical activation functions and biases. This allows the linear components of the two information transfers (from the input to the first layer and from the first layer to the second) to be isolated and compared. Specifically, by examining the weights represented by $W=w^{(2)}w^{(1)}$, we can understand how activation moves linearly from the input to the second hidden layer. Each matrix entry, $W_{km}$, is defined by the equation: $W_{km}=\\sum_{j=1}^{N_i=20} w^{(2)}_{kj} {w^{(1)}_j}_m$ [sorry for the misalignment but it seems there is a bug on Markdown]\n\nHere, it becomes evident that we're aggregating all the pathways from neuron $k$ to neuron $m$. Our \"Path analysis\" section delves into the distribution of these pathways, which represent the cumulative effects of two linear operations. The term \"path\" that is quite unconventional, stems from the association between linear operations and bipartite graphs.\n\n**MINOR QUESTIONS (sorted by line reference)**\n- _line 9)_ Yes, the referee is right, the term is a substitute for \"frozen\" and we are willing to implement the correction in the following version of the paper.\n- _line 28)_ We apologize for the lack of clarity, we are willing to rephrase this in ”organizing complex\ndata by clearly understanding how different data pieces relate to each other.\"\n- _line 33)_ Again we are sorry for the lack of clarity. Spatial in this context is inherited from a jargon typical of dynamical systems on networks where the nodes are intended as points of a discrete space. We will modify this sentence as: \"These latter reflect the spatial sorting of the chosen reservoir of computing neurons\" $\\rightarrow$ \" These latter represent how the selected group of computing brain-like cells are organized.\"\n- _line 46)_ The referee is right as more conventionally the sparse implementation is the one more popular. However, there are applications where the advantage of having a smaller network structure with respect to a larger sparse one is appreciated. An example of these ML models is at the Large Hadron Collider at CERN, where the inference time required, on the order of nanoseconds, makes the smaller model preferrable. (see for example *Fast and resource-efficient Deep Neural Network on FPGA\nfor the Phas EPJ Web of Conferences 245, 01021 (2020)* )\n- _line 75)_ We will change slightly the notation in the following version of the paper.\n- _line 105)_ We are sorry for the confusion, we will double check the equations and be sure that with lowercase $\\phi^{(k)}$ we refer to the off-diagonal block of size $N_{k}\\times N_{k-1}$ and of the square matrix $\\Phi^{(k)}$ of size $(N_{k-1} + N_k)\\times (N_{k-1} + N_k)$.\n- _line 202-204)_ The alternative scenario we are referring to, which will be also included is something we are willing to analyze in follow-up papers and it is the effect of adding also the input eigenvalues, labelled $\\lambda^{(in)}$ as trainable variables. This will lead (for the input layer) also to the possibility of finding the relevant input space features by inspecting their after-training modulus (of course employing the regularization).\n- _line 215)_ We agree that it would be better rephrased in the suggested way.\n- _line 224)_ We can rephrase the word in the suggested way.\nline 226) Again, the referee is right, indeed there is a mistake. The Blue one, that gets more peaked in zero is the one of the spectral layer, eq. (10).\n- _Figure 3)_ We are willing to implement the suggested modification in the following version.\n\nMoreover, we have extended our analysis to 4 more relevant and complex datasets, more specifically in Figure 1 of the PDF page we show the same analysis carried out with for: Shuffled MNIST, Shuffled Fashion MNIST, California Housing and a more elaborated Teacher structure. In the latter, the size of the first hidden layer is different from the one of the second hidden. The student structure is left unchanged and the performances (in terms of accuracy and loss function) of both, the spectral Student and the classical student, are the same in each dataset. The results shown are in line with the simplified scenario that we have presented in the original version of the paper and we believe extend considerably the validity of our results.\n\nWe hope that, in light of the aforementioned changes, the referee will reconsider the rating given and we thank them again for the feedback."}, {"review_id": "MszzvIfnDc", "reviewer": "Reviewer_YyTW", "summary": "The authors propose a novel technique that allows identifying an invariant subnetwork in a student model that mirrors the characteristics of the teacher in terms of computing neurons, path distribution, and topological attributes.", "questions": "GENERAL COMMENTS:\n - (1) The authors have used a single synthetic dataset to perform the experiment. The results could be strengthened by considering multiple datasets (different synthetic parameterizations and publicly available datasets).\n - (2) We miss a more detailed related work. The authors provide a brief introduction, but references to related work are scarce. The authors may reference some works related to model distillation and pruning. In particular, we think the following works may be useful, given they mention some of the concepts and works related to those used in the manuscript:\n \t- Liang, Tailin, et al. \"Pruning and quantization for deep neural network acceleration: A survey.\" Neurocomputing 461 (2021): 370-403.\n \t- (old, but relevant) Elizondo, David, and Emile Fiesler. \"A survey of partially connected neural networks.\" International journal of neural systems 8.05n06 (1997): 535-558.\n \t- Gou, Jianping, et al. \"Knowledge distillation: A survey.\" International Journal of Computer Vision 129 (2021): 1789-1819.\n - (3) The authors should acknowledge the limitations of their research, discuss whether these limitations impact the results, and provide some insights on how these limitations could be addressed in future work.\n\n\nFIGURES: \n - (4) Figure 1: enhance the wording of the caption. \"Histogram for the quantities (10) (in blue) and (9) (in orange)\" -> \"Histogram for the quantities (Eq. 10) (in blue) and (Eq. 9) (in orange)\"\n\nSPELLING/WORDING:\n - (5) \"Remarkably several topological proprieties of the teacher\" -> \"Remarkably several topological properties of the teacher\"\n - (6) \"These latter parameters are denoted λ(k) for reasons that will become clear in the following, \" -> in the following?\n - (7) \"The dataset will be generated by choosing the probability distribution\" -> \"The dataset was generated by choosing the probability distribution\"?\n - (8) \"values stay stably across all choices of h above 20\" -> \"values stay stable across all choices of h above 20\"\n - (9) \"To assess feature localization within the network, we generated aggregated histograms of the scalars (9) and (10).\" -> \"To assess feature localization within the network, we generated aggregated histograms of the scalars (Eq. 9) and (Eq. 10).\"\n - (10) \"To ensure that the achieved conclusions are not influenced by the size of the second hidden layer, various student configurations have been tested, yielding equivalent results.\" -> Please provide some details.\n - (11) \"The blue histograms stands for (9),\" -> \"The blue histograms stands for (Eq. 9),\"", "limitations": "The authors should acknowledge the limitations of their research and provide some insights on how these limitations could be addressed in future work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": " - The manuscript is clearly structured, and the subject of research is relevant\n - The authors have developed a novel technique to identify invariant characteristics of a student model mirroring key characteristics of the teacher network.", "weaknesses": " - The authors have used a single synthetic dataset to perform the experiments.\n - There is little reference to related work, and no baselines are considered when comparing the proposed approach", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude to the referee for their positive review and insightful suggestions. In relation to the 'Figures' and 'Spelling/Wording' sections, we are prepared to implement the suggested corrections should the paper progress to the camera-ready stage.\n\nRegarding the General Comments section:\n\n- _1)_ We concur with the referee on this point. We have conducted additional experiments that we are eager to incorporate into the manuscript. Specifically, in Figure 1 of the PDF, we present an analysis for various datasets: Shuffled MNIST, Shuffled Fashion MNIST, California Housing, and a more intricate Teacher structure. In the latter, the size of the first hidden layer differs from that of the second, ,removing the high degeneration of the original example and making it a more relevant case of study. The student structure remains consistent with the dimension of the second hidden layer set to 50. In this context, the performance metrics (in terms of accuracy and loss function) for both the spectral Student and the traditional student are comparable across all datasets. The findings align with the scenario described in our original paper: the spectral regularization is capable of finding a computational core (panels $a-d$) whose MSE behaviour, after perturbation, is independent from the initial layer size (panels $e-h$). Due to space limitations in the PDF, we opted against including supplementary plots, such as the loss function values and the eigenvalue histograms.\n\n- _2)_ We are in agreement with the referee on this matter. We faced challenges in locating relevant references specific to node pruning, as the majority of existing literature appears to concentrate on weight pruning. We appreciate the valuable references provided by the referee and intend to incorporate them into our manuscript in conjunction with others we have found.\n\n- _3)_ The referee's observation is on point. We will acknowledge the limitations of our study. In particular, we recognize that computational constraints prevented us from employing highly complex models where outcomes might vary. We hope to explore such scenarios in future research. Furthermore, we haven't fully explored the capabilities of the spectral decomposition. Indeed, it's feasible to conduct a feature relevance analysis in the input space. By setting $w_{ji} = (\\lambda_j^{in}-\\lambda_i^{out})\\phi_{ji}$ in the first layer, the eigenvalues $\\lambda^{in}$ could emphasize the relevance of the input's $j$-th component.\n\nWe hope the referee appreciates the aforementioned changes and we thank them again for the interesting feedback."}]

Author rebuttal:
We express our gratitude to the chairs and referees for their valuable effort in providing constructive feedback on our manuscript. We have carefully considered each of the comments made and addressed them individually and in-depth in the replies to the referees. We have taken note that all the referees highlighted the need for more than one teacher and the use of more complex and realistic datasets. We acknowledge that this could be a weakness of our work. To address these concerns and enhance the quality of our paper, we conducted various new experiments on different datasets and with a more elaborate teacher structure. The results of this supplementary analysis are available in the attached PDF and demonstrate that all the spectral regularization effects still persist in these complex scenarios. 

For this scope, we have analyzed various datasets, including Shuffled MNIST, Shuffled Fashion MNIST, California Housing, and a more complex Teacher structure. Both the spectral Student and classical one achieved similar accuracy and losses across all datasets. Although we regret not being able to include additional plots, such as the loss function value and eigenvalue histogram, due to space constraints in the PDF, the plots show the clear presence of a regularization effect in the spectral parametrization irrespectively of the initial size of the Student and of the task assigned to it (see panels $a-d$). Moreover the computational core found behaves in the same way when its neurons are removed (panels $e-h)$

These additional findings will surely strengthen the claims of our paper and will be incorporated into the main manuscript if it is accepted. Hoping that these new results and the point-by-point reply that we have given to each referee can convince them of the quality and robustness of our work, we remain.

Respectfully yours,
The authors

Meta-review: Reviewers agree that the paper addresses an interesting question regarding teacher-student setting. and demonstrates results on both synthetic and real-world dataset (after rebuttal). Reviewers are satisfied with the rebuttal. I would encourage the authors to further improve the presentation, in particular by writing down formal definition and algorithms, and properly incorporating new results in the camera ready.
Decision: Accept (poster)