SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By taking the structure of the matrix  $\phi$  as the binarization version of such simplified Toeplitz matrix, the  $\lambda^{in}$  of the notation introduced in [12] play the exact same role of the filters degree of freedom in the CNN layer.
We recall, indeed, that the weights strength of a feedforward fully connected layer can be written in terms of eigenvalues and eigenvectors of the equivalent directed bipartite graph as follows:
$$
w _ {i j} = \left(\lambda_ {j} ^ {(i n)} - \lambda_ {i} ^ {(o u t)}\right) \phi_ {i j} \tag {B.5}
$$
The effect of  $\lambda^{(in)}$  can be therefore considered as a column-wise product whereas the  $\lambda^{(out)}$  as a row-wise one. By setting  $\lambda^{(out)} = 0$  each  $\lambda^{(in)}$  can be adjusted to the same value of the convolutional weight that is solely present in a given column. This operation is graphically presented on the bottom part of Figure B.3.
B.2 Filter relevance as eigenvalue magnitude: a bridge with existing literature
Due to equation (B.5) another correspondence can be elaborated upon: the matrix  $\mathcal{M}(w)$  can be constructed using as a degree of freedom the eigenvectors components  $\phi_{ij}$  as shown in Figure B.4. By setting  $\lambda^{(in)} = 1$  the filters action could be accounted for by imposing an apposite structure of matrix  $\phi$ . The one shown in the figure is a representative example of a convolution with non overlapping filters after the input has been adjusted. The same conclusion can also be achieved for a  $\mathcal{M}(w)$  structure like the one depicted in Figure B.2.
The remaining degree of freedom of the  $\lambda^{(out)}$  can be used as a measure for the filter relevance if every  $\lambda_i^{(out)} = \lambda \forall i$ .
We posit, therefore, that the regularization proposed in [2] is equivalent to a spectral regularization for the case of a CNN with an imposed parametrization that goes as follows: every  $\lambda^{(in)} = 0$ , Toeplitz structure of  $\phi$  and  $\lambda_i^{(out)} = \lambda_j^{(out)} \forall i,j$ .
Figure B.3: In the upper part a feedforward layer equivalent to a CNN one is shown. Input components have been duplicated in order to have a single non zero component per column. In the bottom part the same weight matrix is rephrased in terms of the spectral decomposition of (B.5). The equivalence can be achieved setting  $\lambda_{1,2,3\dots} = w_{1,2,3\dots}$ ,
Figure B.4: The weight matrix is rephrased in terms of the spectral parameters. In this case the eigenvalues are exploited as a filter relevance proxy.
C Code availability
The python code used to conduct the analysis is fully available at the following GitHub repository https://github.com/Jamba15/Spectral-regularization-teacher-student/tree/master. The code allows both the retrieval of the full set of results presented in this work and the run of new experiments with different settings.
The implementation of the spectral layer and the Structural Pruning functions can be found in https://github.com/Jamba15/SpectralTools. The repository, at the moment of publication, contains a TensorFlow and PyTorch implementation with documentation.