Weight matrices compression based on PDB model in deep neural networks
Authors: Xiaoling Wu
Junpeng Zhu
Zeng Li
Venue: ICML 2025 poster (2025)
Keywords: Weight Matrix Compression
Noise-filtering
Generalization Performance
Random Matrix Theory
Primary area: probabilistic_methods->spectral_methods
TL;DR: Weight Matrix Compression with PDB Model

Abstract: Weight matrix compression has been demonstrated to effectively reduce overfitting and improve the generalization performance of deep neural networks. Compression is primarily achieved by filtering out noisy eigenvalues of the weight matrix. In this work, a novel **Population Double Bulk (PDB) model** is proposed to characterize the eigenvalue behavior of the weight matrix, which is more general than the existing Population Unit Bulk (PUB) model. Based on PDB model and Random Matrix Theory (RMT), we have discovered a new **PDBLS algorithm** for determining the boundary between noisy eigenvalues and information. A **PDB Noise-Filtering algorithm** is further introduced to reduce the rank of the weight matrix for compression. Experiments show that our PDB model fits the empirical distribution of eigenvalues of the weight matrix better than the PUB model, and our compressed weight matrices have lower rank at the same level of test accuracy. In some cases, our compression method can even improve generalization performance when labels contain noise. The code is avaliable at https://github.com/xlwu571/PDBLS.

Weight matrices compression based on PDB model in deep neural networks
Xiaoling Wu
Junpeng Zhu Zeng Li
Abstract
propose a training scheme to promote the filters' low rank by using nuclear regularization and achieves the low-rank approximation of the original filters. (Idelbayev & Carreira- Perpinán, 2020) emphasize determining the ideal rank for each layer and weight approximation via adding a penalty term of the rank. (Liebenwein et al., 2021) further propose compression architecture using SVD for local layer com- pression and minimizing the maximum relative error across different layers for global compression. However, these methods rely on hyperparameters and are generally difficult to optimize.
Weight matrix compression has been demon- strated to effectively reduce overfitting and im- prove the generalization performance of deep neu- ral networks. Compression is primarily achieved by filtering out noisy eigenvalues of the weight matrix. In this work, a novel Population Double Bulk (PDB) model is proposed to characterize the eigenvalue behavior of the weight matrix, which is more general than the existing Population Unit Bulk (PUB) model. Based on PDB model and Random Matrix Theory (RMT), we have discov- ered a new PDBLS algorithm for determining the boundary between noisy eigenvalues and in- formation. PDB Noise-Filtering algorithm is further introduced reduce the rank of the weight matrix for compression. Experiments show that our PDB model fits the empirical distribution of eigenvalues of the weight matrix better than the PUB model, and our compressed weight matrices have lower rank at the same level of test accu- racy. In some cases, our compression method can even improve generalization performance when labels contain noise. The code is avaliable
Recently, to understand the training mechanism and gen- eralization performance of neural networks, RMT has suc- ceeded in providing theoretical explanation and improv- ing generalization ability by analyzing spectral properties of the weight matrices of neural networks, see (Berlyand et al., 2023; Martin & Mahoney, 2020). For weight matrix W RMT is mainly employed to study the eigenval- ues {Λ 2 ... 2 Ap} of W W, in particular the limit of FW W (c) - Σ 1 (A, <w) called Limiting Spec- tral Distribution (LSD). According to (Martin & Mahoney, 2021), the training process of weight matrices mainly goes through three phases: initial phase, bulk+spikes phase and heavy-tailed phase (Fig. The randomly initialized weight matrix Wo satisfies EW Wo = IpI where oğ depends on the initial distribution of the entries of Wo- At this stage, the LSD of W W, matches Marčenko-Pastur (MP) law. During the training process, the information learned by the weight matrix is mainly reflected in few spiked eigenval- ues exceeding the MP edge A+, and the initial variance also changes from of to o². this phase, the PUB model which assumes EW EWTY W diag [eag(a1..... (a1, OK
1. Introduction
Deep Neural Networks (DNN) have achieved outstanding performance in many fields such computer vision (Leek et al., 2022), speech recognition (Mohanty et al., 2022) and recommendation systems (Da'u & Salim, 2020). Deeper and wider DNN frameworks have demonstrated superior learning performance (Yang et al., 2019), but many these networks are extremely over- wer-parameterized and prone to overfitting. Recently, some low-rank compression tech- niques have been applied to avoid overfitting by removing small singular values of the weight matrix. (Xu et al., 2019)
spikes
bulk
E is often used to analyze the trained weight matrices. Correspondingly, WT w exhibits several comparatively large sample spiked eigenvalues, alongside clustered bulk eigenvalues. In the final training phase, the eigenvalues of WT W gradually follow heavy-tailed distribution, but this phase is not very common. Thus, we mainly focus on the bulk+spikes phase.
Equal contribution 'Department of Statistics and Data Science, Southern University Science and Technology, Shenzhen, China. Correspondence to: Zeng Li <liz9@ sussecssssccccc..c...
Furthermore, (Staats et al., 2023) has verified that in the bulk+spikes phase, large spiked eigenvalues learn rules from data while small bulk eigenvalues do contribute model learning. They propose compress network by removing
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Compress weight matrices
rS0 or w.:w:
LSD JFB axdd
Fab W WWW
06-
E50 of W
0.8
01-
0.
041
0.
0.-
0.-
bulk
balk
spikos
(a) Initial
(b) SulkeSpikes
(c) Hcavy-tailed
Figure 1. The three phases of histogram (ESD) of eigenvalues of W W during network training.
small bulk eigenvalues and recovering spiked eigenvalues of EPUB from WTW, therehy improving the generalization performance. A following question is how to determine the boundary between informative spikes and noisy bulk eigenvalues. (Staats et al., 2023) determine the boundary by cross-validation, which is purely data-driven without theoretical support and is quite computationally expensive. (Ke et al., 2023) propose the Bulk Eigenvalue Matching Analysis (BEMA) algorithm based on the PUB model. Us- ing BEMA, a weight matrix pruning algorithm is further developed by (Shmalo et al., 2023). However, the existing compression algorithm either rely on the PUB model or empirical adjustments without any theoretical support.
keeping the bulk1. The spiked information of EW7 W is also recovered. Experiments demonstrate that our pro- posed PDB model outperforms the PUB model in fitting the eigenvalues of WTW, and our compressed weight matrices exhibit lower rank while preserving test accuracy or even improving it when label contains noise. In summary, our contributions include:
We propose a generalized PDB model to characterize the eigenvalue behavior of V W in the bulk+spikes phase. This model accurately captures the empirical distribution of eigenvalues, thus confirming its validity.
Regrettably, the assumption of the PUB model is too re- strictive, as it requires WTW to have a homogeneous population variance o². It also fails to accurately cap- ture the eigenvalues distribution of w in many em- pirical studies. Therefore, we consider a more general PDB model accommodate heterogeneous population variances. Specifically, propose that during the training process, the initial EW! Wo = Ii will evolve into EWTW diag diag((1)..... (a,,..., ,KK of. ...,,
We propose an efficient hyperparameter-fiee algorithm PDBLS to estimate EW" W. which establishes the relationship between the eigenvalues of W and EWTW. This enables us to establish boundary between noise and information and determine how much information of W W needs to be retained.
o...... o2. 22.0..... 2...., .2,....0). Note that when bulk2
3.
We introduce a novel PDB Noise-Filtering algorithm to compress the weight matrix by removing noisy eigen- values and retrieving information from E EWWW. Our algorithm also recommends the best compression ratio of weight matrix where further compression will lead to significantly loss of generalization ability. Experi- ments show that our approach can significantly reduce the rank of weight matrix W while preserving network generalization performance, and even enhancing it in the presence of noise.
balk1 of ===3 02. our PDB model will degenerate to the PUB model. We didn't adopt more general M-Bulks (M > 3) models because observe that the proportion of additional bulks is negligible and the PDB model has shown sufficiently superior performance than PUB model in Fig. 2.
K' spikes
Based on PDB model, we posit that both the spikes and the bulk1 contain valuable information, whereas the bulk2 predominantly represents noise. We further propose Pop- ulation Double Bulk Least Squares (PDBLS) algorithm to estimate the structure of LE W W. from which we can determine the boundary between noisy eigenvalues and in- formation. Moreover, as shown in Fig. 5, we develop new PDB Noise-Filterne Filtering algorithm to compress the weight ma- trix by only removing the smaller bulk2 eigenvalues while
2. Motivation: from PUB to PDB model
In this section, we introduce the motivation that why we choose the PDB model.
2

---

Compress weight matrices
ES0 or w.W.
of model
LSD of PSB modd ESD W"W
0.5-
ESD of WWW
0.-
B(W
B(W W ding (
BWW dingl C4
0.4-
7
0.4-
0.4
spiked elgenvalues
0.2-
0.-
0.0
2_
8
bulk
sgrikets
(0-x)(--0 -)
bulkk2
(p- Kbr
bulkl
spikes
X'
(a) Initial matrix
(b) PUB model
(c) PDB model
Figure 2. The histogram (ESD) and LSD of eigenvalues of WTW, where (a) shows initial matrix, (b) and (c) show trained matrix with PUB and PDB model. The vertical dashed lines represent the noise-information boundaries under the two models.
2.1. PUB model
Recently, for weight matrix W E Ruxp, (Martin & Ma- honey, 2021; Staats et al., 2023) use RMT to study the eigenvalue behavior of W W before and after training. In the initialization phase, the entries of Wo are i.i.d. gener- ated satisfying En = JW, Wo = Tp and the LSD of WJ Wo has density function (MP law):
Eig(E = ffa1..... okg
0² ".....,²'! o²)
K
K
$ spikes
$ bulk
Big(W" {A1, - AK,
A 2 \
K
p-K K
(x;,,,88) (2)-2)(r x)(x - ) 2Tozer
2.2. PDB model
However, the assumption of the PUB model is quite restric- tive, as it requires W to have homogencous popula- tion variance o². Additionally, the eigenvalue distribution of W W does not perfectly align with the MP law with o² under PUB model, as shown in Fig. 2(b). To address these limitations, we consider a more general model which can accommodate heterogeneous population variances. Specif- ically, extend o² to M different positive constants (of, 1 S < - M) with proportions {t,, 1 S i M):
Here the boundary points X (0ğ) = ob + VCJ P.
(Martin & Mahoney, 2021) point out that after training, the eigenvalues of / can be divided into two categories: the bulk eigenvalues and the spiked eigenvalues. As shown in Fig. 2, the bulk eigenvalues are tightly clustered with their histogram conforming to the LSD. On the contrary, the spiked eigenvalues are far from bulk eigenvalues and lie outside the boundaries of LSD. In response to this phe- nomenon, (Shmalo et al., 2023; Staats et al., 2023) explore PUB model to characterize the eigenvalues of WTW W in the spikes phase. They assume, after training.
ΣPUB = EW TW = diag( (01, ak 0²
$
Σ PUB = EWT W = diag aq...... aK 0²
EW W = diag diag((11..... (1,...,
ak
o of
M Rz
p-
K
K)t
(p- K)aa
In empirical studies, observe that M = 2 is sufficient, because the proportion of additional a; terms is negligible. For example, Table I presents the estimated proportions for M = 4, where fg. f are very small. Therefore, in this study,
Here {a),..... [kk are K population spiked eigenvalues and (o²..... 0²} are p p-A K bulk cigenvalues of EUBB During training process, the initial variance of changes to o² and extra K spikes appear. Correspondingly, the LSD of sample bulk eigenvalues of WTW follows the MP law with parameter o². While the K sample spiked eigenvalues lie outside the boundary of the LSD, i.e., (1 + vo)2. The relationship between the sample eigenvalues of W W and population eigenvalues of ΣrUB is given below.
Table I. The estimated proportion of each bulk when M = 4.
o
02
03
0.6803
0.2719
0.0474
0.0004
3

---

Compress weight matrices
propose generalized Population Double Bluk (PDB)
model for trained weight matrix W, satisfying
F18-Saml
708-F18LX
EPDB: = EW
diag( a1, ak of i.... of 2 22..... o3.
(2)
L:
(p-K)t (p- K)(K))(
62
where fai...., 0gk are K population spikes of EPDB and t is the proportion of of among all bulk eigenvalues. Similar PUB model. PDB model also has one-to-one corresponding relationship between the sample eigenvalues of W W and population cigenvalues of EPDB:
(a) FCNN on MNIST
(b) Vggló on CIFARIO
728 ROLA
Eig (EPDB) = ((1
iy of (p-K)
(p- K)--)
spikes
$ bulk1
$ bulk2
Fig(W Hig("WW) W) D1..... AK
A 2
> (p- ()(-))
(p- Kjt
where B = AK+(p-KJe KYe is the boundary point between sample bulk1 and bulk2 of W'W. Note that the PUB model (1) is a special case of our PDB model when = 0.
(c) ResNet18 on ImageNet
(d) ResNet18 on CIFAR10
Figure 3. Histograms of the eigenvalues of W W and density curves of LSDs based on different models. The solid blue line represents PDB model, and the dashed red and green for PUB model.
More importantly, the proposed PDB model demon- strates significantly better empirical performance than the PUB model. Firstly, the density curve of LSD under the PDB model aligns more close the histogram of W w, see Fig. 3. The PDB model parameters for Fig. 3 are listed in Table 2.
Table 3. Comparison of theoretical and empirical spectral ma- ments for FCNN and VGGI6, 63 1 (W"w)* .j = 1,2,
FCNN:MNIST
CIGI6ICIFARIO
Table 2. The results of Obulk (of,oz,t) f} for the DNN.
Model Method
71
722
733
711
7/2
73
PUB BEMA
2.27
7.74
32.23
0.92
1.26
2.12
Estimator
67
62
PUB Kernel
1.37
2.81
7.04
0.90
1.22
2.03
FCNN: MNIST
0.17
3.37
1.36
PDB PDBLS 1.71
4.94
18.79
0.96
1.53
3.11
VGG16: CIFAR10
0.25
1.61
0.75
empirical îg 1.74
5.50
24.17
0.97
1.59
3.51
ResNet18 :: ImageNNt
0.35
7.54
0.96
ResNet18: CIFAR10
0.25
5.06
0.95
Table 4. Comparison theoretical and empirical spectral mo- ments for FCNN and ResNet18.
ResNet18: I
8esNet18:CIFARIO
Secondly, PDB model achieves superior alignment with the spectral moments of W W compared to the PUB model. Specifically, we employ different methods to es- timate model parameters and compare the first three the- oretical moments with the empirical values, including the BEMA method (Ke al., 2023) and the Kernel approach (Staats al., 2023) for the PUB model, and the PDBLS algorithm for the PDB model (see Section 3.2). Detailed results are presented in Tables 3-5, where Table 3-4 corre- spond to the FCNN and convolution networks, and Table 5 presents the large language models. The theoretical spectral moment formulas can be found in the Appendix. It's clear that our PDB model matches the empirical moments much better than PUB model, especially higher order moments.
Model Method 71
7/2
73
71
72
73
PUB
BEMA 2.94
17.26
126.79
76.79
1.80
4.82
15.84
PUB Kernel
1.11
2.45
6.77
1.83
5.02
16.84
PDB PDBLS 3.23
30.62
377.50
2.27
9.14
49.08
empirical ny 3.32
31.48
402.79
2.28
9.38
52.65
3. PDB estimation
In this section, we provide a detailed characterization of the asymptotic properties of the sample cigenvalues of W w, including the bulk eigenvalues, bulk boundaries, and spiked eigenvalues. Building on these properties, we propose the PDBLS algorithm to estimate the model parameters of

---

Compress weight matrices
Table Comparison of theoretical and empirical spectral mo-
LSD of PDB mudkl BSD of wr
ments for TS-base and BERT.
06-
T5-base: RTE
BERT: SCITAIL
Model
Method 1/1
72
33
71
12
73
04-
PUB BEMA
0.67
0.90
1.51
0.59
0.69
1.02
PUB Kernel
0.53
0.56
0.74
0.56
0.63
0.88
Step
PDB PDBLS
0.77
1.55
4.17
0.67
1.18
2.78
empirical % 0.72
1.83
5.35
0.71
1.38
3.95
Step
Step
EPDB, which allows us to determine the boundary between noise and information.
8
12
Step
3.1. Sample eigenvalue behavior of PDB model
Figure 4. The histogram (ESD) of eigenvalues and LSD of PDB model. The estimation procedure (PDBLS Algorithm) for Obut ot, oī). Oupike (K, a1...., ax) and Obound {++, (8} are also illustrated in this figure.
The following three theorems illustrate the relationship be- tween sample eigenvalues of W W and population param- eters {K, .1..................... aKK of o2, t of ΣPDB-
Theorem 3.1. (sample bulk eigenvalues) Under PDB model (2), as n + ,0, p/n - c, the empirical spectral dis- tribution of WT W. FW'W(x) (x) =
3.2. PDBLS algorithm: Estimation of PDB model
p E SaySa) a.s. converges to the LSD with the density function:
Based on Theorems 3.1-3.3, we introduce an algorithm called Population Double Bulk Least Squares (PDBLS) to estimate ΣPDB EWTW and two sample boundary points: one is the boundary between the sample spikes and the bulkl, X4. and the other is the boundary be- tween bulk1 and bulk2, B. Our goal is to estimate the bulk parameters Obukk = {oî t2. the spike parame- ters Opiike = (K, eters Obound = (}+ We denote these estimators as [o}, 02. f}. {K, a1.,a,..,, .gg and (}.. We start with the estimation of Obul and Obound- Although spiked eigen- values don't affect LSD, some exceptionally large spikes can result in an overestination of bulk parameters in the finite sample case. As depicted in the Fig. there sev- eral extremely large eigenvalues highlighted in the blue box, indicating the necessity for to give an initial estimation of the number of spikes Kg. By mitigating the influence of (5) the top Ko spikes, we provide estimates for Obulk first, fol- lowed by Obound: and further refine the number and values of spikes in Oxpike- In detail, there are four main steps to obtain these estimators.
Im (2)
pxx) = lim n0
2 x= in.
1 > O,
(3)
TC
where Im m(2) represents the imaginary part of m(2) and m(z) satisfies the following equation:
). and the boundary param-
?to?
+ c - t)o3
2
(2) 1 mf)(2)
1 02m(2)
(4)
An example of p(x:) is shown in Fig. 2(c) where {of
4.63, o3 = 1.67, t 0.203.
Theorem 3.2. (sample boundaries) The boundary between sample spikes and bulk1 of WT W is
x4 = g(v), y = arg max (g((x) =
zEk
g(z) = + to? oi
(1 - t)o
+
(6)
Step Initially estimate the number of spikes Ko-
02
Inspired by (Liu et al., 2023), the location that corresponds to the smallest distance between the consecutive eigenvalues can be used to determine the number of spikes. The initial number of spikes based on this criteria in (Liu al., 2023) can be calculated by:
The boundary between sample bulkl and bulk2 of W W is
8 = Xtt
Theorem 3.3. (sample spiked eigenvalues) Under PDB model (2), as - :0, p/n - the sample spiked eigen- values (A....,,>p } of WTW converges to functions of population spiked eigenvalues {aI,.... ..3 } of ΣPDB:
Ko arg min 2 [ [n((Ar -ot1))t((ppp---tt log 0, x 2pk),
(8)
where 0p. p-k- ExrL exp(2(A: - (1+1)} and w
1
>; - g(a;), je (1..... K)....
[6n [.] represents the floor function. Then we remove
5

---

Compress weight matrices
Algorithm 1 PDBLS algorithm (For estimation)
From Fig. 2(c), we can see that the number of spikes is equal to
Input: Eigenvalues Aj of matrix W W.
Output: Obuik
{0%, 02 f}.
θ spiike
K= #(g E (X4;Amax))
(1I)
(K,ā,,... KK,á,,..,,g&!! .g)), Obound = 1},
where #1-} represents the cardinality of the set. With the estimated Obutk in Step 2, we can obtain the estimator of boundary point X via (q)( Then we can further refine the estimation of the number of spikes K via Eq. (11). Subsequently according to Theorem 3.2, we estimate , by
II Initially estimate Ko
1: Roughly estimate the number of spikes Ko by Eq. (8);
2: Remove the top Ko spikes;
II Estimate Obulk
3: Calculate (:(buu))) by Eq. (10);
3 > -Ri
(12)
4: Minimize Eq. (9) to solve (oî 02.00..0 t);
// Estimate O bounnd and Refine K
Step Estimate Oppike = {ai, (ai,...agh
5: Calculate boundary point X+ by Eq. (5);
6: Estimate the number of spikes K by Eq. (11);
The relationship between spikes of W w and EPDB is established in Theorem 3.3. Thus we can obtain {aI....., .g gk J by Eq. (6) and (7).
7: Estimate boundary point 3 via Eq. (12).
// Estimate sppike
8: Recover population spikes {ay)f= from sample
In summary, the estimation algorithm is thoroughly de- tailed in Algorithm 1. Based on Theorems 3.1-3.3, we can show that PDBLS is consistent estimator of ΣPDB- Theorem 3.4. Under PDB model (2), as n - 30.p/n + C.
spikes by Eq. (6) and Eq. (7);
the top Ko spikes which may affect the estimation of Oulkk-
(Obutks @spikes @bound)
(Obuiks
+ spikes θbound
Step 2 Estimate Obakk = {oj, t3.
The bulk parameters can be estimated by solving the follow- ing least squares optimization problem (Li et al., 2013):
4. Matrix compression of PDB model
In this section, we determine the boundary between noise and information and introduce novel PDB Noise- Noise-Filtering algorithm to compress weight matrices.
Σ (Obbkk) - ) Orakk
arg min
(9)
1=1
Now with the estimation of our PDB model, we have a clear picture about two boundaries of W, one is the boundary between the sample spikes and the bulk1, X+, and the other is the boundary between bulk1 and bulk2, 8. This sheds new light on determining the noise-information boundary, which facilitates the compression of the weight matrix.
where (u....., } take equally spaced j-points in each in- terval of U. Here
10,0) U (0, 0.5.min) U (5Amazi 10 P F n.
u =
(-10, 0) U (5Amar; 10 ), =
In the current literature, existing method to determine the boundary between noise and information are either purely data-driver or based on the PUB model. As for the PUB model, (Shmalo et al., 2023) and (Staats al., 2023) treat the MP edge X of LSD, the vertical dashed line in Fig. 2(b), as the noise-information boundary. They assume that all the information is contained in the spikes of W W while bulk eigenvalues only represent noise. Accordingly, in their compression algorithm, they remove all the bulk singular values and recover the population spiked singular values from sample spikes of weight matrices. However, the boundary inferred from the PUB model is quite stringent. Consequently, the corresponding compression algorithm would lead to a substantial loss of useful information. As a result, the compressed network would struggle to maintain the original generalization performance.
We set 20 in all experiments of Section 5.
According to (Yao et al., 2015) and Eq. (4) in Section 2, for each Uh we can calculate ú; (Obute) as:
1
+ p Ko
to
0, (Obulk) =
mA (u.)
om (u:)
(10)
p- Ko
(1 - 2
+
n
1 1 oz m, (u:)'
where m, (u:) = 1- 1-(p-Ro)/n
Rol/n Ui
Ro
12
By solving (9), we obtain the bulk estimators Obuik (ōf. oz, i). Subsequently, the density curve of LSD of W W naturally follows from Eq. (3)-(4)
Step 3 Estimate Obound = {4,,8] 8} and refine the num- ber of spikes K.
Therefore, to strike a balance between the amount of in- formation retained and the degree of weight matrix rank
6

---

Compress weight matrices
Algorithm 2 PDB Noisc-Filtering algorithm (For matrix
5. Experiments
compression)
In this section, we conduct numerical experiments to demon- strate the superiority of our PDB model and the effectiveness of the weight matrix compression algorithm.
I: Perform the singular value composition of the weight matrix W UD V, where D diag( (01ggg(1....00..... A1,.... Ap)- 2: Apply the PDBLS algorithm to estimate PDB model in- cluding Obulk = {6t. 02. f). Ospike = {K, â.....a R and Obound = B8.
5.1. Experimental setups
3: Replace sample spikes {Ag E (λ+,, Amar] with pop-
We evaluate generalization performance using test accu- racy and employ three basic neural network architectures, the three-layer Fully Connected Neural Network (FCNN), Residual Network-18 (ResNet18) (He et al., 2015), and Vi- sual Geometry Group- 16 (VGG16) (Simonyan & Zisserman, 2015) The FCNN is trained on MNIST, while ResNet18 and VGG16 are evaluated on CIFARIO (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009). The three architec- tures configurations including the following:
ulation spikes {āi...., dg). retain the bulkl {>g E [8, 2+1 in D, and replace the bulk2 (Ag E [Amin, 9)) with zero. The new diagonal matrix is formulated as
Dnew = diag {(a....................
fAy E [3, 18,0+13.0
4: Obtain a new weight matrix W new whose rank is K + +pp K)t.
UD new V,
FCNN trained on MNIST.
The FCNN for MNIST fc784 - fc512 - fc512 + fc350 + fc10, where fc denotes fully connected layer.
SVD of
2. ResNet18 trained on CIFAR10 and ImageNet.
U
X
V
We add linear layer of size 512 256 before the linear layer in the original ResNet18 when training CIFARIO.
Algorithm2 2
Matrix Compression
3. VGG16 trained on CIFARIO and ImageNet.
new matrix W new
The size of last two linear layers is changed from 4096 256 when training CIFAR10.
spikes 1/2
recover 1/22
U
X
retain bulk I
Additionally, we assess the generalization of three represen- tative pre-trained architectures: BERT (Devlin al., 2019) and TS-base (Raffel et al., 2020) for natural language pro- cessing, and VIT-L (Dosovitskiy, 2020) for computer vision. VIT-L is variant of the CLIP vision encoder (Radford et al., 2021). In the CLIP model, the text encoder remains frozen, and text embeddings are generated by processing the class labels through it. For the language models, experiments are conducted on the RTE (Wang et al., 2018) and SciTail (Khot et al., 2018) datasets, while the vision model is tested on DTD (Cimpoi et al., 2014) and SUN397 (Xiao et al., 2016).
butk 21/2 wt
Figure 5. The process of matrix compression
reduction, we extend the PUB model to the PDB model, where only one bulk represents noise, while all the trained information is retained in all the spikes and the other bulk. In other words, our proposed noise-information boundary is the boundary between sample bulk1 and bulk2, 3, the vertical dashed line in Fig. 2(c). In this way, we retain more important eigenvalues of W W and find a more suitable way to recover the important information ΣpDB. The detailed process of matrix compression is given in Fig. 5. Specifically, with the estimated EPDB, we remove all the smaller eigenvalues in the bulk2 of W W while keeping the 1 as a significant source of information. Simulta- neously, we incorporate the recovered population spikes θ aaiike as another valuable source of information learned from W during the training process. The detailed com- pression algorithm is outlined in Algorithm 2.
Each image is normalized to the range of [0,1]. and the weight matrices of the networks are initialized by the Glorot uniform distribution (Glorot & Bengio, 2010). For basic architectures, we employ SGD with an exponential decay learning rate during the training phase. The activation func- tions used are Relu(.) for hidden layers and Softmax(-) for output layer. For large-scale model, utilize the AdamW (Loshchilov & Hutter, 2017) optimizer in conjunction with cosine learning rate scheduler. Regarding convolutional layers, we follow the scheme in (Idelbayev & Carreira- Perpinán, 2020) for reshaping the convolution kernel into 2D matrix. In particular, since a convolution layer has

---

Compress weight matrices
groups of convolutional kernels with channelss, each of size d x d, reshape them into a 2D matrix of size nd x ed for compression. After compression, we can reshape them back to their original 4D convolutional kernel form.
compression level. The boundary point obtained by the PDB model retains more important information.
3. The empirical compression ratio 0.55 proposed in (Shmalo et al., 2023) does not fully compress the weight matrix. On the contrary, the noise-information boundary identified by our method accurately captures the point at which test accuracy starts to decrease dur- ing compression.
We compare two classes of matrix compression methods: the PUB-based methods and SVD-based methods.
PUB-based methods include:
1. Bulk Eigenvalue Matching Analysis (BEMA) (Ke et al., 2023): find the boundary of density curves by quantile fitting.
Table 7 presents the test accuracy of various neural networks when compressed using different algorithms. Our method achieves the best accuracy, with maximum improve- ment of 5% rable outcomes are listed in the Appendix. These results demonstrate the effectiveness of our proposed method.
2. Kernel Estimation (Kernel) (Staats et al., 2023): use Gaussian broadening to fit MP distribution and find the boundary.
(0.7536
for T5-base: RTE. More compa-
0.7174
SVD-based methods include:
Table 8 compares the computational efficiency of four meth- ods, with the time measured in seconds for the total exccu- tion and inference phases following data processing. Among them, the computational overhead of our PDB model is slightly higher than that of PUB. However, this additional cost is relatively modest and is outweighed by the improve- ments in compression performance and model accuracy.
1. Sparse low rank (SLR) (Swaminathan et al., 2020): obtain low-rank matrices by imposing rank-sparsity constraints.
2. Naive SVD (Shmalo et al., 2023); set 45% small sin- gular values to zero.
Table 8. The computation time of different networks across various comparable algorithms.
5.2. Generalization and compression performance
In this section, we train the network on data sets with and without noisy labels, shuffling 60% of the labels to intro- duce noise. Then, we compress networks according to Algorithm 2 and Fig. 5. The test accuracy before and after compression for different methods is compared to evaluate the generalization performance of compressed networks. We denote the test accuracy before compression as the Base All codes the experiment are conducted on the server equipped with NVIDIA L40 GPUs and Ubuntu 22.04.
Network
PDB
PUB
Naive SVD
SLR
ResNet
19
15
18
23
VGG16
42
41
47
53
BERT
20
15
18
29
TS-base
47
44
45
42
VIT-L
236
231
243
256
Average
72.8
69.2
74.2
80.6
6. Conclusions
Our algorithm will recommend the best compression ratio of weight matrix where further compression will lead to significantly loss of generalization ability. Table 6 lists the rank of the weight matrix before and after compression for various neural networks under four algorithms across different datasets. Fig. reports the test accuracy of various compression algorithms for the dataset with and without label noise. From these results, we obtain the following conclusions:
In this work, propose a novel Population Double Bulk (PDB) model which more accurately characterize the singular value behavior of weight matrix than the exist- ing PUB model. An effective PDBLS algorithm is further developed for model estimation and determination of the noise-information boundary. Subsequently, we propose a PDB noise-filet filteriil algorithm to compress weight matrices. Our algorithms will also recommend the best compression ratio of weight matrix where further compression will lead to significantly loss of generalization ability. Experiments demonstrate superiority of our PDB model and effectiveness of our compression method.
1. Compared with the original neural network, our com- pressed model significantly reduces the rank of weight matrices while maintaining its test accuracy. In case of contaminated labels, our compression methods can even improve the test accuracy.
2. Compared with other compression algorithms, our al- gorithm can achieve better test accuracy at the same
8

---

Compress weight matrices
6955
015
014
0.00
035
PC2Sent
(a) FCNN on MNIST
(b) ResNet18 on CIFARIO
(c) Vggl6 on CIFARIO
(d) Vgg16 imagenet
See
(e) FCNN MNIST with noise (1) ResNet18 CIFARIO with (g) Vgg16 CIFARIO with noise
(h) Resnet imagenet
noise
Figure The vertical axis represents test accuracy and horizontal axis represents the proportion removed singular values of W in ascending order. The horizontal dashed black line reports test accuracy before compression. The vertical dashed lines represent the compression ratios recommended by different models, where red A4 for the PUB model and blue 3 for our PDB model.
Table Comparison of compression ratios across different compression algorithms, where the ratio equals the quotient of rank of matrix after and before compression.
Model
FCNN
ResNet18
VGG16
Datasets (0% noise)
MNIST
CIFARIO
ImageNet
CIFARIO
ImageNet
PDB(3)
55/350 (15.72%)
165/768 (21 (88%)
68/192 (35.42%)
208/1536 (13.54%)
200/768 (26.04%)
PUB(A4)
30/350 (8.57%)
80/768 (10.42%)
25/192 (13.02%)
84/1536 (5.47%)
123/768 (16.02%)
SLR
80/350 (22.86%)
200/768 (26.04%) )
100/192 (52.08%)
200/1536 (13.02%)
260/768 (33.85%)
Naive SVD
192/350 (55%)
422/768 (55%)
106/192 (55%)
845/1536 (55%)
422/768 (55%)
Table The accuracy obtained by training different network models of different data (0% noise) under the four compression methods. Base denotes the test accuracy before compression. SLR performs poorly on VIT-L (accuracy 0.1), thus omitted.
Network
Datasets
Base
PDB
PUB
SLR
naive SVD
FCNN
MNIST
0.9799
0.9804
0.9791
0.9799
0.9799
ResNet18
CIFARIO
0.8349
0.8384
0.8338
0.8357
0.8354
VGG16
CIFARIO
0.8418
0.8422
0.8405
0.8415
0.8419
RTE
0.7029
0.7319
0.7174
0.7246
0.7029
BERT
SciTail
0.9055
0.9155
0.9130
0.9008
0.9055
RTE
0.7174
0.7536
0.7319
0.7174
0.7174
T5-base
SciTail
0.9025
0.9243
0.9167
0.9182
0.9196
DTD
0.7452
0.7533
0.7482
0.7405
VIT-L
SUN397
0.7680
0.7771
0.7720
0.7716
Average
0.7783
0.7891
0.7827
0.7858
0.7789
Acknowledgments
Key R&D Program China(223YEA10114400 The au- thors are grateful to the reviewers for their thoughtful com- ments and suggestions.
This research is supported by National Nature Science Foun- dation of China NO. 1247 1258, NO. 1005 and National

---

Compress weight matrices
Impact Statement
In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 8049-8059, 2020.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Ke, Z. T., Ma, Y., and Lin, X. Estimation of the number of spiked cigenvalues in a covariance matrix by bulk eigenvalue matching analysis. Journal of the American Statistical Association, 1186441)3774-92, 2023.
References
Bai, Z. and Silverstein, J. W. Spectral analysis of large di- mensional random matrices, volume 20. Springer, 2010.
Khot, T., Sabharwal, A., and Clark, P. Scitail: A textual en- tailment dataset from science question answering. In Pro- ceedings of the 32nd AAAI Conference on Artificial Intel- ligence (AAAI-18), AAA''1 18/EAA/1/ AAAI Press, 2018.
Bai, 7. and Yao, J. On sample eigenvalues in a general- ized spiked population model. Journal of Multivariate Analysis, 106:167-177, 2012.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Berlyand, L., Sandier, E., Shmalo, Y., and Zhang, L. En- hancing accuracy deep learning using random matrix theory. arXiv preprint arXiv:2310.03165. 2023.
Leck, E. C., Leonardis, A., and Heinke, D. Deep neural networks and image classification in biological vision. Vision Research, 197:108058, 2022.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Li, W., Chen, J., Qin, Y., Bai, Z., and Yao, J. Estimation of the population spectral distribution from large dimen- sional sample covariance matrix. Journal of Statistical Planning and Inference, 143(1 143(11):1887-189) 2013.
Da'u, A. and Salim, N. Recommendation system based on deep learning methods: a systematic review and new directions. Artificial Intelligence Review, 53(4):2709- 2748, 2020.
Liebenwein, L.., Maalouf, A., Feldman, D., and Rus, D. Compressing neural networks: Towards determining the optimal layer-wise decomposition. Advances in Neural Information Processing Systems, 14:5328-5344 2021.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Liu, X., Liu, Y., Pan, G., Zhang, L., and Zhang, Z. Asymp- totic properties of spiked eigenvalues and eigenvectors of signal-plus-nois matrices with their applications. arXiv preprint arXiv:2310. 13939, 2023.
Devlin, J., Chang, M.-W.. Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 confer- ence of the North American chapter of the association for computational linguistics: human language technologies, volume I (long and short papers), pp. 4171-4186 2019.
Loshchilov, and Hutter, Decoupled weight decay regu- larization. arXiv preprint 171 2017.
Martin, C. H. and Mahoney, M. W. Heavy-tailed universality predicts trends in test accuracies for very large pre-trained deep neural networks. In Proceedings of the 2020 SIAM International Conference Data Mining, pp. 505-513. SIAM, 2020.
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Glorot, X. and Bengio, Understanding the difficulty of training deep feedforward neural networks. In Pro- ceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Martin, C. H. and Mahoney, M. W. Implicit self- regularization in deep neural networks: Evidence from random matrix theory and implications for learning. The Journal of Machine Learning Research, 22(1):747--751, 2021.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition, 2015.
Mohanty, A., Cherukuri, R. C., and Prusty, A. R. Improve- ment of speech emotion recognition by deep convolu- tional neural network and speech features. In Congress on Intelligent Systems, pp. 117-129. Springer, 2022.
Idelbayev, Y. and Carreira- Perpiia,, M. A. Low-rank com- pression of neural nets: Learning the rank of each layer.
10

---

Compress weight matrices
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763 PmLR,, 2021.
Yao, J., Zheng, S., and Bai, Z. Sample covariance matrices and high-dimensional data analysis. Cambridge UP, New York, 2015.
Raffel, C.,, Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, (140):1-67, 2020.
Shmalo, Y., Jenkins, J., and Krupchytskyi, O. Deep learning weight pruning with rmt-svd: Increasing accuracy and reducing overfitting. arXiv preprint arXiv:2303.08886, 2023.
Silverstein, J. W. and Choi, S.-L Analysis of the limiting spectral distribution of large-dimensional random matri- ces. J. Multivariate Anal., 54(2):295-309, 1995. ISSN 0047-259X.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition, 2015.
Staats, M., Thamm, M., and Rosenow, B. Boundary between noise and information applied to filtering neural network weight matrices. Physical Review E, L0223),, 2023.
Swaminathan, S., Garg, D., Kannan, R., and Andres, Sparse low rank factorization for deep neural network compression. Neurocomputingg 398:185-1966, 2020.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and anal- ysis platform for natural language understanding. CoRR, abs/1804.074611, 2018. URL http::/arxiv..r abs/1804. 07461.
Xiao, J., Ehinger, K. A., Hays, J., Torralba, A., and Oliva, A. Sun database: Exploring a large collection scene categories. Int. J. Comput. Vision, 199(1):3-2,, August 2016. ISSN 0920-5691. URL attps: / /ww.ioiooooooooooooooooo org/ 10. N0.1007/a11263-014--
Xu, Y., Li, Y., Zhang, S., and et al. Trained rank pruning for efficient deep neural networks. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-Neurlle Edition (EMC2-NIPS), pp. 14-17. IEFE, 2019.
Yang. Z., Dai, Z., Yang. Y., Carbonell, J., Salakhutdinov, R. R., and Le, Q. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.

---

Compress weight matrices
A. Experimental results
Table summarizes the test accuracy achieved by training various network models on various data sets using the four compression techniques. We can further show that our compression algorithm validated by the outcomes.
Table test accuracy obtained by training different network models different data sets under four compression methods. Base denotes the test accuracy before compression.
network
dataset
noise
Base
PDB
PUB
SLR
naive SVD
0%
0,9799
0.9804
0.9791
0.9799
0.9799
10%
0.9414
0.9454
0.9418
0.9439
0.9428
20%
0.8921
0.8968
0.8901
0.8918
0.8906
fc784 > fc512 + fc512
MNIST
30%
0.8013
0.8078
0.7934
0.7893
0.8033
fc350 fc10
40%
0.7247
0.7319
0.7119
0,7304
0.7278
50%
0.6520
0.6551
0.6381
0.6531
0.6387
60%
0.5455
0.5544
0.5331
0.5536
0.5401
0%
0.8349
0.8384
0.8338
0.8357
0.8354
10%
0.7360
0.7446
0.7439
0.7391
0.7382
20%
0.6701
0.6845
0.6831
0.6747
0.6709
Resnet18
CIFARIO
30%
0.6417
0.6576
0.6399
0.6187
0.6466
40%
0,4995
0.5051
0.5019
0.5027
0.5033
50%
0.4533
0.4607
0,4597
0,4496
0,4523
60%
0,2960
0.2982
0.2942
0.2975
0.2960
0%
0.8418
0.8422
0.8405
0.8415
0.8419
10%
0.7882
0.7937
0.7931
0.7935
0.7901
20%
0.7474
0.7496
0.7452
0.7449
0.7478
VGG16
CIFARIO
30%
0.6992
0.7048
0.6862
0.7011
0.6998
40%
0.6276
0.6341
0.6255
0.6303
0.6298
50%
0.5986
0.6122
0.6025
0.6048
0.6024
60%
0.5570
0.5686
0.5498
0.5591
0.5576
Resnet18
Imagenet
0%
0.5473
0.5469
0.5440
0.5467
0.5452
VGG16
Imagenet
0%
0.6160
0.6164
0.6126
0.6078
0.6076
RTE
0%
0.7029
0.7319
0.7174
0.7246
0.7029
BERT
SciTail
0%
0.9055
0.9155
0.9130
0.9008
0.9055
RTE
0%
0.7174
0.7536
0.7319
0.7174
0.7174
TS-base
SciTail
0%
0.9025
0.9243
0.9167
0.9182
0.9196
DTD
0%
0.7452
0.7533
0.7482
0.7405
VIT-
SUN397
0%
0.7680
0.7771
0.7720
0.7716
B. Spectral moment of PDB model
The spectral moment of a matrix an important statistic related its eigenvalues, often used to describe the spectral distribution the matrix. Denote the empirical spectral moment of WT
1 -
( j 1,2,3.
12

---

Compress weight matrices
Under PDB model, we have the following theoretical spectral moment,
72 122 + cui,
73 = 13 + 3cu1lu2 + n
71 -==1 Ml.
where
By f + (1 (1-1)62 j= 121,233,
and (1,01, come from PDBLS algorithm. Theorem B.1. Under PDB model (2) with p/n
as + oc, we have
9
7 an j j= 1,
(13)
C. The difference between PUB and PDB model In this section, we compare theoretical properties between PDB model and the PUB model. Recall the theory PDB model has been established in Theorems 3.1-3.3 and B.I. Next we list the corresponding results about PUB model(1).
The LSD of WTW W under PUB model has density function (Martin & Mahoney, 2021; Staats et al., 2023):
(11GG00²) V(++ - x) X_)
SOSA
(14)
2no2er
Here the boundary points A_ { = o1 VO2/ n.
And the sample spikes of W W have the following relationship
with population spikes a, of EWW W (Bai & Yao, 2012):
A = + caj
aj 2 j (1....., K].
(15)
Moreover, the relationship between the moments of W W and ELUB, mainly determined by the bulk eigenvalues, is established in (Yao et al., 2015):
1 (W"
as. + O, j= 1,233
7
P
where
14= = 021 j 1, 3. Here g2i can be obtained by BEMA (Ke et al., 2023) or Kernel (Staats et al., 2023) method.
71 Pr. 72 = H2 + cuT. 33 = M3 30 + -2 =2
(16)
The difference between the PUB and PDB model is summarized in Table 10.
Table 10. The difference between the PUB PDB model.
PUB model
PDB model
LSD
(14)
(3)-(4)
spiked eigenvalues
(15)
(6)-(7)
spectral moment
(16)
(13)
D. Proof of Theorem 3.1
Firstly, we give some preliminary knowledge about Stieltjes transform, which plays a key role in our proofs. For any distribution function G(x), its Stieltjes transform m²(z) is defined as
mG(2) = x -
dG(x), SECt
where C+ {a E C Im(2) 0} denotes the upper complex plane and Im(z) represents the imaginary part of z. Moreover G(x) and m²(2) have one-to-one relationship. The density function G" of G(x)) given by Theorem 2.1 in (Silverstein & Choi, 1995):
Im(ma(=)) 1-00
(x) = lim
2 == x x=
in.
(17)
π
13

---

Compress weight matrices
According the main results in (Bai & Silverstein, 2010), the Stieltjes transform of LSD of W W is m(=), which satisfies the following equations:
1
+c/ 1 um(z) dH(v),
+ f
v
(18)
m(z) 1-c + +cm(2),
m(2)
(19)
where (v) be the LSD of EPDB. By the relation m(=) -1 c)/2 + cm((2), along with Eq. (17), the density function of LSD of WT is
Im m(z)
Im m(2)
p(x) = lim
lim
x in. > 0,
we
from which we get Eq. (3).
And since the number of spiked eigenvalues K is fixed, have
1
H(v) = lim -
< (p- ((lp-)(1--00))) t) S v)
S0)+
p000 p 1=1
= v) + (1 (1- 1) u))
(20)
which implies
2 =
+
dH(()
((x)) (2)
+ cl
+ c(1 - 1) 1 0 m
(z)
Thus we obtain Eq. (4). The proof of Theorem 3.1 is complete.
E. Proof of Theorem 3.2
By Eq. (4), this inverse function of z + -1/m(z) is
to of
1)02 (1
g(x) = x + cz
+
cr
o2
Here g(x)= = and x = -1/m(z) in Eq. (4). Then according to Proposition 2.17 in (Yao et al., 2015) we obtain Theorem 3.2.
F. Proof of Theorem 3.3
By Theorem 4.1 in (Bai & Yao, 2012). the relationship between sample spikes A, and population spikes ay is established by
> aj + caa, /
dH(v), j (1. aj
Moreover, according Eq. (20), have
to?
(1 (1- t)o
a₂ dH(v)
+
Ojj o
Qj 02
from which for j E (1...., KY, we obtain
to²
(1 t)o
> + ca2
+ Co2
= g(a,))
7
aj
0j 0
Thus we complete the proof Theorem 3.3.
14

---

Compress weight matrices
G. Proof of Theorem 3.4
Recall estimators
Onulk = 02,,,0204 1, Obonnd =
{ { [}+.8}. Ospike
along with their corresponding population parameters
bulk o2.0. bound = ()+ Spiike (K,
We now proceed to establish the consistency of these estimators.
For Obulk: its consistency relies on the convergence of the empirical spectral distribution (ESD)
(0yso)
P
to the limiting spectral distribution (LSD) characterized in Theorem 3.1. Based on the ESD, obtain θbak via the procedures defined in Eq. (9)-(1).. Moreover, Eq. (4) ensures one-to-one correspondence between the population parameter Dbuk = { {oî 02 t and LSD. Therefore, by Theorem 3.1, the ESD converges almost surely to the LSD, and applying Theorem 3.1 in (Li et al., 2013), we conclude that
Obuik 2.5. bulk-
(21)
For Oround, the consistency of x_ relies the the convergence of Obulk and the results in Theorem 3.2, while the consistency of 3 depends on the convergence of K. Based on Obuik, compute 24 through Eq. (5)-6), i.e.
io? x of
(1 1o 02
d = (()), ú = arg max (#'()) = 0}, ((x) = x + Cx =ER
+ Cn
Ca = pp/n.
Moreover, Theorem 3.2 establishes a one-to-one correspondence between x and Obak- Therefore, by Eq. (21) and the as and
condition Gr > C, we have 24 ass, A4. Then by Eq. (11)-(12), we have K # (A > E (2+; Amax
hence 3 X Atpi-ki _ B. Thus we conclude that
Ohound
a.5 + bound-
(22)
For θ spiike its consistency relies the convergence of @batk. @bound and the results in Theorem 3.3. By the convergence K. Given @bulks according to Eq. (7), we compute à; by solving the equation
d A in Obound we have
((á,) = Aj s.t. f'(,,) >0, jE (L,....,,K K}.
Morcover, from Theorem 4.1 in (Bai & Yao, 2012) and Theorem 3.3, we have X + g(a)), 9 (aj) > 0, je (1...., K}.
Therefore by Eq. (21), K- us. K and c, - we obtain
spike
+ spskee
(23)
Finally, by Eq. (21), (22), (23), we complete the proof of Theorem 3.4.
H. Proof of Theorem B.1 According to Lemma 2.16 in (Yao al., 2015), have the following relationship:
71 ,1, 72 = μl2 cpt, 73 = Mt3 + 3044/12 + o² 2
15

---

Compress weight matrices
where
lim 1 tr (WTw)', Mj M-= foaH(()) j= 1, 7j p-000 p
Moreover, together with Eq. (20), we have
My = top + (1 t)02² j= 1,23.
And by the definition of EPDB,
1 lim tr(Enn)) = =to + (1 - (1-1)og j= 1,2,3 p0000 p
we obtain (13). Thus we complete the proof Theorem B.
16


Reviews:
[{"review_id": "CF6cqx2K2v", "reviewer": "Reviewer_nNCM", "summary": "This paper presents the Population Double Bulk (PDB) model, an extension of the Population Unit Bulk (PUB) model, for the effective compression of deep neural network weight matrices. The authors leverage a dual-cluster structure to more accurately analyze the eigenvalue distribution and employ the PDBLS algorithm to efficiently estimate the boundary between informative signals and noise. Additionally, they introduce the PDB Noise-Filtering algorithm, which selectively removes unnecessary eigenvalues while preserving crucial information, thereby improving compression performance.\n\nThe experimental results demonstrate that the proposed PDB-based compression technique can maintain the same test accuracy as existing methods at lower ranks or even enhance generalization performance. Furthermore, the approach exhibits robustness in noisy data scenarios, suggesting its practical applicability in real-world deep learning tasks.\n\n## update after rebuttal\nMy main concern was the lack of evaluation on other models such as LLMs and the comparison with SVD-based methods, which was well addressed in the authors’ reply. For this reason, I have raised my score from 2 to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your recognition and the valuable suggestions. Please find our response below.(**T** for Theoretical Claims, **W** for weakness, **Q** for Questions)\n\n**1. Comparison with additional methods [W1]**\n\nWe include additional experiments on two other methods: \n\n​    1. naive SVD (using 0.55 as the empirical compression ratio, Shmalo et al. (2023)),\n\n​    2. sparse low-rank (SLR) (Sridhar Swaminathan et al., *Neurocomputing*, 2020). \n\nTable below shows accuracy and compression ratio (ratio of rank of $W$ after and before compression).\n\n| Network          | Noise | Base  |       PDB        |      PUB      |  Naive SVD  |      SLR      |\n| ---------------- | :---: | :---: | :--------------: | :-----------: | :---------: | :-----------: |\n| FCNN:MNIST       |  0%   | 0.98  |   0.98 (15.7%)   | 0.979  (8.6%) |    0.98     | 0.98 (15.6%)  |\n|                  |  30%  | 0.801 |  0.808 (30.1%)   | 0.793 (13.9%) |    0.803    | 0.789 (27.3%) |\n| ResNet18:CIFAR10 |  0%   | 0.835 |  0.838 (21.5%)   | 0.834 (10.4%) |    0.835    |  0.836 (26%)  |\n|                  |  30%  | 0.642 |  0.658 (15.8%)   | 0.64  (7.2%)  |    0.647    | 0.619 (11.1%) |\n| VGG16:CIFAR10    |  0%   | 0.842 |  0.842 (13.5%)   | 0.841  (5.5%) |    0.842    |  0.842 (13%)  |\n|                  |  30%  | 0.699 |  0.705  (6.4%)   | 0.686  (4.3%) |     0.7     | 0.701  (9.1%) |\n| **Average**      |       | 0.800 | **0.805(17.2%)** | 0.795  (8.3%) | 0.801 (55%) | 0.794 (17.2%) |\n\n**2. Additional experiments on LLM [W2,W3,Q1,Q2]**\n\nWe include additional experiments on BERT, ViT, and T5-base. Table below presents accuracy and compression ratio. SLR performs poorly on VIT(accuracy 0.1), thus omitted.\n\n|     Network      | Base  |       PDB        | SLR           | PUB          | naive SVD   |\n| :--------------: | ----- | :--------------: | ------------- | ------------ | ----------- |\n|    BERT: RTE     | 0.703 |   0.732 (2.3%)   | 0.717 (3.3%)  | 0.717 (9.6%) | 0.703       |\n|  BERT: SCITAIL   | 0.906 |   0.916 (2.1%)   | 0.918 (3.3%)  | 0.913 (12%)  | 0.906       |\n|   T5-base: RTE   | 0.717 |   0.754 (1.7%)   | 0.725 (16.3%) | 0.732 (8.1%) | 0.717       |\n| T5-base: SCITAIL | 0.921 |   0.924 (2.3%)   | 0.901 (9.8%)  | 0.917 (4.7%) | 0.92        |\n|    VIT-L: DTD    | 0.745 |   0.753 (11%)    | -             | 0.748 (8.2%) | 0.741       |\n|  VIT-L: SUN397   | 0.768 |  0.777 (13.4%)   | -             | 0.772 (11%)  | 0.772       |\n|   **Average**    | 0.793 | **0.809 (5.5%)** | -             | 0.800 (8.9%) | 0.793 (55%) |\n\n**3.Computational efficiency [W4]**\n\nThe table below compares the computational efficiency, with time measured in seconds for total execution and inference after data processing on NVIDIA L40 GPUs (Ubuntu 22.04).\n\n|   DNN   |   PDB    | PUB  | naive SVD | SLR  |\n| :-----: | :------: | :--: | :-------: | :--: |\n|  VGG16  |    42    |  41  |    47     |  53  |\n| Resnet  |    19    |  15  |    18     |  23  |\n|   VIT   |   236    | 231  |    243    | 256  |\n| T5-base |    47    |  44  |    45     |  42  |\n|  Bert   |    20    |  15  |    18     |  29  |\n| Average | **72.8** | 69.2 |   74.2    | 80.6 |\n\n**4. Validity of PDB[T1]**\n\n**On the Empirical Side**: Our  motivation is that the spectral distribution of $W^TW$ does not perfectly align with PUB. This led us to consider more general models. Since the size of $W$ is typically too small to guarantee an accurate estimation of a continuous population bulk, we opted for a discrete surrogate of a M-bulk model ($M\\geq2$). However, through extensive experiments (Section 5 included), we found that the proportions of  **bulks beyond M=2** are nearly negligible (Table 1). Therefore, we adopt PDB as a practical and effective representation.\n\n**On the Theoretical Side**: The entries of the initial $W_0$ are pure noise, and $EW_0^TW_0=\\sigma_0^2I_p$. The eigenvalues of $W_0^TW_0$ follow unimodal MP.  As training progresses, $W$ is continuously updated with new gradients, altering the structure of $EW^TW$. Martin \\& Mahoney (JMLR, 2021) conducted extensive analyses on the eigenvalue distribution of $W^TW$ during training. They claim that, as training proceeds, eigenvalue distribution becomes bimodal (Figures 11,18 in their paper). We provide a theoretical explanation (PDB for $EW^TW$) for this phenomenon and establish a systematic compression framework.\n\n**5.  Additional moments comparison[T2]**\n\nWe compare the first 3 theoretical spectral moments of $W$ with the empirical values on additional neural networks.\n\n|           | |   T5-base: RTE     |       |  |     Bert: SCITAIL  |       |\n| :-------: | :----------: | :---: | :---: | :-----------: | :---: | :---: |\n|  Moments  |    $\\gamma_1$     | $\\gamma_2$ | $\\gamma_3$ |     $\\gamma_1$     | $\\gamma_2$ | $\\gamma_3$ |\n|    PUB    |     0.67     |  0.9  | 1.51  |     0.59      | 0.69  | 1.02  |\n|    PDB    |     0.77     | 1.55  | 4.17  |     0.67      | 1.18  | 2.78  |\n| empirical |     0.72     | 1.83  | 5.35  |     0.71      | 1.38  | 3.95  |", "claims_and_evidence": "- The paper conducts LSD (eigenvalue distribution) comparison experiments, asserting that the PDB model provides a more accurate representation of the weight matrices in real neural networks than the PUB model.\n- The study suggests that the PDB model can enhance generalization performance by regulating the Lipschitz constant.\n- While the paper claims that PDB-based compression outperforms the PUB-based approach, a more comprehensive comparative analysis with competing methods is necessary to elucidate the reasons behind its superior performance.", "methods_and_evaluation_criteria": "- To address the limitations of the existing PUB model, the authors introduce the Population Double Bulk (PDB) model and develop the PDBLS algorithm and PDB Noise-Filtering algorithm based on it.\n- Notably, the proposed approach effectively analyzes the eigenvalue distribution of neural networks and applies spectral analysis using Random Matrix Theory (RMT) to provide theoretical justification for the compression method.\n- However, since the evaluation is limited to a comparison with the PUB model, the study lacks a broader assessment against state-of-the-art compression techniques.\n- Additionally, it remains unclear whether the proposed method is optimized specifically for image data or if it can be generalized to other modalities, such as speech or text data. Further experiments on diverse datasets would strengthen the claims regarding the method’s applicability.", "theoretical_claims": "- While the PUB model assumes a single variance (σ²), the PDB model extends this assumption by incorporating two bulk distributions, claiming to better model general cases. However, additional experimental and theoretical validation is required to determine whether the number of bulks is inherently limited to two in real-world data.\n- The study leverages Random Matrix Theory (RMT) to demonstrate that the eigenvalue distribution in the PDB model follows a specific equation, ensuring theoretical consistency. However, further empirical validation on diverse neural network architectures is necessary to confirm the practical applicability of the proposed approach.", "experimental_designs_or_analyses": "- The paper evaluates the PDB model against the PUB model using a range of neural networks (FNN, ResNet18, VGG16) and benchmark datasets (MNIST, CIFAR-10, ImageNet).\n- To assess generalization, experiments were also conducted on noisy data.\n- However, the study focuses solely on CNN-based architectures, leaving open the question of whether the PDB model would perform similarly well in Transformer-based models like BERT or ViT.\n- Additionally, the method has only been tested on computer vision tasks, and its effectiveness in other domains, such as NLP or speech recognition, remains unexplored.", "relation_to_broader_scientific_literature": "- Previous research has used the Population Unit Bulk (PUB) model to analyze the eigenvalue distribution of weight matrices, but its single-variance assumption fails to fully capture the complexity of real neural networks.\n- To address this, the paper introduces the Population Double Bulk (PDB) model, which considers two bulk distributions, allowing for a more realistic representation of eigenvalue behavior.\n- However, further comparative experiments with existing compression methods are needed to better assess the practical performance of the PDB model.", "essential_references_not_discussed": "- None", "other_strengths_and_weaknesses": "Strengths\n- This paper expands the spectral analysis of neural network weight matrices by introducing the PDB model, which provides a more accurate explanation of eigenvalue distributions compared to the existing PUB model.\n- By applying Random Matrix Theory (RMT), the study establishes a theoretical foundation for understanding eigenvalue behavior and introduces the PDBLS algorithm, which enables a more precise separation between informative signals and noise.\n- The effectiveness of the proposed approach is demonstrated through experiments on a variety of neural network architectures (FNN, ResNet18, VGG16) and benchmark datasets (MNIST, CIFAR-10, ImageNet), highlighting its practical applicability.\n- The PDB model shows strong generalization performance, even in the presence of noisy data, and in some cases, it improves over existing methods, suggesting its potential to help mitigate overfitting.\n\nWeaknesses\n- The study does not include a direct comparison with other well-established neural network compression techniques (e.g., SVD), making it difficult to determine whether the PDB-based approach offers a clear advantage.\n- The experiments are limited to CNN-based architectures, leaving it unclear how well the PDB model performs on other structures, such as Transformers (BERT, ViT).\n- The method has only been tested on computer vision tasks, so its applicability to other domains, such as natural language processing (NLP) and speech recognition, remains uncertain.\n- (Optional) The paper does not discuss the computational efficiency of the compressed models (FLOPs reduction, inference speed, memory savings), making it hard to assess their practical benefits.", "other_comments_or_suggestions": "- None", "questions_for_authors": "- Can the authors provide experiments on how well the PDB model performs on other architectures such as Transformers (BERT, ViT)?\n- Could the authors provide experiments on application to other domains such as Natural Language Processing (NLP)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GJW3f04S5q", "reviewer": "Reviewer_4M7a", "summary": "The paper proposed a population double bulk model for compressing weight matrices in neural networks. Compared to previous pupulation unit bulk model, the PDB model has more parameters and better approximates the weight matrices. Theoretical analysis (drawing tools from random matrix theory) and algorithms are proposed to estimate the parameters in PDB model. Experiments on neural networks shows an accuracy improvement using PDB model compared to using PUB model.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your recognition and the valuable suggestions. Please find our response below. (**W** for weakness)\n\n**1. Model details [W1]**\n\nThe entries of the **initial** weight matrix  $W_0$ are **i.i.d.** with mean 0 and variance $\\sigma^2$, resulting in $\\mathbb{E}W_0^TW_0=\\sigma_0^2 I_p$. As **training progresses**, PDB assumes that $\\mathbb{E}W^TW=\\operatorname{diag}\\left(\\alpha_1, \\ldots, \\alpha_K, \\sigma_1^2, \\ldots, \\sigma_1^2, \\sigma_2^2, \\ldots, \\sigma_2^2\\right)$ , indicating that the entries are **no longer i.i.d.** due to  heterogeneous variances.\n\nWe use  the density curve (Figure 3) and spectral moments of $W$ (Tables 3-4) to assess model approximation performance. We also compare the first three theoretical spectral moments with the empirical values on **additional neural networks**.\n\n|           |  |     T5-base: RTE  |       | |      Bert: SCITAIL  |       |\n| :-------: | :----------: | :---: | :---: | :-----------: | :---: | :---: |\n|  Moments  |    $\\gamma_1$     | $\\gamma_2$ | $\\gamma_3$ |     $\\gamma_1$     | $\\gamma_2$ | $\\gamma_3$ |\n|    PUB    |     0.67     |  0.9  | 1.51  |     0.59      | 0.69  | 1.02  |\n|    PDB    |     0.77     | 1.55  | 4.17  |     0.67      | 1.18  | 2.78  |\n| empirical |     0.72     | 1.83  | 5.35  |     0.71      | 1.38  | 3.95  |\n\nActually, the method proposed by Staats et al. (Physical Review E, 2023) is **data-driven** and  we provide a  detailed comparison in Table 6 of our paper Appendix.  We also include **additional experiments on BERT and T5-base** and compare with **two other compression methods** : \n\n 1.naive SVD (using 0.55 as the empirical compression ratio, Shmalo et al. (2023)),\n\n 2.sparse low-rank (SLR) (Sridhar Swaminathan et al., *Neurocomputing*, 2020). \n\nThe table  below shows the accuracy and compression ratio (ratio of rank of $W$ after and before compression) .\n\n|     Network      | Noise | Base  |       PDB        |     SLR      |     PUB      | naive SVD  |\n| :--------------: | :---: | :---: | :--------------: | :----------: | :----------: | :--------: |\n|    FCNN:MNIST    |  30%  | 0.801 |   0.808(30.1%)   | 0.789(27.3%) | 0.793(13.9%) |   0.803    |\n| ResNet18:CIFAR10 |  30%  | 0.642 |   0.658(15.8%)   | 0.619(11.1%) | 0.64 (7.2%)  |   0.647    |\n|  VGG16:CIFAR10   |  30%  | 0.699 |   0.705(6.4%)    | 0.701(9.1%)  | 0.686 (4.3%) |    0.7     |\n|    BERT: RTE     |  0%   | 0.703 |   0.732(2.3%)    | 0.717(3.3%)  | 0.717 (9.6%) |   0.703    |\n|  BERT: SCITAIL   |  0%   | 0.906 |   0.916(2.1%)    | 0.918(3.3%)  |  0.913(12%)  |   0.906    |\n|   T5-base: RTE   |  0%   | 0.717 | **0.754(1.7%)**  | 0.725(16.3%) | 0.732(8.1%)  |   0.717    |\n| T5-base: SCITAIL |  0%   | 0.921 |   0.924(2.3%)    | 0.901(9.8%)  | 0.917(4.7%)  |    0.92    |\n|     Average      |       | 0.77  | **0.785(8.67%)** | 0.767(11.5%) | 0.771(8.5%)  | 0.771(55%) |\n\nIt can be seen that while the compression raio of PDB lies between PUB and SLR, it achieves the best accuracy, with a **maximum improvement of 5.2%** ($\\frac{0.754}{0.717}-1$).\n\n**2. About overfitting [W2]**\n\nOur primary goal is to **reduce model complexity** while **preserving generalization ability**. During the matrix compression process, some model information is inevitably lost. Our approach aims to **identify the optimal boundary between noise and information**, effectively removing **noisy eigenvalues** to maintain the model's generalization without excessive information loss. When the data is clean, the impact of noisy eigenvalues is minimal, making the difference between compressed and uncompressed matrices negligible. However, when the data quality is poor, the noise embedded in $W$ can negatively affect test accuracy. Since our compression method essentially **filters out noise**, it **enhances generalization** and improves the model’s robustness. Therefore, our method shows a more significant advantage in the presence of noise, whereas when the data quality is good, the difference between compressed and uncompressed models is minimal.\n\n**3. Is the matrix compression performed for each layer or specific layers? [W3]**\n\nWe **selectively compress specific layer matrices**, particularly those where the size of $W$ is sufficiently large. For smaller matrices, the **limited number of eigenvalues** results in **less accurate model estimation**, making compression ineffective.\n\n**4. About spike and bulk phase [W4]**\n\nMartin & Mahone (JMLR, 2021) conducted a **comprehensive and in-depth analysis** of the eigenvalue distribution of $W^TW$ throughout the training process. They found that when the DNN weight matrices exhibit **weak self-regularization**，**eigenvalue mass shifts to larger values**, forming the Bulk+Spikes phase during the first few epochs . Once the spikes appear, **substantial changes** in the distribution become hard to see.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Makes sense to me.", "theoretical_claims": "Looks correct to me.", "experimental_designs_or_analyses": "Looks reasonable to me.", "relation_to_broader_scientific_literature": "The paper discussed the problem of reducing overfitting through the compression of weight matrices, which seems to be of interest.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths: The paper provides solid works including proposing an extended model of PUB, providing corresponding theoretical analysis and algorithms to estimate the parameters, and running experiments on neural networks that show improvement.\n\nWeaknesses: These are not necessarily weaknesses but more of questions.\n\n1. Is it assumed that the weight matrices are i.i.d.? How accurately does the PUB/PDB model approximate the real weight matrices. Were comparisons between PUB and data-driven methods made in existing literatures? Is it possible to comment on how well PUB/PDB models perform compared to data-driven methods?\n\n2. While it is mentioned that matrix compression is set out to mitigate the overfitting, it seems that no compression performs almost equally well in the experiments and better when the dataset is good enough.\n\n3. Is the matrix compression performed for each layer or specific layers?\n\n4. It was mentioned that in the training process the third phase, the flat-tail phase is less common. Is it the case that most models stop at the spike and bulk phase during the training proccess?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vxoSvQuUUS", "reviewer": "Reviewer_9rcu", "summary": "The paper introduces the population double bulk (PDB) model, an extension of the population unit bulk (PUB) model, to provide a more accurate description of the spectral properties of the weights in deep neural networks. In the PDB model, the informative components of the spectrum are captured by the spikes and the first bulk, while the second bulk corresponds to noise. Empirical results demonstrate that the PDB model outperforms the PUB model in terms of both performance and alignment. Furthermore, the PDB model is used to compress the weight matrices of convolutional neural networks (CNNs) into a low-rank structure by discarding the singular values associated with the second bulk. Networks compressed using the PDB model achieve superior accuracy compared to those compressed with the PUB model.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your recognition and the valuable suggestions. Please find our response below.\n\n(**C** for Claims And Evidence, **E** for Experimental Designs,  **W** for weakness, **Q** for Questions)\n\n**1. Whether the ratio is the best [C]**\n\nOur primary goal is to identify the noise-information boundary and remove the noisy eigenvalues. Once the signal eigenvalues are removed, accuracy begins to degrade. Therefore, we aim to locate the point where **accuracy first starts to decline**, rather than the point of significant decline. In Figure 6, a **significant drop in accuracy** indicates that the model is **over-compressed**. The boundary point $\\beta$ from PDB better captures the **initial decline points** compared to $\\lambda_{+}$from  PUB, showing that PDB better aligns with the behavior of eigenvalues of $W$.\n\nMoreover, most $\\beta$ in Figure 6 are located near the initial decline points, though a few cases are less obvious. This is because $\\beta$ is an asymptotic limit based on PDB when the dimension of $W$ tends to infinity. There exists a gap between the asymptotic limit and the finite-sample empirical value. Additionally, the random error of  PDB estimate can also contribute to this effect.\n\n**2. Additional experiments [E,W2,Q1]**\n\nWe are sorry for the lack of a detailed comparison and clear explanation in the main text. We include **additional experiments on BERT and T5-base** and compare with **two other methods** : \n\n​    1. naive SVD (using 0.55 as the empirical compression ratio, Shmalo et al. (2023)),\n\n​    2. sparse low-rank (SLR) (Sridhar Swaminathan et al., *Neurocomputing*, 2020). \n\nThe table below presents the accuracy and compression ratio (ratio of rank of $W$ after and before compression). \n\n|     Network      | Noise | Base  |       PDB        |     SLR      |     PUB      | naive SVD  |\n| :--------------: | :---: | :---: | :--------------: | :----------: | :----------: | :--------: |\n|    FCNN:MNIST    |  30%  | 0.801 |   0.808(30.1%)   | 0.789(27.3%) | 0.793(13.9%) |   0.803    |\n| ResNet18:CIFAR10 |  30%  | 0.642 |   0.658(15.8%)   | 0.619(11.1%) | 0.64 (7.2%)  |   0.647    |\n|  VGG16:CIFAR10   |  30%  | 0.699 |   0.705(6.4%)    | 0.701(9.1%)  | 0.686 (4.3%) |    0.7     |\n|    BERT: RTE     |  0%   | 0.703 |   0.732(2.3%)    | 0.717(3.3%)  | 0.717 (9.6%) |   0.703    |\n|  BERT: SCITAIL   |  0%   | 0.906 |   0.916(2.1%)    | 0.918(3.3%)  |  0.913(12%)  |   0.906    |\n|   T5-base: RTE   |  0%   | 0.717 | **0.754(1.7%)**  | 0.725(16.3%) | 0.732(8.1%)  |   0.717    |\n| T5-base: SCITAIL |  0%   | 0.921 |   0.924(2.3%)    | 0.901(9.8%)  | 0.917(4.7%)  |    0.92    |\n|     Average      |       | 0.77  | **0.785(8.67%)** | 0.767(11.5%) | 0.771(8.5%)  | 0.771(55%) |\n\nOur method achieves the best accuracy. Moreover, Figure 6 presents the parameters for both PDB and PUB. The vertical dashed lines represent the suggested optimal compresion ratio, $\\beta$ for PDB and  $\\lambda_+$ for PUB. \n\n**3. Assumptions to produce the bulk+spike phase [W1]**\n\nMartin & Mahone (JMLR, 2021) point out that the bulk+spike phase typically emerges when $W$ exhibit **weak self-regularization** during training. To the best of our knowledge, so far there is no rigorous mathematical hypothesis specifying when the bulk+spike phase occurs. It is more of an **empirically observed phenomenon**. Both PUB and PDB provide mathematical frameworks for modeling this phenomenon.\n\n**4. About noise-information boundary $\\beta$ [W3,Q3]**\n\nThe entries of the **initial** $W_0$  are pure noise, and $EW_0^TW_0=\\sigma_0^2I_p$. The eigenvalues of $W_0^TW_0$ follow  a  unimodal MP law which contains **no information**. As training progresses, the weight matrix is updated with new gradients, which alters the structure of  $EW^TW$.  Martin \\& Mahoney (JMLR, 2021) conducted extensive analyses on the eigenvalue distribution of $W^TW$ during training. They claim that, as training proceeds, the bulk distribution becomes **bimodal** and **large eigenvalues emerge**, forming the **bulk+spike phase**, see Figures 11,18 in their paper.  We provide a theoretical explanation for this phenomenon,  which is PDB for $EW^TW$. Correspondingly, we treat the **emerging additional bulk and spikes** as **information**, while the **smaller bulk** is still **noise**.  $\\beta$  separates the first and second bulk, thus it's chosen as the noise-information boundary.\n\n**5. Key motivations for choosing  PDB [Q2]**\n\nBoth PDB and PUB are low-rank compression methods. The key challenge lies in choosing the optimal compression ratio. We first provide a mathematical framework PDB for the eigenvalues in the bulk+spike phase. Then the compression ratio is determined based on their own eigenvalue behavior. Additionally, PDB debiases large eigenvalues to enhance robustness. It can improve accuracy with noisy labels. In contrast, naive methods rely on **empirical heuristics** (same ratio for all matrices) and don't correct for signal eigenvalues, lacking theoretical support.", "claims_and_evidence": "Tables 3-4 and Figure 3 demonstrate that the PDB model captures the empirical observations more effectively than the PUB model.\n\nThe paper claims that the proposed algorithms offer the optimal compression ratio. While Figure 6 provides some support for this claim, there is a concern regarding whether the ratio is indeed the best recommendation. Specifically, the performance does not show a significant decline immediately after the values of $\\beta$ or $\\lambda_+$ in Figure 6.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriately chosen for the problem.", "theoretical_claims": "I checked the correctness of the proof of Theorem 3.1 and it seems correct.", "experimental_designs_or_analyses": "The experimental setup for the DNN training is consistent with established practices in the literature. However, it remains unclear whether the PDBLS and noise-filtering algorithms offer a better compression ratio than the PUB models. Figure 6 presents only the parameters of the PDB models, and Table 5 displays only the compression results for the PDB model, making it difficult to directly compare the compression quality between PDB and PUB models.\n\nAdditionally, it would be insightful to include a comparison between the PDB model and a naïve low-rank compression method that selects the top-K singular values.", "relation_to_broader_scientific_literature": "The algorithm to identify the signal and the noise in the spectral density would be useful in many fields of machine learning and numerical methods for scientific applications.", "essential_references_not_discussed": "Related works are well addressed.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The paper is well-motivated by the fact that the PUB model is based on restrictive assumptions. The proposed spectral model effectively addresses this issue, enhancing the practicality of the population bulk model family.\n2. The proposed method has significant potential for adoption across various subcategories of machine learning and deep learning.\n\n**Weaknesses**\n\nMost of my concerns relate to clarity.\n\n1. It is unclear under what assumptions the training process produces the bulk+spike phase.\n2. The experimental results could be further strengthened, as discussed in the \"Experimental Designs or Analyses\" section. Specifically, the proposed matrix compression scheme should be compared with both the PUB model and the naïve top-K singular value selection method.\n3. Although the double bulk model resembles the empirical spectral distribution, it is unclear why the first bulk is considered the information criterion, rather than the noise criterion.", "other_comments_or_suggestions": "The text in the figures is too small. It is recommended to increase the font size in the figures for better readability.", "questions_for_authors": "1. How much improvement in compression quality (e.g., accuracy change vs. compression ratio) does the proposed PDBLS + noise-filtering algorithm offer compared to the PUB model or naïve low-rank compression methods?\n\n2. What are the key motivations for choosing the PDB model over the naïve low-rank compression methods?\n\n3. Could the authors provide insights or a theoretical discussion on why $\\beta$ is chosen as the information-noise boundary? Does the double-bulk distribution emerge naturally during the training process?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L4IAnWFmsn", "reviewer": "Reviewer_LPdE", "summary": "This paper studies the problem of DNN weight compression for seek of better generalization. They propose the Population Double Bulk (PDB) model to characterize the eigenvalue behavior of $\\mathbf{W}^\\top\\mathbf{W}$ in the bulk+spikes phase during DNN training, generalizing the existing Population Unit Bulk (PUB) model to allow for finer characterization of the bulk eigenvalues (from one bulk in PUB to two bulks in PDB). The paper further investigates the asymptotic limits of the proposed model based on Random Matrix Theory (RMT), which allows for new algorithm design to (i) estimate the PDB model parameters based on observed empirical eigenvalues; and (ii) compress the model based on the estimated PDB model by pruning the eigenvalues inside the smaller bulk (second bulk). Empirical results demonstrate the effectiveness of the proposed method in terms of improving generalization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your recognition and the valuable suggestions. Please find our response below.\n\n(**T** for Theoretical Claims, **W** for Weakness)\n\n**1. Detailed proof of Theorem 3.4  [T]**\n\nWe are sorry for the overly simplified proof of Theorem 3.4 in our paper.  We will include more detailed and concrete proofs of Theorem 3.4 in the final version. Below is a brief outline.\n\n1. For $\\widehat\\Theta_\\text{bulk}$ , its consistency relies on the convergence of ESD $F_n^{W^TW}(x)=\\frac1p\\sum_\\text{j=1}^p \\mathbb{1}(\\lambda_j \\leq x)$ to the LSD (defined on page 1 in our paper). Specifically,  \n\n$$\n\\widehat\\Theta_\\text{bulk} \\xLeftrightarrow{Eq. (9)(10)} ESD\\xrightarrow{\\text{Theorem 3.1}} LSD \\xLeftrightarrow{Eq. (4)} \\Theta_\\text{bulk}\n$$\n\n2. For $\\widehat\\Theta_\\text{bound}$, its consistency relies on the convergence of $\\widehat{\\Theta}_\\text{bulk}$. Specifically, \n\n$$\n\\widehat\\Theta_\\text{bound} \\xLeftrightarrow{Eq. (5)(11)(12)} \\widehat\\Theta_\\text{bulk} \\xrightarrow{\\text{Theorem 3.1}} \\Theta_\\text{bulk} \\xLeftrightarrow[Eq.(5)(6)]{\\text{Theorem 3.2}} \\Theta_\\text{bound}\n$$\n\n3. For $\\widehat\\Theta_\\text{spike}$, its consistency relies on the convergence of $\\widehat\\Theta_\\text{bulk}$, $\\widehat\\Theta_\\text{bound}$ and $\\lambda_\\text{spike}$. Specifically, \n\n$$\n\\widehat\\Theta_\\text{spike}\\xLeftrightarrow{Eq. (6)(7)(11)}\\widehat\\Theta_\\text{bulk}, \\widehat\\Theta_\\text{bound},\\lambda_\\text{spike} \\xrightarrow{\\text{Theorems 3.1-3.3}}\\Theta_\\text{bulk},\\Theta_\\text{bound},g(\\alpha_\\text{spike})\\xLeftrightarrow{Eq.(7)(11)}\\Theta_\\text{spike}\n$$\n\n**2.About improvement of test accuracy  and comparison with additional methods [W1,W2]**\n\nIn model compression, besides accuracy, another important metric is the **compression ratio** (ratio of rank of $W$ after and before compression). We aim to compress $W$ while maintaining accuracy, seeking **a balance between accuracy and compression ratio**, rather than merely pursuing high accuracy. In this rebuttal, we include **additional experiments on BERT and T5-base** and compare with **two other compression methods** : \n\n​    1. navie SVD (using 0.55 as the empirical compression ratio, Shmalo et al. (2023)),\n\n​    2. sparse low-rank (SLR) (Sridhar Swaminathan et al., *Neurocomputing*, 2020). \n\nThe table below presents the accuracy and compression ratio. \n\n|     Network      | Noise | Base  |       PDB        |     SLR      |     PUB      | naive SVD  |\n| :--------------: | :---: | :---: | :--------------: | :----------: | :----------: | :--------: |\n|    FCNN:MNIST    |  30%  | 0.801 |   0.808(30.1%)   | 0.789(27.3%) | 0.793(13.9%) |   0.803    |\n| ResNet18:CIFAR10 |  30%  | 0.642 |   0.658(15.8%)   | 0.619(11.1%) | 0.64 (7.2%)  |   0.647    |\n|  VGG16:CIFAR10   |  30%  | 0.699 |   0.705(6.4%)    | 0.701(9.1%)  | 0.686 (4.3%) |    0.7     |\n|    BERT: RTE     |  0%   | 0.703 |   0.732(2.3%)    | 0.717(3.3%)  | 0.717 (9.6%) |   0.703    |\n|  BERT: SCITAIL   |  0%   | 0.906 |   0.916(2.1%)    | 0.918(3.3%)  |  0.913(12%)  |   0.906    |\n|   T5-base: RTE   |  0%   | 0.717 | **0.754(1.7%)**  | 0.725(16.3%) | 0.732(8.1%)  |   0.717    |\n| T5-base: SCITAIL |  0%   | 0.921 |   0.924(2.3%)    | 0.901(9.8%)  | 0.917(4.7%)  |    0.92    |\n|     Average      |       | 0.77  | **0.785(8.67%)** | 0.767(11.5%) | 0.771(8.5%)  | 0.771(55%) |\n\nOur method achieves the best accuracy, with a **maximum improvement of 5.2%** ($\\frac{0.754}{0.717}-1$), and maintains a competitive compression rate, reaching 1.7%  for T5-base: RTE.\n\n Moreover, in the model compression literature, accuracy improvements in general are not substantial, e.g., 0.7% in Hyeji Kim et al. (CVPR, 2019) and 0.36%–0.55% in Georgios Georgiadis (CVPR, 2019). \n\n**3. Computational overhead of compression algorithm [W3]**\n\nThe table below compares the computational efficiency, with time measured in seconds for total execution and inference after data processing on NVIDIA L40 GPUs (Ubuntu 22.04).\n\n|   DNN   |   PDB    | PUB  | naive SVD | SLR  |\n| :-----: | :------: | :--: | :-------: | :--: |\n|  VGG16  |    42    |  41  |    47     |  53  |\n| Resnet  |    19    |  15  |    18     |  23  |\n|   VIT   |   236    | 231  |    243    | 256  |\n| T5-base |    47    |  44  |    45     |  42  |\n|  Bert   |    20    |  15  |    18     |  29  |\n| Average | **72.8** | 69.2 |   74.2    | 80.6 |\n\n As shown, the computational overhead of our PDB model is slightly higher than PUB. However, this additional overhead is relatively modest and is outweighed by the improvements in compression performance and model accuracy.", "claims_and_evidence": "Mostly yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. But the reviewer is concerned with the proof of Theorem 3.4 (Appendix G). The proofs are overly simplified and should be more concrete to make the paper self content and convincing.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The paper contributes to the literature of DNN weight compression based on the model of population bulks. Specifically, the paper proposes a finer characterization of the limiting spectral distribution via including two classes of bulked eigenvalues. New algorithms to compress the weight matrices are proposed accordingly.", "essential_references_not_discussed": "To the knowledge of the reviewer there are no lacking essential references.", "other_strengths_and_weaknesses": "**Strengths:** From the reviewer's perspective, the most important contribution of the paper is that the proposed new model of the weight matrix LSD (PDB) gives better approximation of the empirical spectral distribution given the experimental results shown in the paper. This better approximation also results in potentially improved compression algorithm in terms of generalization performance. The model is clean and quite direct to obtain from existing models.\n\n**Weakness:**\n1. The improvement of the testing accuracies w.r.t. the base model on the tasks considered in the paper is relatively limited (mostly less than 1%), which makes the strength of the overall method weakened.\n2. The paper only discusses PUB model and the compression methods based PUB. Little is discussed and compared about other potential compression based method for improving generalization.\n3. What is the computational overhead of the compression algorithm proposed in this paper compared with the algorithms mentioned in the paper that are based on PUB? The paper does not touch on this perspective.", "other_comments_or_suggestions": "Please see the **Weakness** part of the review.", "questions_for_authors": "Please see the **Weakness** part of the review.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a population double bulk (PDB) model for compressing weight matrices in neural networks, improving upon prior approaches such as the population unit bulk (PUB) model. 
The authors provide both a theoretical analysis (with random matrix theory) and algorithmic procedures for estimating the parameters of the PDB model. 
Experimental results on neural networks demonstrate improved accuracy when using the proposed PDB model compared to the PUB baseline.

The authors have done an excellent job during the rebuttal phase, and all reviewers are now convinced of the significance and merit of the work.

As a result, I recommend acceptance of this paper to ICML 2025.
Please ensure that the additional experiments and clarifications provided during the rebuttal are included in the final version of the paper.
Decision: Accept (poster)