Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks using the Marginal Likelihood
Authors: Rayen Dhahri
Alexander Immer
Bertrand Charpentier
Stephan Günnemann
Vincent Fortuin
Venue: NeurIPS 2024 poster (2024)
Keywords: Bayesian deep learning
Bayesian model selection
Marginal likelihood
Laplace approximation
Sparsification
Pruning
Primary area: probabilistic_methods
TL;DR: We show that the Bayesian marginal likelihood can be used to make neural networks more sparsifiable.

Abstract: Neural network sparsification is a promising avenue to save computational time and memory costs, especially in an age where many successful AI models are becoming too large to naively deploy on consumer hardware. While much work has focused on different weight pruning criteria, the overall sparsifiability of the network, i.e., its capacity to be pruned without quality loss, has often been overlooked. We present Sparsifiability via the Marginal likelihood (SpaM), a sparsification framework that highlights the effectiveness of using the Bayesian marginal likelihood in conjunction with sparsity-inducing priors for making neural networks more sparsifiable. Our approach implements an automatic Occam's razor that selects the most sparsifiable model that still explains the data well, both for structured and unstructured sparsification. In addition, we demonstrate that the pre-computed posterior precision from the Laplace approximation can be re-used to define a cheap pruning criterion, which outperforms many existing (more expensive) approaches. We demonstrate the effectiveness of our framework, especially at high sparsity levels, across a range of different neural network architectures and datasets.

Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks using the Marginal Likelihood
Rayen Dhahri! Alexander Immer Betrand Charpentier', Stephan Günnemann" 12, and Vincent Fortuin
1,2,6
Department Computer Science, Technical University of Munich
Munich Center for Machine Learning
3 Department of Computer Science, ETH Zürich
MMax Planck Institute for Intelligent Systems, Tübingen
Pruna AI, Munich
Helmholtz AI, Munich
Abstract
Neural network sparsification is promising avenue computational time and memory costs, especially in an age where many successful AI models are becoming too large naïvely deploy on consumer hardware. While much work has focused on different weight pruning criteria, the overall sparsifiability of the network, i.e., its capacity to be pruned without quality loss, has often been over- looked. We present Sparsifiability via the Marginal likelihood (SpaM), pruning framework that highlights the effectiveness of using the Bayesian marginal likeli- hood in conjunction with sparsifiable. Our approach implements an automatic Occam's ruzor that selects the most sparsifiable model that still explains the data well, both for structured and unstructured sparsification. In addition, we demonstrate that pre-computed posterior precision from the Laplace approximation can be re-used to define cheap pruning criterion, which outperforms many existing (more expensive) approaches. We demonstrate the effectiveness of our framework, especially at high sparsity levels, across range of different neural network architectures and datasets.
priors for making neural networks more
Introduction
The availability of large datasets and powerful computing infrastructure has fueled the growth of deep learning, enabling the training of increasingly complex neural networks (NNs). While catalyzing performance gains across various domains, such as image recognition [1] and text generation [2], this development has amplified the challenge of werrparrameterization [3, 4] and raised concerns about the increase in model size and computational cost. Over paremmetamtttei neural networks present significant deployment challenges, particularly in sardware-constraile sparked the research field of NN sparsification or pruning, where the goal is to remove a (potentially large) number of parameters from trained network make smaller and ultimately cheaper to apply [7, 8]. However, most existing research in this domain has focused on the question of finding better pruning criteria, that is, scoring functions that decide which parameters prune away [9, 4, 10]. This ignores the challenge that many trained networks are not inherently sparsifiable, i.c., they resist
environments [5, 6]. This has
Correspondence to rayen.dhahriee tttm.de
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

KFACIA
Dagg
Spok Trmeiing Unsinddued Prucing Pruring
Compute ORD imporianee score
PLOPS
log pDD (4) loga( 6.MM)
Loul Pi.l
Aspragate artactuee
86)
(U0,
Pr
Figure 1: Overview of our proposed SpaM method. We start by training the network maximize the marginal likelihood using the Laplace approximation, while simplifying the Hessian computation through either the KFAC or a diagonal approximation. We can then use our precomputed posterior precision pruning criterion (OPD). For the of unstructured pruming, to achieve different target sparsities, compute the mask, and apply it, while for the structured approach, aggregate the score per layer for casier weight transfer, compute the mask, and then delete the masked structures to obtain a smaller model.
compute thresholds
effective pruning, regardless of the chosen criterion. Indeed, standard training of NNs does encourage sparsifiability at all, so should not be surprising that such trained NNs would use all of their parameters some degree fit the data.
Our work tackles this problem by modifying the training process itself, showing that more sparsifiable networks can be achieved through Bayesian model selection using the marginal likelihood [11, 12] in conjunction with an adequate prior that will induce such sparsifiability. We call this Sparsifiability via the Marginal likelihood, or SpaM. Our approach implements automatic Occam's 's razor [13], guiding the training process towards models that are faithfully fitting the data using only a small subset of their available parameters, such that the remaining ones be pruned afterwards. This is achieved by optimizing thousands of prior hyper-parameters to adaptively regularize weight magnitudes. We make of recent advances in Laplace inference for Bayesian neural networks (BNNs) [12, 14], allowing us to approximate the marginal likelihood [1 [] efficiently.
Once trained, can any pruning criterion more effectively sparsify these networks. Notably, pre-computed posterior precision of the Laplace approximation obtained from the marginal likelihood training readily translates into powerful pruning criterion, which we call Optimal Posterior Damage (OPD), similar to the popular Optimal Brain Damage [OBD; 3]. Since reuses existing computations, it is cheaper than many existing criteria in practice, and it often performs par or even better.
Extensive empirical evaluations demonstrate the strength of our SpaM approach and the derived OPD pruning criterion in both unstructured and structured sparsification tasks across various datasets and architectures. Moreover, they show that our framework strikes compelling balance between performance and computational cost.
We make the following contributions:
We propose SpaM, a novel approach to improve the sparsifiability of neural networks during training, using Bayesian model selection with the Laplace marginal likelihood, which works well structured and unstructured pruning scenarios and with different pruning criteria.
We provide evidence- based recommendations for prior selection within the SpaM framework, showing that some priors improve sparsifiability significantly better than others.

---

We present OPD, cheap pruning criterion similar to the popular OBD, which performs comparably or better than many other criteria practice and conveniently reuses computa- tions performed in the context of SpaM.
2 Background
We use deep neural networks to model learning tasks with inputs x,, E RD and targets yn E R collected in a dataset D = {(x,, Yn ">))>= of N pairs. A model is parameterized by weights R and maps from imputs targets using the neural network function fx)x.. Assuming the data are ii.dd, we have likelihood p(D = II p(yn (xx,)) We minimize the negative log likelihood, which corresponds common losses like the cross-entropy in classification. Additionally, regularization in the form of weight decay is commonly used and corresponds to a Gaussian prior on parameters p(0) (00 diag(o)) with diagonal precision. We employ Gaussian priors due their analytical tractability and seamless integration with the Laplace approximation, which requires differentiability. This is essential for maintaining computational efficiency in framework to approximate the marginal likelihood in a practical and scalable manner. Furthermore, Gaussian priors enable automatic relevance determination by allowing each parameter to have its own variance, facilitating the regularization process without introducing significant computational overhead.
2.1 Marginal Likelihood for Deep Learning
The marginal likelihood serves the probabilistic foundation for model evaluation and selection. provides an objective to optimize the tradeoff between data fit and model complexity, akin to the concept of Occam's 's razor [13, 15], by quantifying how well model M, with all its inherent uncertainties, explains the observed data:
MD)
do.
However, it requires computing an intractable integral over all neural network parameters.
The Laplace approximation [LA, 16] provides tractable and effective approximation to the
marginal likelihood for deep learning [12]. It arises from a second-order Taylor approximation around an estimate of the mode, 0., resulting in
log (PIM) ~ (. log
2 Po_ (MM).
where Pe. is the posterior precision given by the Hessian of the negative log joint distribution, -33 log PDD 0 MlM), evaluated at 0,. Defining Hg. as the Hessian of the negative log likelihood
objective V log pD) the posterior precision decomposes as Po. Ho. + diag(6).
In practice, the Hessian of the negative log likelihood is often approximated by the positive semidefi- nite generalized Gauss-Newton [GGN, 17],
Hg ~ EM Vefe (x,) (
V log ( (x,,,))) V (x,)),
(3)
which relies on the Jacobians of the neural network function and second derivative of the negative log likelihood at the output. Further, it is amenable to efficient structured approximations like diagonal or layer-wise variants [e.g., 18, 19].
Diagonal and block-diagonal GGN approximations are efficient, and therefore commonly used for Laplace approximations in deep learning [20, 14]. The diagonal LA is cheap in terms of storage and computation by only modeling the marginal variances of parameters. Kronecker- LA, 20) instead relies on a block-diagonal approximation to the GGN of the parameters O; in the /th layer,
LA [KFAC
(4) where the factors are given by the outer products of re-aacivvaioons and Jacobians w.r.t. the output of a layer, respectively [18, 19J. Here, A; and G; are the uncentered covariances of the respective layer inputs and output gradients. The top left of Figure shows comparison of both structures.
Ho, x X G
2.2 Neural Network Pruning
The goal of the pruning procedure is to remove parameters from 0 without affecting the quality of the model output fx). While unstructured pruning consists in zeroing individual entries 6, of the
3

---

weight matrices, structured pruning consists in deleting entire structured sets of parameters g. like rows or columns [21. 22]. The results of structured pruning enable smaller matrix multiplications that directly provide real-world efficiency gains on most hardware, including GPUs.
Pruning procedures usually follow three steps: (1) We use scoring function S(.) evaluate the importance of each individual parameter S(O,) for unstructured pruning, or of structured set of parameters S(g) for structured pruning. (2) We compute binary mask m with the same dimensions as 0, which assigns O values to parameters whose unstructured or structured pruning scores are below threshold T, and otherwise. While the threshold T is determined based on the target sparsity across layers for global pruning, it is determined per layer for uniform pruning [21]. (3) We apply the mask on the weight matrix with element- wise multiplication, m to effectively remove the least important parameters. Alternatively, structured pruning enables us to directly remove rows or columns whose mask values are to reduce weight matrix dimensions.
3 Shaving Weights with Occam's Razor
We identify sparsifiable neural networks by automatically regularizing (groups of) parameters to have small magnitudes, facilitate pruning the least important ones, within a probabilistic framework. Specifically, we utilize priors that regularize parameters in potentially structured ways, leading to smaller magnitudes. To optimize the resulting prior hyperparameters, we employ the Bayesian marginal likelihood as differentiable objective function, effectively implementing Bayesian variant of Occam's 's razor that drives irrelevant parameters towards smaller absolute magnitudes. The regularized networks can then be pruned with any method. However, we additionally propose to reuse the computed posterior precision for sparsification as a cheap and effective criterion.
3.1 Structured Priors for Regularization
To reduce the magnitude of parameters and make them more amenable to pruning, we introduce structured priors and show how to combine them with diagonal and KFAC Laplace approximations. While a scalar prior, corresponding to weight decay, is the most common, it suggests that all parameters in neural network are equally relevant and favors a uniform magnitude of parameters, which is suboptimal for pruning [23, Sec. 3.6].
Instead of scalar priors, regularize parameters with different strengths using layer,, unit-, and parameter-wise priors. Layer-wise priors regularize individual layers differently and have been shown to aid pruning and improve generalization [12, 24, 25, 26). Unit-wise regularization has been used mostly in traditional statistics, for example, for group sparsity [27], but recently also for channels or feature dimensions in neural networks [28, 29].
We consider different priors in the context of the Laplace approximation for marginal likelihood optimization and pruning: Scalar priors correspond to standard weight decay and are identical for all weights. Layer-wise priors provide a scalar regularizer ón per layer that is stacked into a vector 6 in line with the number of parameters per layer. Parameter-wis wise priors allow to specify 8, for each parameter individually. We define unit-wise priors so that each unit, which denotes a channel for convolutional and a hidden neuron for fonneccon layers, has regularization strength for incoming and outgoing weights separately. Thus, a weight 6p that connects unit at layer I-1 with unit j in layer I has prior ((0, [ōa]: [dr];), that is, each layer with Mi hidden units has prior vector A weight is thus regularized more strongly whenever both its in- and output neurons are.
6, E R
Our different priors are simple to combine additively with a diagonal Hessian approximation for the Laplace approximation (Equation (2)) but not with KFAC structure. For that reason, so far, only scalar or layer-wise priors have been used for KFAC posterior approximations [14]. The main issue is that we have to preserve the Kronecker factors to keep the resulting memory cost low. For scalar or layer-wise priors, this can be achieved by an eigendecomposition of the individual factors
A & G + 18 QAA @ QcAoQ + 15 = (Qa @ (A @ AG + [6) @ Q%). (5)
which means that the precision only needs to be added to the diagonal eigenvalues and no Kronecker product needs be calculated for inversion or determinant calculation.
To add diagonal prior precision ō, to the KFAC of the Ith layer, we derive an optimal approximation in the KFAC eigenbasis, so to maintain the frattceer-factored structure of the posterior:

---

Proposition 3.1 (Diagonal Prior in KFAC Eigenbasis), Considering the Frobenius norm, the optimal diagonal perturbation of the KFAC eigenvalues A A @ AB to add a diagonal prior precision given by A & Ag B + 8 with mat(o) = (o)) mat Q' where the square is element-wis and mat(.) reshapes the vector to match the parameter shape used in KFAC. Thus, it can be computed efficiently without computing a Kronecker product.
We provide the proof in Appendix A. The approach is similar to that of George et al. [30], who correct KFAC's eigenvalues towards the diagonal Gauss-Newton, but solves the problem of adding a full-rank diagonal instead of rank-1 outer product the KFAC eigenbasis.
3.2 Learning Regularization with the Marginal Likelihood
To optimize the potentially millions of regularization parameters, for example, arising from a parameter-wise prior, we employ the marginal likelihood as a differentiable objective. Optimizing regularization parameters has the advantage that different (groups of) parameters will be regularized differently and, therefore, become easier to prune. While it would be intractable to optimize that many regularization parameters using validation-based forms of optimization, the marginal likelihood can be estimated and differentiated during training [12, 31, 32].
Automatically determining the relevance of trrameteer-gouppe (ARD) is a common approach in Bayesian learning that can lead to sparsity and smaller parameter magnitudes [33, 34] and has been used especially in linear models. The marginal likelihood provides an objective that automatically regularizes irrelevant parameter groups more to lower their magnitude. Therefore, it implements a Bayesian variant of Occam's razor, finding the simplest model that explains the data well [13].
Mathematically, all the prior parameters ō constitute the hyperparameters of the model M in the log marginal likelihood (Equation (1)) that we optimize interleaved with the neural network parameters. When optimizing the prior parameters, we use gradient ascent
8c+1 8 + V p(D) 8=)6
(6)
or adaptive optimizers like Adam [35]. We follow Immer et al. [12] and optimize the Laplace approximation to the marginal likelihood after an initial burn-in phase with a certain frequency. We describe the optimization process and the related hyperparameters in Appendix D.4
3.3 Optimal Posterior Damage (OPD)
While sparsity regularization learned by marginal likelihood training can be advantageously combined with any pruning criterion, like Single-shot Network Pruning [SNIP: 36], variants of Gradient Signal Preservation [GraSP: 37, 38, 39], or magnitude pruning [7], we further propose a new pruning criterion that uses our Laplace approximation and extends the unstructured Optimal Brain Damage (OBD) pruning criterion [3]. While OBD traditionally uses the Hessian (approximation) H, of the loss, we propose to adapt it to use the posterior precision Pe, which additionally includes the prior
precision 6. The importance score S(0,,) for parameter fp becomes
S(0,) = Ppp x 03 P
(7)
where Ppp denotes the posterior precision for the parameter 8p, extracted from the diagonal of the posterior precision matrix Pa. We call this novel posterior- posterior-based pruning criterion Optimal Posterior Damage (OPD). Intuitively, individual weights with high scores indicate certainty of the posterior distribution and a significant contribution to the model's functionality, as indicated by the magnitude.
We also propose structured version of OPD by aggregating the score over a set of parameters g, i.e.,
S(g)
Σ s(G,))
Σ Ppp 62
(8)
PEg
PEg
In practice, the structured set of parameters g corresponds to all parameters along one dimension of the weight matrix inside layer, in order to reduce the size of the matrix multiplications. Since subsequent layers might have significantly different weight matrix dimensions impacting the magnitude of the aggregated sum, we opt for uniform structured pruning guarantee fair pruning treatment across all layers. This means we prune each layer by the same target percentage, reducing the dimensions of each layer by the same proportion relative to the unpruned model. This approach contrasts with
5

---

Table 1: Accuracies of pruned ResNets CIFAR-10. The best training method for each pruning criterion is highlighted in green, where we see that SpaM improves performance for all criteria except the random baseline. The best performances overall at each sparsity level are shown in bold, showing that our cheap OPD criterion outperforms the others at high sparsities.
Criterion
Training
Sparsity (%)
80
85
90
95
99
MAP
88.06 (#0.12)
82.32 (#0.44)
64.08 (#1.32)
37.52 (#2.34)
17.32 (+1.01)
OPD
SpaM
90.78 (#0.66)
90.78 (#0.65)
90.68 (#0.65)
89.98 (#0.61)
66.28 (+5.89)
MAP
82.87 (#0.48)
68.78 (+1.88)
48.65 (+2.69)
26.46 (=1.66)
15.75 (+0.80)
GraSP
SpaM
91.50 (#0.66)
90.94 (+0.65)
89.42 (+0.71)
82.18 (+2.65)
41.48 (+7.95)
MAP
53.96 (+2.72)
37.74 (#2.21)
26.74 (#3.17)
13.88 (=0.87)
12.58 (+0.36)
SNIP
SpaM
67.40 (+5.68)
52.62 (#6.84)
33.75 (+5.71)
17.06 (+2.23)
11,90 (+0.51)
MAP
88.17 (#0.12)
81.92 (#0.37)
61.60 (+1.1)
32.88 (=1.52)
16.12 (+0.90)
Magnitude
SpaM
91.55 (#0.64)
90.92 (#0.64)
89.23 (+0.62)
81.80 (#2.22)
41.78 (+7.20)
MAP
11.25 (#0.48)
12.15 (#0.92)
11.65 (#0.62)
10.45 (#0.17)
10.27 (+0.17) 17)
Random
SpaM
11.00 (#0.48)
10.47 (+0.86)
10.56 (#1.15)
10.01 (=0.45)
9.81 (#0.61)
achieving a global target sparsity that varies across layers, which can make more difficult to consistently compress and adjust the model's size.
Moreover, removing a full structure is more aggressive, we also apply gradual pruning during training. Finally, we omit pruning the final layer to mitigate overly strong impact on classification accuracy and computational stability [40].
When using our SpaM approach, the precomputed precision matrix from the Laplace approximation can be reused to compute OPD without computational overhead, in contrast to the other pruning criteria, which often require additional computations to be performed. Note that we will also show in our experiments that OPD additionally avoids the need for potentially expensive fine-tuning after pruning. Moreover, even in the case of maximum a posteriori (MAP) training, Laplace approximations of the inverse Hessian at O. can be additionally computed to approximate OPD. Finally, the OPD criterion can not only be computed post-hoc after training, but even online during training.
4 Related work
Laplae--aproximmaee BNNs. From the early inception of Bayesian neural networks [41, 42], the Laplace approximation was a popular inference method [16]. In recent years, it has undergone a renaissance [18, 19, 20, 14], including critical work on using more scalable approximations for the associated marginal likelihood in the context of model selection [11, 12, 43], which we use in our framework. To the best of our knowledge, we are the first to study the benefits of this Laplace- approximated marginal likelihood in the context of sparsification of deep neural networks. However, similar methods that automatically quantify the relevance (ARD) of parameters have been derived and used for linear, bilinear, and kernel models [34, 44, 45] as alternative to the Lasso. More recently, van der Ouderaa et al. [46] and Bouchiat et al. [47] used the ARD mechanism in deep learning to select layers and features by regularization, respectively.
Pruning neural networks. Various pruning criteria have been proposed determine the importance of model parameters. Many criteria prune based on the weight magnitude [7, 48, 49] but usually required additional fine-tuning to recover accuracy. Sun al. [8] proposed to combine activation and weight norms for pruning without fine-tuning. Other approaches include pruning using first- order information based on connectivity [36] or synaptic flow conservation [50], or second-order information aiming at preserving gradient flow [37, 38, 39]. Recently, van der Ouderaa et al. [51] focused on pruning LLMs based on second-order Taylor expansion. contrast, OPD uses second- order information provided by the posterior precision given by the Laplace approximation. Beyond pruning criteria, there have been many approaches to prune initialization [36, 37, 50), during training [52, 53], and after training [7, 8). In particular, multiple works proposed to leverage specific

---

Test Accuracy (%) 1 Accuracy (%)) 1
FCN (Breast Cancer)
LeNet (FashionMNIST)
ResNet (CIFAR-IO-Augment
76
75
50
50
25
25
100
10
60
40
60
100
WRN-16-4 (CIFAR- 100)
ViT-Scratch (MNIST)
DistilBERT (IMDb)
100
50
50
60
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Sparsity (%)
Sparsity (%)
Sparsity (%)
MAP-OPD
MAP-GraSP
MAP-SNIP
MAP-Magniuude
MAP-Random
SpaM-OPD
SpaM-GraSP
SpaM-SNIP
SpaM-Magnitude
SpaM-Random
Figure 2: Predictive performance as a function of sparsity level in unstructured pruning. We see that SpaM improves the performance over MAP training across most architectures, datasets, and pruning criteria, and that OPD often outperforms the other pruning criteria. Both of these effects are particularly visible at higher sparsity levels. The black star each subfigure denotes the performance of the unprumed models, which is often identical to the performance of models pruned at 20% sparsity.
training schemes promoting zero-invariant parameter groups for structured pruning [54, 55]. In contrast, SpaM induces sparsifiability during training, and is agnostic about the criterion.
Experiments
We conduct experiments various datasets and models and outline experimental setup in detail in Appendix D. We compare MAP training with our proposed SpaM approach with different priors, comparing
OPD pruning criterion with random pruning. magnitude pruning, SNIP [36], GraSP [37, 38), and SynFlow [50]. We show that SpaM improves pruning performance with different pruning criteria, especially at higher sparsities, and that OPD criterion often outperforms the other criteria. This observation extends not only to predictive accuracy, but also uncertainty estimation. Moreover, show that the choice of prior can play a significant role, and we introduce parameter-wise and unit-wise priors for the KFAC approximation. Finally, show that SpaM and OPD also work in structured pruning setting, leading significant computational benefits. The code for our methods and experiments can be found at https: wtt.//uw...oooooouunllaaaaaaaaaa wtt.:uuuinnaaaepaammmmmiiiigg
5.1 SpaM Improves Performance at High Sparsities
We compare SpaM to MAP training with different pruning criteria, including OPD, across different models on tabular, vision, and language datasets. For SpaM in this unstructured pruning context, we use the diagonal Laplace approximation with parameter-wise prior. Encouragingly, MAP and SpaM reach comparable performance during training, showing that the increased sparsifiability of SpaM comes at no additional cost unpruned performance (sec Figure B1 the appendix).
We see in Table and Figure 2 that SpaM drastically improves the performance for many pruning criteria, especially magnitude pruning, GraSP, and OPD. We also see that OPD, despite being cheap byproduct of our marginal likelihood computation, often outperforms the other pruning criteria, especially at higher sparsities. For instance, at 95 % pruning rate (i.e., with 20x fewer parameters), our combination of SpaM and OPD still retains almost the same performance the unpruned model on vision tasks, while the other pruning criteria with MAP training have dropped to essentially unusable performance levels at this sparsity.

---

0.100
Brierr
10C
0.0
100
100
Sparsity
Sparsity
Sparsity (96)
Spol SNIP
SpoM OPD Post Boo
SpaM Randon
SyaMt SyeMi Megritde
MAP-5KIP
MAPGGWSP
MAP-OFD--hee-ltee
MAP Roncom
MAP-Maggshhle
Figure 3: Uncertainty estimation with pruned ResNets on CIFAR-1O. We see that SpaM improves
uncertainty estimation in terms of NLL, ECE, and Brier score for many pruning criteria and that our OPD criterion outperforms the other criteria, especially at high sparsities.
Test Accuracy (%) 1 Test Accuracy (%6 1
FCN (Breast Cancer)
FCN (MNIST)
LeNet (MNIST)
90
80
60
60
60
40
45
20
40
60
LcNet (FashionMNISTT)
LeNet (CIFAR- 10)
ResNet (CIFAR-10)
80
60
80
60
60
30
40
20
20
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Sparsity (%)
Sparsity (%)
Sparsity (%)
A Approximation, Prior
Diug LA, wise
Diag LA, layer-wise wise
Diag LA, scalar
Diag LA, unit-wise
KFAC LA,
KFAC LA, layer-wise
KFAC LA, scalar
KFAC LA, uni--wiee
Figure 4: Comparison different priors and Hessian approximations for SpaM-OPD unstructured pruning. The unit-wise and parameter-wiss priors show better performance at high sparsity levels, with the parameter-we wise one bridging the gap between Diag and KFAC LA.
Fine-tuning. We in Figure B4 in the appendix that some of this performance difference be remedied by costly tie-ttning of the networks after pruning, which however still does not allow the other methods reach the full SpaM-OPD performance. Interestingly, in the case of OPD, this does not further improve its already near-optimal performance.
Online pruning. Figure B5 in the appendix shows that our online version of SpaM, which uses the marginal likelihood and OPD during training to iteratively prune the network, reaches comparable performance levels to the post-hoc version, thus offering a computationally even more convenient way to effectively sparsify neural networks.
Uncertainty estimation. Given that SpaM is Bayesian method, it does not only offer high predictive accuracies but also calibrated uncertainty estimates. Indeed, we see in Figure that the trends we have seen for accuracy also apply for negative log-likelihood, expected calibration error, and the Brier score. Again, SpaM improves the uncertainty estimates over MAP training, OPD outperforms most other criteria, and achieve well-calibrated models up until very high sparsity levels. Note that the random baseline also achieves low ECE at high sparsity levels because it essentially reverts to random guessing, which is known weakness of the ECE metric [56].

---

Test Accuracy (9) Test Accuracy (%) 1
FC-3Layers (Breast Cancer)
FC-2Layers (MNIST)
LeNet (MNIST)
100
100
75
50
50
20
80
100
20
40
60
100
40
60
80
100
LeNet (FashionMNISST)
LeNet (CIFAR-10) 10)
ResNet (CIFAR-IO-AUS) Augggmmeed))
50
75
50
25
25
20
100
20
40
60
100
20
60
80
100
Sparsity (%)
Sparsity (%)
Sparsity (%)
SpaM-SNIP
SpuM-GraSP
SpaM-OPD
SpuM-SynHlow
SpaM-Random
MAP-SNIP
MAP-GraSP
MAP-OPD
MAP-SynFlow
MAP-Random
Figure 5: Similarly to unstructured pruning, in this experiment on structured pruning that SpaM (using unit-wise prior) improves performance over MAP and that OPD mostly outperforms other pruning criteria, especially at higher sparsity levels. The black stars reflect the performance of the unprumed models.
80
90
Test accuracy (%)
40
40
6.2c+04
10a
FLOPS
Size (KB)
SpaM-GGraSt
SpaM-SNIP
SpaM- Rand
SpaM SynFlow
MAP-GuSP
MAP-SNIP
... MAP-Raad
Mya-Sw
SyuM-OPD
Figure 6: Structured pruning with LeNet on FashionMNIST, using unit-wise priors. We see that our SpaM-OPD dominates the Pareto frontier, in terms of predictive performance function of computational time memory cost, and particularly competitive at lower costs.
5.2 Inffuence of Priors Sparsifiability
To understand the influence of the prior and Hessian approximation on performance in our proposed SpaM-OPD approach, we compare diagonal and KFAC approximations with scalar, layer-wise, unit-wise, and parameter- wise priors. Note regarding the latter two, that in this work, first implement them for the KFAC approximation, thus contributing to the general framework of Laplace-approximated BNNs [14]. independent of the pruning use case.
the
We see Figure that our newly introduced unit-wise and parameter- wise priors for KFAC indeed outperform the others, especially at high sparsities. When comparing KFAC to the diagonal ap- proximation, we see that KFAC often leads to slightly better performance at lower sparsity levels. However, we also see that the relatively simple choice of parameter-we wisee prior and diagonal Hessian approximation, as used in our previous experiments above, strong baseline across the board and can be recommended safe default option for unstructured pruning. Note that the unit-wise priors

---

can be especially useful for structured pruning, as we will see in the following experiment. More detailed prior comparisons can be found in Appendix B.3.
5.3 SpaM Extends to Structured Sparsification
Here, we study the effect of SpaM and OPD in the more challenging setting of eliminating entire network structures, such as convolutional kernels. Studying different network architectures, we aim generalize unstructured pruning approach the setting of structured pruning, where the structures can be freely defined depending on the use case.
Encouragingly, we see in Figure that our findings from the unstructured case transfer qualitatively also to the structured case, with SpaM-OPP -PD outperforming the baselines high sparsities. Crucially, while the sparsity patterns generated by unstructured pruning are more difficult to translate into computational benefits, structured pruning directly leads to computational savings on standard GPUs (see also Figure B11 in the appendix). We see in Figure 6 that SpaM-OPD dominates the Pareto frontier of the tradeoff between performance and computational cost at high sparsities (i.e., low costs), yielding 10x-20x savings in FLOPS and memory consumption with only minimal deterioration in performance. This positions our proposed framework as a potentially promising approach for the deployment of AI models in resource-constraine environments.
6 Conclusion
We have shown that the Bayesian marginal likelihood, with its associated Occam's razor effect, can be used during training to select neural network models that are inherently more sparsifiable. Crucially, we have shown that this sparsifiability extends across different pruning criteria and enables large gains in performance and uncertainty estimation, especially at high sparsity levels. Conveniently, the computations needed for the marginal likelihood estimation using the Laplace approximation can be re-used to define a novel pruning criterion called OPD, which outperforms many existing (more expensive) criteria in experiments. We have also presented guidelines for choosing priors within our framework and have shown that even in the challenging setting of structured pruning, our proposed SpaM approach can yield up 20x savings in computational time and memory, with only small reductions in performance. Our work thus offers promising path towards pruning large Al models at high sparsity levels for deployment cesotrce-contrrained devices.
Limitations. Our approach naturally inherits some limitations of the Laplace approximation, for instance, the fact that it only captures the local geometry of single posterior mode or potential numerical instabilities in the Hessian computations when used with low-precision weights. Moreover, accrues an additional computational cost compared to MAP training, which is then, however, amortized by the computational savings during the deployment of the sparsified model.
Acknowledgments
AI acknowledges funding through a Max Planck ETH Center for Learning Systems (CLS) fellowship. VF was supported by Branco Weiss Fellowship.
References
[1] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge, 2015.
[2] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust Speech Recognition via Large-Scale Weak Supervision, 2022.
[3] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pages 598-605, 1990.
[4] Jonathan Frankle and Michael Carbin. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. In International Conference on Learning Representations (ICLR), 2019.
10

---

[5] Partha Pratim Ray. A review TinyML; State-of-in ahe and Prospects. Journal of King Saud University Computer and Information Sciences, 34(4):1595-1623) 2022. ISSN 1319-1578.
[6] Andrei Paleyes, Ghoul-Gabriel Urma, and Neil Lawrence. Challenges in Deploying Machine Learning: A Survey of Case Studies. ACM Computing Surveys, 55(6):1-29 December 2022.
ISSN 1557-7341.
[7] Song Han, Huizi Mao, and William J Dally. Deep Compression: Compressing Deep Neu- ral Networks with Pruning, Trained Quantization and Huffman Coding. In Proceedings of international conference learning representations (ICLR), 2016.
[8] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A Simple and Effective Pruning approach for Large Language Models. arXiv preprint arXiv:2306. 1695, 2023.
[9] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv: 16/11116... 1.06440, 2016.
[10] Lucas Liebenwein, Cenk Baykal, Brandon Carter, David Gifford, and Daniela Rus. Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy, 2021.
[11] David John Cameron MacKay. Probable networks and plausible predictions a review of practical Bayesian methods for supervised neural networks. Network: Computation Neural Systems, 505, 1995.
[12] Alexander Immer, Matthias Bauer, Vincent Fortuin, Gunnar Rätsch, and Mohammad Emtiyaz Khan. Scalable marginal likelihood estimation for model selection deep learning. In International Conference on Machine Learning. PMLR, 2021.
[13] Carl Rasmussen and Zoubin Ghahramani. Occam's Razor. In T. Leen, T. Dietterich, and Tresp, editors, Advances Neural Information Processing Systems, volume 13. MIT Press, 2000.
[14] Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace Redux-Effortless Bayesian Deep Learning. In NeurIPS, 2021.
[15] David C. MacKay. Information Theory, Inference Learning Algorithms. Cambridge University Press, USA, 2002. ISBN 0521642981
[16] David J. C. MacKay. A Practical Bayesian Framework for Backpropagation Networks. Neural Computation, 4(3):488444) 472, 05 1992. ISSN 0899-7667.
[17] Nicol N Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. Neural compulation, 143)):123-1 2002.
[18] James Martens and Roger Grosse. Optimizing neural networks with sonecker-factored approx- imate curvature. In International conference on machine learning, pages 2408-244 2417. PMLR, 2015.
[19] Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical Gauss-Newton Optimisation for Deep Learning, 2017.
[20] Hippolyt Ritter, Aleksandar Botev, and David Barber. A Scalable Laplace Approximation for Neural Networks. In International Conference on Learning Representations, 2018.
[21] Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. Pruning and
quantization for deep neural network acceleration: survey. Neurocomput, 461(C):770-40) oct 2021. ISSN 0925-2312.
[22] Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16091-16101, 2023.
[23] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. The Journal of Machine Learning Research, 2281):0882111005 2021.

---

[24] Alireza Aghasi, Afshin Abdi, Nam Nguyen, and Justin Romberg, Net-trim: Convex pruning of deep neural networks with performance guarantee. Advances neural information processing systems, 30, 2017.
[25] Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi. Morphnet: Fast & simple resource- constraied structure learning of deep networks. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 1586-1595, 2018.
[26] Javier Antorán, David Janz, James U Allingham, Erik Daxberger, Riccardo Rb Barbano, Eric Nalisnick, and José Miguel Gernández-Lobatee Adapting the Linearised Laplace Model Evidence for Modern Deep Learning. In International Conference on Machine Learning, pages 796-821. PMIR, 2022.
[27] Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(1):49 67, 2006.
[28] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in neural information processing systems, 29, 2016.
[29] Simone Scardapane, Danilo Comminiello, Amir Hussain, and Aurelio Uncini. Group sparse regularization for deep neural networks. Neurocomputing, 241:81-89, 2017.
[30] Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in kronecker factored eigenbasis. Advances in Neural Information Processing Systems, 31, 2018
[31] Alexander Immer, Tycho FA Van Der Ouderaa, Mark Van Der Wilk, Gunnar Ratsch, and Bernhard Schölkopf. Stochastic marginal likelihood gradients using neural tangent kernels. In International Conference on Machine Learning, pages 14333-14352. PMLR, 2023.
[32] Jihao Andreas Lin, Javier Antorán, and José Miguel Hernández-Lobato Online Laplace model selection revisited. arXiv preprint arXiv:2307.06099 2023.
[33] David JC MacKay et al. Bayesian nonlinear modeling for the prediction competition. ASHRAE Iransactions, 100(2): 1053-106, 1994.
[34] Michael E Tipping. Sparse Bayesian learning and the relevance vector machine. Journal of machine learning research, 1(Jun):111244 2001.
[35] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14X1v:1412.66800, 2014.
[36] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. SNIP: Single-shot Network Pruning based on Connection Sensitivity. CoRR, abs/1810.02340, 2018
[37] Chaoqi Wang, Guodong Zhang, and Roger Baker Grosse. Picking Winning Tickets Before Training by Preserving Gradient Flow. CoRR, abs/2002.07376, 2020.
[38] Ekdeep Singh Lubana and Robert P. Dick. Gradient Flow Framework For Analyzing Network Pruning, 2021.
[39] John Rachwan, Daniel Zügner, Bertrand Charpentier, Simon Geisler, Morgane Ayle, and Stephan Günnemann. Winning the Lottery Ahead of Time: Efficient Early Network Pruning, 2022.
[40] Erich Elsen, Marat Dukhan, Trevor Gale, and Karen Simonyan. Fast Sparse ConvNets, 2019.
[41] Radford Neal. Bayesian learning via stochastic dynamics. Advances in neural information processing systems, 5, 1992.
[42] Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pages 5-13, 1993.
12

---

[43] Alexander Immer. Tycho van der Ouderaa, Gunnar Rätsch, Vincent Fortuin, and Mark van der Wilk. Invariance learning in deep neural networks with differentiable Laplace approximations. Advances in Neural Information Processing Systems, 15:12449-12663 2022.
[44] David P Wipf and Bhaskar D Rao. Sparse Bayesian learning for basis selection. IEEE Transactions on Signal processing, 52(8):2153-2164, 2004.
[45] Alexander Immer, Stefan G Stark, Francis Jacob, Ximena Bonilla, Tinu Thomas, Andre Kahles, Sandra Goetze, Emanuela Milani, and Bernd Wollscheid. Probabilistic pathway-based multimodal factor analysis. Bioinformatics, 40, 2024.
[46] Tycho FA van der Ouderaa, Alexander Immer, and Mark van der Wilk. Learning Layer- wise Equivariances Automatically using Gradients. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[47] Kouroche Bouchiat, Alexander Immer, Hugo Yèche, Gunnar Ratsch, and Vincent Fortuin. Improving Neural Additive Models with Bayesian Principles. In Forty-first International Conference on Machine Learning, 2024.
[48] Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask, 2020.
[49] Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep Rewiring: Training very sparse deep networks, 2018.
[50] Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow, 2020.
[51] Tycho A. van der Ouderaa, Markus Nagel, Mart van Baalen, Yuki M. Asano, and Tijmen Blankevoort. The LLM Surgeon, 2023.
[52] Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning. arXiv preprint arXiv: 044776 2019.
[53] Xiao Zhou, Weizhong Zhang, Zonghao Chen, Shizhe Diao, and Tong Zhang. Efficient Neural Network Training via Forward and Backward Propagation Sparsification, 2021.
[54] Tianyi Chen, Bo Ji, DING Tianyu, Biyi Fang, Guanyi Wang, Zhihui Zhu, Luming Liang, Yixin Shi, Sheng Yi, and Xiao Tu. Only Train Once: A One-Shot Neural Network Training And Pruning Framework. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
[55] Tianyi Chen, Luming Liang, DING Tianyu, Zhihui Zhu, and Ilya Zharkov. Otov2: Automatic, generic, user-friendl.. In International Conference on Learning Representations, 2023.
[56] Sebastian Gruber and Florian Buettner. Better uncertainty calibration via proper scores for classification and beyond. Advances in Neural Information Processing Systems, 35:8618-8632. 2022.
[57] D. Dua and C. Graff. Breast cancer wisconsin (diagnostic) data set. UCI Machine Learning Repository, 2019.
[58] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applicd to document recognition. Proceedings of the IEEE, 86(1 1):2278-222 2324, 1998.
[59] Han Xiao, Kashif Rasul, and Roland Vollgraf. mashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint ftf.::700.07744/ 2017.
[60] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[61] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning Word Vectors for Sentiment Analysis. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea, editors, Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 201 1. Association for Computational Linguistics.
13

---

[62] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucie, and Alexey Dosovitskiy. MIPP-Mxer: An all-MLP Architecture for Vision, 2021.
[63] Kazuki Osawa, Satoki Ishikawa, Rio Yokota, Shigang Li, and Torsten Hoefter. ASDL; A Unified Interface for Gradient Preconditioning in PyTorch, 2023.
[64]
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks, 2017.
[65] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, 2021.
[66] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[67] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, 2020.
[68] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ily Sutskever. Language Models are Unsupervised Multitask Learners. 2019.
[69] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NeurlPS, 2015.
[70] Hesham Mostafa and Xin Wang. Parameter Efficient Training of Deep convolutional Neural Networks by Dynamic Sparse Reparameterization, 2019.
[71] Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of Bayesian neural nets via local lincarization. In International conference on artificial intelligence and statistics, pages 703-711. PMLR, 2021.
14

---

A Proof for Diagonal Prior in a Kronecker-factrred Eigenbasis
Proposition A.1 (Diagonal Prior in KFAC Eigenbasis). Considering the Frobenius norm, the optimal diagonal perturbation of the KFAC eigenvalues A & AB to add a diagonal prior precision is given by A & AB + å with mat(ā) = (9)) maa))m(())))))) (6) Q' where the square is element-wid and mal(.) reshapes the vector to match the parameter shape used KFAC. Thus, it can be computed efficiently without computing a Kronecker product.
Proof. We prove this result in two steps. First, we show what the optimum looks like in terms of the Frobenius norm. Second, show how simplify the results to enable efficient computation without computing Kronecker products. We have KFAC Hessian approximation A @ B with A E Da Dh and B I x Ban where the dimensionalities D. depend on the layer type [18]. In the case of cully-coneected layer, these are simply the dimensionality of the in- and output hidden representation. The same layer will have Din D parameters and thus the corresponding diagonal prior precision is given by 5 E R..Daa For the Laplace approximation, the eigendeconposition of individual Kronecker factors is already computed as A Q Q/4 and similarly for G as shown in Equation (5). Recall also that diag(-) turns a vector into diagonal matrix and extracts the diagonal entries of a matrix into a vector. We are interested in the Frobenius-optimal diagonal perturbation of the eigenvalues so to maintain the efficiency structure of the KFAC, and, thus, the downstream Laplace approximation:
arg min (Q @ (A)(AA @ AG + diag (6) & Q2)
- -QQa @ (A @ AG) (Q) @ Q%) + diag (
arg min |AA AG + diag (8) Aa AG + (Qa Q6) diag (o) & Qc)
diag ((Q)= & diag (Qx A @ Qc)),
where we first multiplied the orthogonal bases from left and right and then realized that the values of need to be set to the entries of the prior ā projected into the basis.
Naively, computing the optimum of å would require expanding the Kronecker product above and lead to potentially intractable complexity of DDD However, it is possible simplify it further to maintain efficient computation: For simplicity, consider the case without Kronecker factorization. We have
diag (Q' diag (d)Q) = (Q' o Qd)d,
where is the element-wing Hadamard product. So we can express the diagonal of the matrix-matrix product matrix-vector product with the diagonal d the vector. In the Kronecker-factoned case, we need just one more simplification:
diag((Q)s @ Q%) diag (Q) @ Qa)) ((Q) & Q (Q4 @ Q%))
((Q), Q & (Q) o Q6))
vec((Q o % mat QQ o Qa))
( mal
where we have used the mixed-product property of the Kronecker product and the propertics for multiplying Kronecker-produce with vector. The operator "flattens" matrix, that is, turns Dow x Di matrix into Dam Dan vector, and mat does the opposite. The final approximation å can be computed efficiently in ( + D2
B Additional Results
B.1
Baseline training
Figure B1 illustrates that both MAP and SPAM achieve similar levels of performance throughout the training process. This observation underscores that SPAM's enhanced sparsifiability is achieved without compromising the unpruned performance. Furthermore, the comparable unpruned accuracies
15

---

Test Accuracy (%)
PCN (Breast Cancer)
FCN (MNIST)
LcNet (MNIST)
1.0
0.8
0.98
0.96
0.96
40
50
100
Test Accuracy (%)
LeNet (FashionMNIST)
LeNet (CIFAR- 10)
ResNet (CIFAR 10)
0.9
0.8
0.6
0.8
0.4
50
100
100
50
100
Epoch
Epnch
Epoch
Approximation, Prior
Diag LA, param-wis
Ding LAA, layer-wise
Diag KFAC LA, scalar
Ding LA,
KFAC LA. param-wise
KFAC LA. layee-ise
KFAC LA, unit-wise
Figure Bl: Training curves for MAP and SpaM training with different priors and Hessian approxima- tions. We that all methods achieve a similar performance by the end of training.
of SPAM and MAP models indicate that SPAM's sparsifiability benefits higher baseline accuracies, but rather distinct ad advantage offered by the SPAM methodology. The sparsification methods are performed on these models way that once the model trained for specific seed, we copy it and use it to perform the different sparsification methods; we repeat the steps for a minimum of 4 different seeds, ensuring the robustness of our findings. In addition, the model trained with SpaM only uses a single forward pass over the pruned architecture during inference, thus guaranteeing a fair comparison with the baselines. The posterior is solely used estimate the marginal likelihood during SpaM training.
merely a result of
B.2 Tables
In tables B1 and B2, present results comparing different methods using MAP and SpaM with various priors. Notably, SpaM with Diag LA and parameter- priors significantly outperforms MAP and other SpaM variants high sparsity levels.
B.3
Prior effects
Figure B2 and Figure B3 illustrate our findings when applying SpaM with various priors for both OPD and GraSP. Notably, Diag LA, using parameter-wise priors, excels in high-sparsity scenarios, even with complex models and datasets like ResNets. Furthermore, for MI.Pmixer, we observe that SpaM variants, employing parameter-wise priors and layerwise approaches, preserve baseline accuracy even at extreme sparsities of 99%.
B.4 Unit-wise and Parameter-wise KFAC for GraSP
As shown in Section 5, networks trained using SpaM and parameter-wise priors were able maintain a high accuracy at challenging sparsity levels up to 99%. Moreover, parameter-wise KFAC and unit-wise priors showed high performance for the OPD pruning approach. We show in Figure B3 that the combination of SpaM and these priors with the GraSP criterion yield qualitatively similar performance rankings with OPD criterion.
B.5
One Shot Efficiency
As seen in Figure B4, our proposed post-hoc pruning criterion, OPD, consistently demonstrates stable performance across diverse model architectures and datasets, achieving significant sparsity levels without the need for fine-tuning. It seamlessly operates either post-training or with pre-trained models, providing highly flexible and versatile solution.
16

---

Table B1: Comparison of pruning accuracies of SpaM training with different pruning criteria, Hessian approximations, and priors for post-hoe pruning LeNet on MNIST.
Sparsity (5%
Criterion
Approximation
Prior
parameter-wise
98.75
98.75
98. 74
98.75
98.75
98.75
98.75
98.74
98.72
52.19
Diag
layerwise
99.08
99.09
99.10
98.96
98.50
97.79
93.50
78.93
63.71
16.67
GraSP
scalar
99.11
99.11
99.12
98.84
98.41
97.74
87.67
57.99
44.60
15.73
KFAC
layerwise
99.25
99.25
99.26
99.01
98.7
97.86
91.48 48
82.78
50.75
13.48
scalar
99.24
99.24
99.24
99.02
98.58
98.02
92.03
73.64
42.18
12.23
MAP
MAP
99.01
99.01
99.01
98.99
98.91
98.75
98.28
96.10
77.30
20.43
parameter-wise
98.73
98.73
98.68
97.95
95.70
83,90
57.55
20.47
13.00
9.10
Diag
layerwise
99.08
98.34
46.42
17.69
21 889
14.61
14.47
13.50
11.94
9.83
SNIP
scalar
99.11
99.11
98.56 55
91 .82
84. 70
62.25
34.56
10.75
16.28
16.70
KFAC
layerwise
99.25
99.20
79.59
27.70
43.76
23.85
10.28
16.19
19.90
10.27
scalar
99.24
99,24
98.55
94,82
70.25
48.28
27.02
26.88
25.95
9.86
MAP
MAP
99.01
99.01
98.95
98.31
97.21
94.26
87.19
42
25.15
12.56
parameter-wise
98.72
98.72
98.72
98.72
98.72
98.72
98.72
98.72
98.72
75.92
Diag
layerwise
99.08
99.09
99.08
99.07
99.04
98.98
98.84
98.40
94.71
36.25
OPD
scalar
99.10
99.11
99.06
98.28
95.32
74.78
16.12
KFAC
layerwise
99.25
99.25
99.24
99.23
99.18
99.11
98.95
98.60
95.90
28.16
scalar
99.24 24
99. 24
99. 24
99. 16
99. 10
98 90
97.97
95 88
76.11
27.38
MAP
MAP
99.01
99.01
99.03
98.99
98.95
98.90
98.71
98.17
92.82
27.19
parameter wise
98.72
98.72
98.72
98.70
98.69
98.67
98.65
98.55
98.03
38.61
Diag
layerwise
99.08
99.09
99.08
99.06
99.01
98.92
98.46
94.20
39.86
10.19
Magnitude
scalar
99.11
99.11
99.09
99.12
99.07
98.98
98.48
95.69
74.30
13.53
KFAC
layerwise
99.25
99.25
99,22
99.18
99.08
98.95
98.38
91.93
28.52
9.80
scalar
99.24
99.24
99.20
98.92
97.08
84.66
22.21
MAP
MAP
99.01
99.01
98.99
98.96
98.93
98.85
98.57
97. .6.
88.82
15.42
parameter-wiss
55.88
25.95
11.15
10,88
11.13
10.56
11.88
11.35
9.95
9.81
Diag
layerwise
78.86
17.47
22.35
14.23
12.06
11.80
10.18
12.82
Random
scalar
88.75
60.17
25.35
15.98
14.26
11.91
8.63
9.74
9.05
9.80
KFAC
layerwise
89.90 99.90
18.70
20.36
14.13
14.56
12.61
9.72
12,00
8.93
9.80
scalar
90.46
34.14
19.98
12.72
8.49
11.40
10.08
10.54
9.74
9.80
MAP
MAP
79.03
43.25
22.86
9.68
10.34
9.96
11,50
9.50
10.85
9.80
B.6
Online pruning.
Figure B5 shows that our online version of SpaM, which uses the marginal likelihood and OPD during training to iteratively prune the network, reaches comparable performance levels to the post-hoc version, thus offering computationally even more convenient way to effectively sparsify neural networks. In the online approach, we prune jointly during SpaM training to reach a target sparsity; we perform the sparsity updates (dynamic masking) based on the marginal likelihood optimization parameters we refer to as _1_ r__pochh___ burnin and marglik_frequene when we start the marginal likelihood optimization and marglik ffeeuuefffueeyyyy ee specifies after how many epochs we update the estimate. Pruning occurs only after new marginal likelihood calculation, and the sparsity percentage is adjusted incrementally to reach the target by the training's 's end. The curve of OPD-Online reflects training progress (x-axis explicitly epochs reached sparsity), which explains the curve of LeNet on CIFAR-10 10 that is still converging while pruning. At the same time, the one-shot post-training approach reflects converged models copied and pruned at different sparsity levels.
Here, n1 eeocchs burnin specifies
B.7 Comparing SpaM to I.1 regularization
We conducted experiments using LI regularization. which is well-known for inducing sparsity in neural networks. To optimize performance, we performed an extensive search for the appropriate L1 regularization coefficient and applied various strengths during training.
We found that SpaM consistently outperforms LI regularization, achieving much higher levels of sparsity while maintaining the network's 's predictive performance as shown in Figure B6. L1 regularization, while effective at inducing sparsity, often proved too aggressive, leading to networks that were excessively pruned, negatively affecting performance. In contrast, our method offers key
17

---

Table B2: Comparison of pruning accuracies of SpaM training with different pruning criteria, Hessian approximations, and priors for post-hoc pruning MLP-Mixce (2 blocks) on MNIST.
Sparsity (%)
20
40
60
70
75
80
85
90
95
99
Method
Approximation
Prior
parmmeter-wise
98,51
98.51
98.51
98,51
98,51
98.51
98.52
98.51
98.52
98,50
Diag
layerwise
97.71
97.71
97.71
97.71
97.71
97.71
97.71
97.71
97.71
90.35
scolar
97.91
97.91
97.91
97.91
97.91
97.91
97.91
97.91
97.91
92.54
GraSP
parameter-wise
98.10
98.10
98.10
98.10
98.10
98.10
98.10
98.10
98.11
95.45
KFAC
Inyerwise
98.16
98.16
98. 16
98. 16
98. 16
98. 16
98. 16
98. 15
97 92
57.14
scalar
98.29
98.29
98.29
98.29
98.29
98.29
98.29
98.29
64.88
MAP
MAP
98.41
98.41
98.38
98.38
98.33
98.35
98.17
97.38
89,43
38.89
parameter-wise
98,51
98.51
98.51
98.51
98,51
98.52
98 49
97.70
83.76
20.00
Diag
layerwise
97.71
97.71
97.71
97.71
97.71
97.71
12.41
scalar
97.91
97.91
97.91
97.91
97.91
97.91
97.91
97.91
97.91
75.84
SNIP
parameter-wise
98.10
98.10
10
98.10
98.10
10
84
KFAC
layerwise
98.16
98.16
98.16
98.16
98.15
98.15
98.14
98.10
92.04
28.16
scalar
98.29
98.29
98.29
98.29
98.25
98.29
98.29
98.29
97.89
54.69
MAP
MAP
98.38
98.36
98.34
98.22
98.02
96.04
92.22
81.42
35.36
parameter-wise
98.51
98.51
98.51
98.51
98.51
98.51
98.52
98.52
98.51
98.50
Diag
layerwise
97.71
97.71
97.71
97.71
97.71
97.71
97.71
97.71
97.71
96.06
scalar
97.91
97.91
97.91
97.91
97.91
97.91
97.91
97.91
97.91
96.47
OPD
parameter-wise
98.10
98.10
98. 10
98. 10
98. 10
98. 10
98. 10
98. 10
98. 10
96.97
KFAC
layerwise
98.16
98.16
98.16
98.16
98.16
98.16
scalar
98.29
98,29
98,29
98,22
98.29
98.29
98.29
98.29
98.23
84.91
MAP
MAP
98.38
98.39
98.38
98.35
98.34
98.32
98.20
97.72
94.26
$7.66
purameter-wise
98,51
98.51
98.51
98.51
98.51
98.52
98.52
98.52
98.48
Diag
layerwise
97.71
97.71
97.71
97.71
97.71
97.71
97.71
97.71
97.70
76.89
scalar
97.91
97.91
97.91
97.99
96.24
Magnitude
parameter wise
98.10
98.10
98.10
98.10
98.10
98.10
98.10
98.10
98.10
94.22
KFAC
layerwise
98.16
98.16
98. 16
98. 16
98. 16
98. 16
98. 16
98. 14
97. 89
36.31
scalar
98.29
98.29
98.22
98.29
98.29
98.29
98.29
98.29
98.24
82.57
MAP
MAP
98.38
98.39
98.38
98.36
98.34
98.30
98.16
97.77
93.56
53.16
parameler-wise
90.09
62.13
39.76
31.51
19.52
22.50
18.93
14.84
15.57
11.81
Diag
layerwise
83.18
53.42
32.12
28.44
22.51
22.82
18.88
17.00
12.95
11.02
scalar
85.66
57.33
37.74
25.43
23.45
20.28
22.31
16.64
14.89
9.85
Random
parameter
39.27
27.96
18.72
17.04
13.66
KFAC
layerwise
87.00
58,40
41.93
33.39
30.62
27.91
20.77
21 02
14.07
10.49
scalar
90.16
64.54
40.36
34.11
30.63
28.92
17.88
18.35
13.82
11.47
MAP
MAP
97.26
87.54
65.94
52.32
47.22
44.12
40.72
22.46
16.59
8.87
Table B3: NLL of unstructured pruned ResNets on CIFAR- The best training method for each pruning criterion is highlighted in green, showing that SpaM improves performance over MAP for most criteria. The best performances (lowest NLL) overall at each sparsity level are shown in bold, showing that our OPD criterion outperforms the others at most sparsity levels.
Criterion
Training
Sparsity (%)
OPD
MAP
SpaM
0.36 0.0016
0.53 1 0.0013
0.52 + 0.0011
0.36 0.0016
0.34 = 0.0011
0.37 0.0014
0.69 2 0.0003
0.38 0.0022
0.44 0.0056
1.31 2 0.0086
2.08 0.0190
0.80 0.0270
2.62 1 0.010
0.0199
GraSP
MAP
0.31 1.000S
0.54 1 0.0032
0.66 2 0.0046
1.0019
1.73 0:0193
2.35 0.0276
2.69 0.008S
SpaM
0.37 + 0.0007
0.38 0.0006
0.40=00005
0.0022
0.51 + 0.0093
0.0321
3.71 + 0.0709
Magnitude
SpaM
0.37 00.0011
0.34
0.33
0.37 0.0012
0.33 200015
0.38 = 0.0016
0.00028
0.49 2: 0.0072
1.54200099
0.92 0.0320
0.0039
3.63 0.0418
0.0113
MAP
2.79 0.0433
2.63 + 0.0089
2.42 0.0146
2.36+00.004
2.33 + 0.0037
2.34 0.0043
2.30 0.0000
Random
SpaM
2.60:40.0292
3.22 0.0701
2.60 0.0230
0.0447
2.38 0.0044
2.31 0.0009
2.31 0.0003
SNIP
MAP
1 0.0235
0.0331
2.51 +00050
2.90 0.0450
4,44 0.0864
3,28 0,0203
SpaM
0.84 0.0320
0.0474
2.01 0.0814
0.1000
3.72 0.1244
0.0841
18

---

Test Accurcy (%) 1 Test Accuracy (%) 1
FCN (Breast Cancer)
FCN (MNIST)
LeNet (MNIST)
90
60
60
40
20
20
40
60
80
100
20
40
60
90
100
20
40
60
100
LeNet (FashionMNISTT)
LeNet (CIFAR-10)
ResNet (CIFAR-10)
80
46
60
40
16
20
60
80
100
20
40
100
20
40
100
Sparsity (%)
Sparsity (%)
Sparsity (%)
Approximation. Prior
param-wise
Ding layer-wise
unii-wise
KFAC LA,
KFAC layer-wise
KFAC LA. scalar
KFAC LA. unit-wise
Figure B2: Effect of different priors and Hessian approximations the sparsification performance with SpaM-OPD. The diagonal approximation with parameter-wise priors is a strong choice, espe-
cially higher sparsities, while the KFAC approximation with layerwise prior yields slightly better performances at lower sparsities.
Test Accuracy (%) f
FCN (Breast Cancer)
FCN (MNIST)
LeNet (MNIST)
100
100
80
75
60
20
60
80
100
20
100
20
40
Test Accuracy (%) 1
LeNet (FashionMNIST)
LeNet (CIFAR-10)
ResNet (CIFAR-I0)
80
60
80
45
30
40
.00
15
20
100
Sparsity (%)
Sparsity (%)
Sparsity (%)
Approximation. Prior
Ding I. param-uise
Ding 1AA, layer-uise
Ding 1.A, scalar
Ding I.A, unit-wise
MAP
KFAC LA, pamm-wise
KFAC LA. layer-wise
KFAC senlar
KFAC LA, unit-wise
Figure B3: Priors and Hessian approximations for GraSP pruning with SpaM and MAP, including our newer priors (parameter-we wise and unit-wise). We see that the effects are qualitatively similar to pruning with OPD.
advantage: it adapts the regularization each individual weight based on the data, rather than relying on single global parameter, allowing for more nuanced control over sparsity.
B.8
Additional Pruning Results
B.8.1
Wide ResNet
In Figure B7, we demonstrate how SpaM enhances the sparsity performance of Wide ResNet models. This is specifically illustrated the case of OPD, GraSP, and Magnitude, while maintaining low Brier score, ECE, and NLL up to 95% sparsity.
19

---

FCN (Breast Cancer)
FCN (MNIST)
LeNet (MNIST)
100
Test Accuracy (%)
Text Accuracy (56)
Accuracy
80
40
100
Sparsity
Sparsity
Sparsity (9))
LeNet (FachionMNIST)
LeNet (CIFAR-10)
MI.PMixer (MNIST)
1000
80
Accuracy (%)
Tixs Accuracy (16)
(Sporill
SNIP Spanddy Taree
Kanders (One Slot
GuSP (Que Spall
40
Spunsing
Megnitade (Spunilly
Spursity
Sparsity (9))
Figure B4: SpaM post-hoc pruning efficiency with optional fine-tuning after the pruning. Unlike other pruning criteria, OPD does not require additional tuning to achieve optimal performance across different architectures and often still outperforms the other fine-tuned methods.
PCN (Broast Cancer)
FCN (MNIST)
LoNet (MNIST)
1000
Test Accunity (96)
00
60
00
Augmented)
Test Accurccy (9611
00
40
20
Spursity (55
Spunsity (%)
Sparsity (9)
SpA5
Spuht
Spolt Magrinie
Spalt
Figure B5: Predictive performance function of sparsity level in unstructured pruning. We include our online pruning approach that progressively prunes a model during the training compared to the other curves demonstrating the performance 10 pruned models based converged baseline. our online pruning approach is often competitive with post-hoc pruning.
B.8.2 Vision Transformer
Figure B8 demonstrates the impact of SpaM on Unstructured Pruning for Vision Transformer (ViT) trained on MNIST. These results align with the findings presented in Section 5 with SpaM diagonal LA and parameter-wise priors leveraging the sparsifiability of models using OPD, Magnitude and GraSP, maintaining a test accuracy of 97% for OPD and Magnitude at 95% sparsity compared to an accuracy of lower than 20% under MAP for the same methods. This serves as a prooffof-concept for vision transformers but efficacy has to be verified at larger scale where such models perform best.
20

---

WRN on CIFAR-100
LeNet on MNIST
100
60
Test Accuracy (%6
80
40
60
40
20
80
100
20
40
60
80
100
Sparsity (%)
Sparsity (%)
SpaM-OPD
SpaM-Magoitude
SpaM-Random
SpaM-SNIP
SpaM-GraSP
MAP-OPD
MAP Magnitude
MAP
MAP-Ramdom Random
MAP-SNI
MAP.GraSP
MLE-LI-OPD
MLE- Magnitude
MLE L1 Random
MLB LI SNIP
MLE GraSP
Figure B6: Comparing SpaM and MAP to LI Regulatization. We that SpaM consistently
outperforms 1.1 regularization.
Aucuray
Spusity
0.32
Blne
6.4
0.06
NI.L
0.04
100
100
Spenityy
Spursity
Spunity
Swit
Spalt Rackes
Spali Nagulak
MAP-SNP
MAPCWAR
S40P Paa Hoy
Figure B7: SpaM post-hoc efficiency for Wide ResNet CIFARIO. Leveraging OPD, GraSP, and Magnitude performance under SpaM in comparison to MAP show superior test accuracy at increased sparsity levels coupled with low ECE, Brier Score, and NLL.
B.8.3 GPT-2
We demonstrate the efficacy of OPD on pre-trained GPT-2 model (124M parameters), fine-tuned for sentiment analysis the IMDB dataset. To manage computational resources, limit both the Laplace approximation and SpaM to two steps. Despite this constraint, OPD maintains high predictive performance SpaM optimization with more epochs and a more refined posterior could further enhance performance.
60% sparsity, shown in Figure B9. This suggests that extending
B.9 Visualization of the Pruning Process
In order to illustrate the structural evolution of the model throughout the different sparsification approaches (unstructured and structured), we provide a sequence of filter bank visualizations that delineate the principal stages of pruning, progressing from the initial dense architecture the ultimate compact configuration. These visualizations also reveal the influence of parameter-wise and unit-wise priors the weights (Figure B10).
21

---

Teet ACCeEry (167
50
Spunity (9)
0.12
0.00
0.0
Drizz
0.04
6.0
20
20
68
20
20
70
60
90
100
20
400
50
50
90
100
20
30
80
53
00
50
100
Spanity (%5
Spanitt (%6
Syarsiy (93)
Figure B8: SpaM post-hoc efficiency for VIT on MNIST. Leveraging OPD, GraSP, and Magnitude performance under SpaM in comparison to MAP show superior test accuracy at increased sparsity levels coupled with low ECE, Brier Score, and NLL.
GPT-2 (IMDB)
SNIP
90
GmSP
Test Accurucy (
80
Magnitude
70
60
50
20
30
40
50
60
70
80
90
100
Sparsity (5)
Figure B9: GPT-2 (124M) on IMDB. We tune GPT-2 for sentiment analysis on IMDB datasets. Our results show that OPD maintains significantly higher accuracy than other methods, which degrade towards random classifier performance (50%) at 60% sparsities.
B.10 Network Compression
In Figures B11 and B12, we demonstrate the efficiency gains achieved by our SpaM-OPD approach. For the fully connected network on the Cancer dataset, it achieves a remarkable reduction of over 20 times disk size and 24 times FLOPs while simultaneously maintaining baseline test accuracy. Additionally, it boasts Brier score of 0. 0.15 and negative log marginal likelihood (Neg Log MargLik) lower than the original model. These results highlight the effectiveness SpaM-OPD achieving significant model compression without compromising performance on key metrics.
C Technical Details
C.1 Resizing and Compression
Post structured pruning, the model may undergo fine-tuning to regain performance. this process, pruned structures completely removed from the architecture rather than merely being zeroed out.
22

---

Unstructured Structured
Baseline- - Unpruned
Lise
95 % Sparse Parameter-wise
95 Sparse Unit-wise
95 % Structured
Visualization of the sparsification effect on model's layers for LeNet on FashionM- NIST sparsified at 95% using both the structured and unstructured approach. Blocks in black refer to masked filters, and columns refer to neurons pruned. We see that for unit-wise priors, 95% sparsity yields more entire kernels and neurons being masked compared to parameter priors.
Figure B10:
90
90
Test accunity (8)
Test accuracy (R)
60
60
50
80
300
300
20
20
1.00004
6.20011
2.5elt5
HLOPS
Disk size (Bytes)
0.110
0.86
Marglik
01.08
Brier
0.00
0.0
0.02
0.4
FLOPS
FLOPS
Figure B11: Structured and unstructured pruning of LeNet FashionMNIST with SpaM-OPD. We see that through structured sparsification, we are able to obtain models that are still performant at significantly reduced computational and memory cost, while unstructured pruning does not directly translate into computational benefits.
23

---

90
Test accumicy (16) 1.
80
80
Text accuracy
60
50
50
unstractured
10a
10
10
10'
FLOPS
Disk size (KB)
0.4
Log MargLik ik
0.3
Brier
0:2
10
10
10a
10a
10
FLOPS
FLOPS
Figure B12: Structured and unstructured pruning FC on Cancer with SpaM-OPD. We see that through structured sparsification, we are able to obtain models that are still performant at significantly reduced computational and memory cost. At the same time, unstructured pruning does not directly translate into computational benefits.
This leads to a network with fewer filters in convolutional layers and a reduced number of neurons in fully connected layers, resulting in leaner and more efficient model.
The process of compaction involves transferring the weights from the pruned model to a newly created, smaller architecture that is aligned with the dimensions of the retained active structures. This results in a denser, storage- and computation-optinizz model.
Algorithm summarizes this entire process of structured pruning and model compaction.
This approach transitions the model from a pruned state compact and optimized architecture. The final compressed model Mcompact not only retains essential predictive capabilities but is also further tuned for performance. The newly configured Moompact saved with updated parameters, ensuring efficient inference and case of deployment, especially on resource- conttree-nntttieeedd edge devices.
The reduced memory footprint and FLOPS of Meompact particularly beneficial for deployment on edge devices with limited computational resources. When models exceed the hardware limits, aggressive compression techniques like quantization may be required, which compromise perfor- mance. Our method aims to significantly reduce the memory size of the model while minimizing performance trade-offs. The effectiveness of our approach in achieving this balance explored in Section 5.
C.2 Pseudocodes
Algorithm outlines our structured pruning procedure, highlighting how efficiently achieve simpler model by transferring weights to smaller one.
D Experimental Setup
D.I Datasets
Breast Cancer Wisconsin (Diagnostic) (UCT): This dataset, derived from digitized images of fine needle aspirates of breast masses, includes features describing characteristics of cell

---

Algorithm Structured OPD pruning
Require: Trained Model M, Target Sparsity Threshold T
Ensure: Compacted Model Mcompuct Count of Pruned Units Nraundd
1: for each layer / in M do
2: if is not the output layer then
3: for each structure layer do
4: Calculate A, Eies Pu: P.
5: end for
6: Sort structures in by A, in ascending order
7: Determine the number structures to prune based on T
8: Prune determined number of structures with the lowest A, values
9: end if
10: end for
Update Npruned with the count of pruned structures
12: Fine-tune the pruned model M
13: Initialize Mommpea with dimensions aligned the unpruned structures of M
14: Transfer weights from unpruned structures of M to Maompect
Save compact with updated parameters
nuclei in the images. It is classic binary classification dataset used extensively in breast cancer research [57].
MNIST: A foundational benchmark dataset in machine learning, MNIST consists of 60,000 training and 10,000 test images of handwritten digits (0 to 9) in 28x28 pixel grayscale format [58].
FashionMNIST:: A drop-in replacement for MNIST, Fashion-MNIST offers greater chal- lenge with 60,000 training and 10,000 test images in grayscale (28x28 pixels). Each image represents one of ten clothing categories [59].
CIFAR-1O: This dataset contains 60,000 color images (32x32 pixels) divided equally among 10 classes (e.g., airplane, bird, cat) [60]. For our ResNet experiments, we augment CIFAR-10 with random flipping and cropping.
CIFAR-100: A more fine-grained version of CIFAR-10, this dataset includes 60,000 color images (32x32 pixels) across 100 classes, with 600 images per class [60]. We apply random flipping and cropping for augmentation.
IMDB Movie Review: This dataset is collection of 50,000 movie reviews, balanced between positive and negative sentiments. is commonly used for binary sentiment classification tasks [61].
D.2 Models
FCN for MNIST (784, 256, 10): This Fully Connected Network (FCN) is specifically designed for the MNIST dataset. It comprises an imput layer with 784 nodes, a hidden layer with 256 nodes, and an output layer with 10 nodes, making it 2-layer FC network. Its architecture is optimized to handle the simplicity and characteristics of handwritten digit images.
FCN CANCER (30, 100, 2): Customized for the CANCER dataset, this FCN includes an input layer of 30 nodes, two hidden layers, each containing 100 nodes, and a final output layer of 2 nodes. The 3-layer structure of this network is instrumental in distinguishing between benign and malignant tumors based on cellular features.
LeNet: As a foundational Convolutional Neural Network (CNN), LeNet has shown excep- tional performance in digit and image recognition tasks. We have applied LeNet to the MNIST, Fashion MNIST, and CIFAR- 10 datasets, leveraging its capability handle varying complexities of image data [58]. LeNet on CIFAR- is not a very common benchmark for pruning; here, is used to demonstrate how SpaM, and specifically SpaM-OPD, able to prune at high percentages without performance loss up to 80% in model that struggles with representing the data's 's complexity, showing that our work extends beyond perammrretri networks for the task hand.
25

---

MLPMixer:
The MLPMixer serves as a streamlined alternative to more complex models like CNNs and transformers. It relies solely on Multi-Layer Perceptrons (MLPs) for integrating inputs across spatial and channel dimensions [62]. We implement an MLPMixer with 2 blocks designed for MNIST.
ResNet with inplanes 64 and depth 18 for CIFAR-IO: We modify the implementation of ResNet and incorporate fixup initialization and custom bias and scale parameters to align with the constraints of the ASDL backend [63] used for the Laplace computations in this work, which does not support batch normalization.
Wide ResNer: decreases depth compared to ResNet and increases the width of residual networks [64] with depth of 16 and widening factor of 4 (WRN16-4). We use fixup blocks to be able to utilize ASDL backend [63].
Vision Transformer (ViT) [65]: unlike CNNs, which extract local features through filters and pooling layers, ViT breaks down images into fixed-size patches, treating each as a "token" in a sequence [65]. This allows it to leverage the Transformer architecture, initially designed for language processing, analyze relationships between patches through self-attention mechanisms [66].
DistilBERT [67] is a smaller, faster, and cheaper version of BERT, achieved by leveraging knowledge distillation during the pre-training phase. This model re- tains 97% of BERT's language understanding capabilities while being 60% faster and 40% smaller. We use the pre-trained DistilBERT hosted in Hugging Face under [67] and tune it for sentiment analysis to classify reviews
DistitBERT:
(distilbert www.aaasssssseeeeeeeeeeaeeaaaaae in the IMDB dataset [61], which involves predicting the sentiment (positive or negative) of user reviews based on their textual content.
GPT-2: a large-scale transformer-basee language model developed by OpenAI, with impres- sive text generation capabilities. Trained on vast corpus of internet text [68]. In our study, we leverage the 124M parameter version of GPT-2, fine-tuning it on the IMDB dataset for sentiment analysis to assess its performance under different pruning conditions.
D.3 Dependencies
For the computation of second-order information (e.g., Hessian, Fisher information) needed for the Laplace approximation, we utilize the ASDL Library [63]. We use the library in its version under https://githun com/kazukiosawa. aaaaaaadl//seeeee 01 (tt09999999333300449999993333339
The ASDL Library is distributed under the MIT License, which allows for reuse with a few restrictions that we respect our work.
D.4 Hyperparameters
Marginal Likelihood
The choice between GGN and EF. GGN was initially employed for fully connected networks, LeNets, and ResNets. However, for complex architectures (WRNs, ViTs, DistilBERT), GGN's computational cost became prohibitive, exceeding MAP runtime by up to 20x and even more for casual modeling tasks. Switching to EF maintained pruning performance while closely matching MAP runtime, which is particularly beneficial GGN scales linearly with the number of classes. We discuss further the cost in Appendix D.7.
Hessian Approximation:
sepochssburnit Dictates the number of epochs after which marginal likelihood optimiza- tion starts. If set superior to the number of training epochs, marginal likelihood is skipped, and the training is equivalent to MAP.
marglik frequencyy Controls the frequency of marginal likelihood estimation. The default value of signifies re-estimation after each epoch, while a value of 5 indicates approximation for every fifth epoch.
We use these parameters to manage the computational cost of our experiments, where for small models like LeNets, FC Networks, the "_epochs_ _pochs_burrin burnin set to zero and reflecting estimating each epoch since the start of the training. In contrast, for complex networks like
to one
26

---

MLPMixer, ResNets, WideResNet, and VIT that we train from scratch, we start after 20 epochs and an interval frequency of 5 epochs.
Hyperparameters Table D4 presents the specific hyperparameters employed for each dataset- architecture combination. We use i to denote the of data augmentation in the training process. The symbols * and o represent the use of the Generalized Gauss- Fisher (EF) approximations for the Hessian, respectively. We use cosine decay scheduler towards fixed minimum learning rate of le-6 across all experiments. The symbols Dj, D2, etc., represent the following datasets:
(GGN) and Empirical
Dj: Breast Cancer Wisconsin (Diagnostic)
D2: MNIST
Dy: FashionMNIST
D4: CIFAR-10
Dg: CIFAR-100
Dg: IMDB Movie Review
All models are trained from scratch, denoted by the symbol A, except for DistilBERT and GPT-2, which are fine-tuned from pre-trained weights and are indicated by V.
Table D4: Hyperparameters used in the experiments.
Dataset (Arch.)
Marglik Freq.
Batch Size
Learning Rate
Optimizer
Temp.
Burn-in Epochs
D4 (FCN)
0.001
Adam
0/
1DA (PCN)
0.001
Adam
0// 100
D (LeNet)
128
0.001
SGD
1.0
0/ 100
D (LeNet)
1
128
0.001
SGD
1.0
0/ 100
D (MLPMixer)
1
128
0.001
Adam
1.0
0/ 100
D (ResNet)
5
128
0.1
SGD
20 100
D (WRN)
5
128
0.1
SGD
20/200 200
D (ViT)
128
0.001
Adam
20
D (DistilBERT)
5
20-5
Adam W
1.0
5/20
D (GPT-2)
5
8
26-5
Adam
1.0
5/10
Computational resources Our experiments are run on GPUs. We run our experiments in a single GPU configuration available variation between 1080 Tis, V100s, and A A100s, with the majority being run on A100s with 40GB memory as we run the experiments intensively one after the other for different architecture on the same allocated GPU and in order provide enough GPU memory. For models such as FCs, LeNets, ResNets, and MLP-Mixe a GPU with 12GB of memory ( (0880 Ti) proved sufficient to parameter-wise priors and reproduce the results. For the sentiment analysis task using GPT-2, we recommend using 32 GB GPU for tuming be able to utilize high batch size and diagonal approximation to fit laplace on the data without running into memory shortage.
method for recommended laplace and prior, which is diagonal with
Runtime
Table D5 presents the training and pruning runtimes A100 for each dataset-architecture combina- tion. Training times are given for both SpaM diagonal with parameter-wise prior and MAP, while pruning time is identical to both. Pruning runtimes refer the time taken for OPD compute and prune model at 10 target sparsities. OPD and magnitude are very close in terms of runtime and the most efficient compared to SNIP, which is slightly slower due to it requiring an additional forward pass, and GraSP, which is significantly slower as it accumulates the gradient as shown in Figure D13.
D.5 Pruning Criteria
SNIP: Uses connection sensitivity, how much specific weight contributes to the output loss, for effective pruning [36].
Employs gradient signal preservation. GraSP relates to the concept of Gradient Flow (GF), defined as:
GraSP:
GF GF (L)()) [0)
(DI)
27

---

Table DS: Runtimes for the experiments. Train: Training time (h:m), Prune: Pruning time (m:s).
Dataset (Arch.)
Train
Prune
SpaM
MAP
OPD
D (FCN)
0:01
0:01
0:04
2 (FCN)
0:15
0:5
0:23
D (LeNet)
0:16
0:10
0:15
D (LeNet)
0:16
0:10
0:21
D (MLPMixer)
0:05
0:07
0:10
4 (ResNet)
1:24
0:25
0:55
D (WRN)
1:17
1:12
0:51
D (VIT)
0:26
0:15
0:24
D (DistilBERT)
5:20
2:15
1:05
D (GPT-2)
17:34
6:24
17:41
Pruning Time with Standard Eroo by Method for WRN CIFAR-100
2500
2121.13 + 45.04
2000
Runtime (s) Standard Error
1500
1000
500
184.08 + 3.62
50.73 + 1.06
51.85 = 0.68
51.91 # 1.11
GreSp
Megnitud
OPD
Randemi
SNIP
Pruning Method
Figure D13: Mean relative pruning time with standard error bars on WRN with CIFAR-100. OPD and Magnitude are the most efficient they use pre-computed parameters, with SNIP being slower due to requiring an additional forward pass, while GraSP is significantly slower as it needs to accumulate the gradient.
emphasizing the impact of pruning the training dynamics [37]. We replicate the GraSP im- plementation Rachwan et al. [39], where we consider the absolute value of the importance score initially proposed by Lubana and Dick [38] given by:
r(O.) = |o Hr. (o.)g)r..(..))
(D2)
Note that while the importance score was initially used before training, we propose to use this importance score one-shot criterion after the training process and show how SpaM can leverage the performance of GraSP.
Structured-SynFLLoww We challenge the capabilities of SynFlow [50], a data-agnostic pruning approach that prevents layer collapse that happens at high sparsities where layers are no longer able to perform at the model's predictive power. This typically occurs when the pruning algorithm, intentionally inadvertently, removes significant portion of weights or filters from specific layer, effectively collapsing its functionality [50]. We push SynFlow to its limits through advanced structured pruning strategies, where prune layers aggressively
28

---

at the same target sparsity. which facilitates the compression process and resizing. By applying rigorous layer-specific filtering and neuron pruning, we aim to test the robustness and effectiveness of SynFlow in extreme sparsity structured scenarios. This approach not only benchmarks SynFlow's performance under stringent conditions but also explores its potential to maintain network functionality and accuracy in highly sparse neural network architectures.
Magnitude Pruning: Relies on the magnitude of weights for pruning, aiming to maintain model performance while reducing complexity [69]. After the success shown by Han al. [69], many methods adapted magnitude pruning criterion coupled with different scheduling [49, 70, 48].
Random Pruning: Prune weights or structure randomly.
D.6 Structured Sparsification Process
For structured sparsification, contrasting with the unstructured approach, the process necessitates reshaping the weight matrices to effectively reduce model complexity. The steps include:
1. One-shot structure masking based aggregated importance scores.
2. Continue training for five epochs using the model from Step 1 for preliminary evaluation.
3. Implementing two software design approaches:
In-place layer replacement in the model with smaller ones fitting the non- non-masked regions.
Creating a new, flexible model initialized match the dimensions of the non-masked areas, requiring repeated reading of the nonzero mask for state-dictionary and metadata alignment.
4. Transferring non-zero structures smaller layers and tuning the model.
Post structure removal, we extend the training phase to adapt the model weights and re-evaluate, ensuring seamless functionality once transferred smaller layers. Particularly after significant structural reduction, our primary objective shifts to maximizing performance in the downsized model. This fine-tuning spans 10 epochs depending on the complexity of the original model's structure, which was initially trained for either 50 or 100 epochs.
D.7
Computational Cost
Instead of using the Generalized Gauss-Newton (GGN) approximation, which scales linearly with the number of classes, we can also use the Empirical Fisher (EF). For most architectures, using EF instead of GGN for SpaM does add very large computational overhead to MAP, as EF costs roughly as much as gradient computation. This is particularly beneficial as GGN scales linearly with the number of classes. The pruning results are not significantly affected by the choice of GGN EF
The runtime of MAP and SpaM was close (roughly 1h and 20 minutes on A100s) for WRN- 16 on CIFARI00 using SpaM (EF) with diagonal LA and parameter-wise priors (our recommended settings for pruning). For language transformers, specifically DistilBERT and GPT-2, SpaM with EF does result in longer training time compared to MAP. However, this increase is considerably less than when using GGN, where a single epoch can take longer than the entire SpaM training with EF
In prior works [71], it was found that GGN gives better posterior predictive approximation, but do not use in this work. find that EF works similarly well for pruning at a much lower cost.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide experimental evidence in Section that supports our claims made in the abstract and introduction.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations of our approach Section 6.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide proof of our theoretical result for diagonal priors when using KFAC (Proposition A.1) in Appendix A. The proof was also numerically verified.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide a detailed description of the SpaM training approach, how we compute it, and how we derive our importance score. We provide a detailed description of pruning criteria and hyperparameters in Appendix D.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We released our code publicly at https: //githu// com/fortuinles spam pruning. The code contains the script for SpaM, unstructured and structured pruning, well as experiment manager.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We explain how we train the models using MAP and SpaM and how we perform the pruning. We describe the experiments in detail in Appendix D.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We conduct each experiment at least for 4 different seeds and compute the standard error to mitigate the effect randomness. the case we do not have enough seeds, which is for experiment of tuning GPT-2, we feature the results only in the appendix.
8. Experiments Compute Resources
30

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide detailed description of our experimental setup and implementa- tion in Appendix D.
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics atttp::/neurr/ oc cc/public/ wttp://tttiiiieeeieeeeeeeeeeeeee
Answer: [Yes]
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We developed method that improves the sparsifiability as well the pruning of neural networks. Pruning weights leads to potentially lower storage and inference costs for any neural network and thus does not immediately imply any possible harm.
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our paper used publicly available datasets and architectures that do not pose (new) risks.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
In Appendix D.3, state that use the ASDL library to compute second-order information needed for the Laplace approximation and align with its license restrictions. The paper itself is under the CC BY 4.0 license and the library is under the MIT license.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We released the code including documentation.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This does not apply as we do not experiment with human subjects.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This does not apply we do not experiment with human subjects.
31


Reviews:
[{"review_id": "YGZXqCoLLi", "reviewer": "Reviewer_rzPD", "summary": "The authors present a framework for assessing the sparsifiability of a parametric model, a measure of how many parameters can be pruned without severely affecting the modelling performance. In essence, the authors suggest training a Bayesian neural network (BNN) using the marginal likelihood estimated via the Laplace approximation. The marginal likelihood's automatic Occam's Razor ability to identify good trade-offs between model complexity and data fit will encourage sparsity, and the trained network can then be subjected to a pruning criterion of choice.\n\nWhile any pruning criterion can be used, the authors propose the Optimal Posterior Damage (OPD), which is computed as a cheap byproduct of their marginal likelihood approximation. This often outperforms more expensive approaches.\n\nThe authors demonstrate the effectiveness of their proposed approach through several experiments covering performance at different sparsity levels, online pruning, uncertainty estimation, the influence of prior choice, and structured sparsification. Generally, their proposed approach and pruning criterion outperforms baselines.", "questions": "1. Additional experiments are probably infeasible for the rebuttal period, but did you already experiment with other prior distribution families? Why did you choose a normal distribution rather than a sparsity-inducing family?\n2. The results for the online pruning approach in figure B6 confuse me a little. The online approach seems to generally perform better for higher sparsity levels for CIFAR-10, with LeNet on CIFAR-10 being particularly extreme (test accuracy around 50% for 20% sparsity compared to 65% for 80% sparsity ). Do you know what happens here?", "limitations": "The authors have adequately addressed the limitations of their approach.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**\n1. Using the marginal likelihood to encourage less complex networks appears original in the context of pruning, although a little incremental. \n2. The proposed pruning criterion, Optimal Posterior Damage (OPD), and the structured priors for the KFAC approximation appear to be original contributions.\n\n**Quality**\n1. Marginal likelihood optimisation is a well-known technique which has many theoretically and intuitively pleasing properties.\n2. The OPD pruning criterion makes intuitive sense and seems remarkably powerful.\n3. The experimental evaluation is quite extensive and includes repetitions over multiple seeds (and the resulting uncertainties over the reported metrics). I find the evaluation of different choices for the structure of the prior is particularly interesting, and the recommendation for a default configuration is great.\n\n**Clarity**\n1. The paper is very well-written, and the plots are generally informative and easy to understand.\n\n**Significance**\n1. The marginal likelihood idea is simple yet works well in practice, which is great for the potential significance of the approach.\n2. The proposed pruning criterion seems bound to become a strong baseline whenever the training approach is used.\n3. The structured priors for the KFAC approximation are a minor contribution, but will likely be used outside of the pruning literature too, thus making them quite significant.", "weaknesses": "**Originality**\n1. While using the marginal likelihood as the training objective in the context of network pruning seems original, it seems somewhat incremental.\n\n**Quality**\n1. The paper proposes a combination of two methods, training a Bayesian network using (an approximation to) the marginal likelihood and a pruning criterion, but only the choice of pruning criterion is evaluated experimentally, not the training scheme. Is training using the marginal likelihood better than a simple maximum likelihood optimisation with, say, L1 regularisation? The authors don't answer this important question.\n2. While the authors test the effect of different prior structures (which are great and interesting experiments!), they do not test the effect of different prior distribution families. They seem to use a normal distribution for all experiments, which is surprising since the normal distribution doesn't induce sparsity and has been shown experimentally to be quite a poor choice for BNNs. Some obvious choices would be to test the horseshoe prior or the Indian buffet process prior, which can both encourage sparsity (e.g., Ghosh et al., 2018; Kessler et al., 2021). Other interesting choices would be the Laplace distribution, for which the MAP solution with a diagonal prior would correspond to L1 regularisation, Student's t and the spike-and-slab prior. Taken together with the remark above, it seems like the paper is missing half of the experimental analysis.\n3. Experimentally, I would have liked to see results for the different pruning criteria in the settings that were used in their original publications (e.g., same training scheme). It is unclear if the paper's results are on par with the literature's. \n4. Minor weakness: it would have been helpful to see the performance of the unpruned networks (i.e., 0% sparsity) to understand the penalty caused by the sparsification fully.\n\nComment: if the authors decide to include more experiments in a future revision, I think sections 1 through 3 could be significantly shortened without losing too much context.\n\n**Clarity**\n1. While the paper is overall very well-written, the terms in Eq. (4) lack definitions, and the appendix appears a little rough. \n2. The font size in the plots could be a little larger.\n\n\n**References**\n- Ghosh et al. (2018). \"Structured variational learning of Bayesian neural networks with horseshoe priors.\" ICML.\n- Kessler et al. (2021). \"Hierarchical Indian buffet neural networks for Bayesian continual learning.\" UAI.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses**\n> While using the marginal likelihood as the training objective in the context of network pruning seems original, it seems somewhat incremental.\n\nThank you for the chance to expand more on the novelty offered with our approach and scalability compared to established works. In fact, our framework addresses existing scalability challenges, making it practical for real-world scenarios, and enables automatic relevance determination in deep learning for the first time.\n\n**Quality**\n> The paper proposes a combination of two methods [...] Is training using the marginal [...] a simple maximum likelihood optimisation with, say, L1 regularisation? The authors don't answer this important question.\n\nThank you for your insightful question. We consistently compare MAP and marginal likelihood training throughout the paper (solid and dashed lines in the plots). MAP with Gaussian priors (L2 regularization) is our primary baseline, as it corresponds to the widely used standard weight decay.\nWe ran an experiment with L1 for the rebuttal (see provided pdf), which gave worse results.\n\n> Exploring other prior distributions like the horseshoe prior or Laplace distribution, which have been shown to induce sparsity and improve performance in Bayesian neural networks [...] the Laplace distribution, for which the MAP solution with a diagonal prior would correspond to L1 regularisation, Student's t and the spike-and-slab prior. \n\nThank you for the opportunity to clarify. We address your concerns about prior choice in the general response. Alternative prior families often require expensive inference, making them infeasible for large-scale models. The Laplace approximation, which we employ for scalability reasons, requires differentiability, excluding many of your suggested priors. We instead use the idea of relevance determination, that is regularization of individual weights or weight groups. With the Laplace approximation, this is scalable to networks at any scale, thanks to modern Hessian approximations.\n\n> Experimentally, I would have liked to see results for the different pruning criteria in the settings that were[...]. It is unclear if the paper's results are on par with the literature's. \n\nThank you for your valuable comments. While SNIP and GraSP are primarily used for pruning at initialization (PAI), their criteria based on weight contribution to loss and gradient preservation can be applied broadly for weight importance evaluation in various scheduling scenarios. However, using them as PAI often requires initial calibration passes for the network, especially at high sparsities, to achieve the desired pruning target. Additionally, compared to post-training methods, PAI can be less efficient, especially for large models that need retraining from scratch or rely on the availability of the original dataset (we have included PAI setup in our submitted code). In contrast, methods like SpaM can be more efficient for networks built on top of transfer learning, while OPD can be directly applied to pre-trained models by computing the inverse Hessian, as demonstrated in our GPT-2 experiments.\nOur main aim was to show the benefits of SpaM in a harmonized experimental setup, regardless of the specific pruning criterion.\n\n\n> Minor weakness: it would have been helpful to see the performance of the unpruned networks (i.e., 0% sparsity) to understand the penalty caused by the sparsification fully.\n\nThank you. We found that most methods, especially OPD, perform at almost identical performance to the baseline at 20% and hence did not report 0% sparsity so far. We will add a horizontal line with the baseline accuracy that will also make it easier to visualize the performance gap at higher sparsities.\n\n**Clarity**\n> While the paper is overall very well-written, the terms in Eq. (4) lack definitions, and the appendix appears a little rough.\n\nThank you  we will improve the definition in the revision and polish the appendix. $A_l$ and $G_l$ are the uncentered covariances of the layer inputs and output gradients, respectively. These are used as defined in the referenced works, for example [1].\n\n[1] Martens, James, and Roger Grosse. \"Optimizing Neural Networks with Kronecker-factored Approximate Curvature.\" 2020\n\n> The font size in the plots could be larger.\n\nWe will increase the font size to improve readability and accessibility.\n\n**Questions:**\n> Additional experiments are probably infeasible for the rebuttal period, [...] experiment with other prior distribution families? Why did you choose a normal distribution rather than a sparsity-inducing family?\n\nSee our arguments regarding scalability above. Specifically, in the context of automatically learning regularization, the Gaussian group-wise prior, as in automatic relevance determination (ARD), has proven effective. Other priors, due to non-differentiability, could require intractable manual hyperparameter tuning.\n\n> The results for the online pruning approach in figure B6 confuse me a little. The online [...] for CIFAR-10, with LeNet on CIFAR-10 being particularly extreme [...]. Do you know what happens here?\n\nThe x-axis in the online pruning curves (section B.6) indirectly reflects training progress, not just sparsity. LeNet on CIFAR-10 is pruned starting from epoch 10 for 100 epochs, aiming for 99% sparsity.  LeNet's curve trend on CIFAR-10 is therefore due to ongoing convergence during pruning, caused by dataset complexity relative to the architecture.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iH62dy58pG", "reviewer": "Reviewer_oKfx", "summary": "The paper introduces a new pruning technique named SpaM, which leverages Laplace approximation and Bayesian marginal likelihood for approximating the posterior in Bayesian Neural Networks. This approach includes two main components: the Gaussian prior variance and OPD, a pruning method that utilizes the Laplace posterior precision.", "questions": "Questions:\n\n- Previous Studies Comparison: A comprehensive comparison between SpaM and previous studies using Bayesian inference or variational methods for pruning would be beneficial.\n- Clarification of Procedures: The paper should explicitly explain the sequence of dense training followed by prune-after-training, as this is critical for reader understanding.", "limitations": "The authors have sufficiently addressed the potential negative societal impacts of their work. For details on the limitations, refer to the Weaknesses and Questions sections.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Strengths:\n- Clarity and Accessibility: The paper is well-written, making it accessible to readers.\n- Motivation and Methodology: The use of marginal likelihood for pruning neural network models is well-motivated. The proposed method is applicable to both unstructured and structured pruning scenarios.\n- Bayesian Justification: The method is grounded in Bayesian theory, presenting a new pruning criterion based on posterior approximation.\n- Simplicity: The approach is relatively simple compared to existing Bayesian pruning methods.", "weaknesses": "Weaknesses:\n\nMethodological Clarity:\n\n- The computational and storage challenges associated with the Fisher matrix or generalized Gaussian Newton matrix in the context of Laplace approximation for Bayesian Neural Networks are not sufficiently addressed in the limitations section.\n- The integration of the method with existing pruning criteria is unclear. The process involving MAP solution, Laplace approximation, and OPD with the precision matrix needs clarification, especially in comparison with zero-shot pruning methods like GraSP, SNIP, and Random.\n- The fairness of comparing the proposed method with Monte Carlo sampling for BMA performance computation against MAP is questionable. It should be clarified if only a single pruned model or Laplace approximation is used.\n\nExperimental Design:\n\n- The computational and memory costs of Laplace approximation should be compared with other baselines.\n- The baseline methods used are too simplistic. Stronger baselines such as IMP, RigL, or DST should be included.\n- The experiments are limited to CIFAR10. Additional datasets like CIFAR100 and ImageNet would strengthen the findings.\n- The datasets and networks used are relatively simple, raising questions about the method's performance on more complex datasets and models.\n- Specific questions about the online pruning process need addressing, such as the timing of pruning, the validity of Laplace approximation during training, and optimization of prior parameters.\n\nNovelty and Comparison:\n\n- The novelty is somewhat limited, as the method relies on known ideas from the literature.\n- The differences in uncertainty estimation between SpaM and MAP are marginal. The presentation of results could be improved to better highlight these differences.\n- Previous studies using Bayesian inference or variational methods for pruning should be compared with SpaM in both related work and experiment sections to enhance understanding.\n\nThe scale of Experiments:\n\n- The experiments are considered limited in scale, with MNIST and CIFAR being seen as insufficient to provide meaningful insights. Larger models and datasets should be used for validation.\nPerformance Goal:\n\n- The ultimate objective in neural network pruning is to achieve performance comparable to the unpruned dense model at a given sparsity level. The paper should position SpaM+OPD in relation to other pruning methodologies more clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Methodological Clarity:**\n\n>  Computation/Storage challenges\n\nWe discuss computational and storage costs of SpaM in depth in the main text but also in appendices D.4 & D.7. A diagonal approximation costs as much to store as a parameter vector and KFAC costs roughly twice as much so we incur minimal storage overhead. Computationally, using the EF is as cheap as gradient computation while the GGN scales with the number of outputs. For the benefit provided, these are minor additional costs.\n>  Integration of the method is unclear.\n\nThere seems to be a misunderstanding as we have already covered these details. In appendix B.1, we explain that we initially train the NN either with MAP (a) or SpaM (b). Once these networks are trained, we prune both models using the different pruning criteria at different sparsity levels. Thus, we can measure the effect of SpaM training and OPD separately and fairly. \n>  Fairness of comparison  [...].\n\nWe do not use the predictive posterior for inference or Bayesian model averaging (BMA) but simply use the point estimate network. During inference, the model trained with SpaM only uses a single forward pass over the pruned architecture, thus guaranteeing a fair comparison with the baselines. The posterior is solely used to estimate the marginal likelihood during SpaM training.\n\n**Experimental Design:**\n>  Computational and memory costs of Laplace approximation.\n\nWe compared the cost of Laplace approximation used in SpaM (both using GGN and EF) with MAP and, as theoretically validated, it only adds minor overhead, especially in the diagonal EF variant.\n>  Baseline methods are too simplistic.\n\nWe appreciate your feedback. To address this point and avoid any misunderstanding, we have addressed your comment in a detailed explanation in our general response.\n> Experiments are limited to CIFAR10. Additional datasets like CIFAR100 and ImageNet [...].\n\nThe experiments are not limited to CIFAR10. We have included CIFAR100 and IMDB as larger and different datasets, respectively.\n>  Datasets and networks used are relatively simple.\n\nWe indeed cover in our paper a large variety of datasets and models, including MLP, CNN, MLP-Mixers, Transformers (Vision Transformers (ViT), Language Transformer (DistilBERT)) and show that our method can be applied to pre-trained models like GPT-2. The list of models and datasets are represented in the appendices D.1 and D.2, respectively.\n>  Specific questions about the online pruning process.[...].\n\nAppendix B.6 addresses these questions and explains how the pruning is conducted during training. Pruning occurs incrementally after a new computation of the marginal likelihood.\n\n**Novelty and Comparison:**\n> The novelty is somewhat limited, [...].\n\nWe address the novelty of our approach with the general response.\n> Differences in uncertainty [...] are marginal. The presentation of results could be improved [...].\n\nWe acknowledge that the difference in uncertainty estimation between SpaM and MAP could be more clearly visualized and will adjust the y-axis scale accordingly (due to random pruning having high values). The difference is indeed significant at high sparsities for criteria like OPD, GraSP, and magnitude (see Figure B8 and Table B3).\n>  Comparison with previous studies using Bayesian inference or variational methods for pruning..\n\nVariational inference for deep neural networks usually slows down training and reduces performance in comparison to the MAP. Further, as found by Blundell et al. (2015) in their seminal Bayes-by-Backprop paper, it does not work in conjunction with prior hyperparameter updates and thus cannot be applied in the automatic regularization setting we are interested in. We will include a more thorough discussion in the related work section.\n\n**The scale of Experiments:**\n>  The experiments are considered limited in scale,[...]. Larger models and datasets should be used for validation.\n\nWe cover a large variety of models and datasets (vision and text):\n\n**Models**: fully connected, LeNets, MLPMixer, (Wide) ResNets, ViTs, DistilBERT, GPT-2\n\n**Datasets**: UCI breast cancer, MNIST, FashionMNIST, CIFAR10, CIFAR100, IMDB\n>  [...]. The paper should position SpaM+OPD in relation to other pruning methodologies more clearly.\n\nBased on Figure 2 which compares SpaM and MAP across different criteria, \nSpaM creates models that are inherently more sparsifiable.\nOPD (blue lines) consistently ranks among the top-performing criteria, maintaining near-unpruned performance even at 95% sparsity in vision tasks. GPT-2 results also highlight OPD's standalone effectiveness without SpaM.\n\nTaken together, SpaM and OPD offer a powerful combination for achieving the ultimate pruning goal: high performance at high sparsity. \n\n\n**Questions**\n>  Previous Studies Comparison: [...] SpaM and previous studies using Bayesian inference or variational methods for pruning [...].\n\nA similar comparison would down-scale our evaluation pipeline, as most VI methods and those relying on full (approximate) Bayesian inference are limited and likely intractable for large-scale models. Such approaches are typically constrained to smaller architectures like MLPs and CNNs [1-3].\n\nMethods like MCMC are impractical for large-scale models due to high computational demands and numerous samples needed to estimate posterior distributions accurately. The Laplace approximation used in SpaM avoids sampling, making it computationally feasible.\n\n[1] Molchanov et al. (2017). Variational Dropout Sparsifies Deep Neural Networks. ICML.\n\n[2] Zhao et al. (2019). Variational Convolutional Neural Network Pruning. CVPR.\n\n[3] Bai et al. (2020). Efficient Variational Inference for Sparse Deep Learning with Theoretical Guarantee. NeurIPS.\n\n> [...] The paper should explicitly explain the sequence of dense training followed by prune-after-training, [...].\n\nWe provide a detailed description of the training approach in the main text and more in detail in appendices B.1 and D.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0bJ5OHmaGQ", "reviewer": "Reviewer_NVyT", "summary": "This paper proposes to sparsify a neural network using Bayesian principles by optimizing the marginal likelihood (SpaM). Specifically, the authors use weight/node/layer-wise Gaussian priors over the weights and learn the corresponding scales by maximizing the marginal likelihood during training using Laplace approximation. Compared with MAP, where a shared regularization is used, SpaM optimizes the prior scale for each weight/node/layer to regularize weights adaptively. Moreover, a new important score, Optimal Brain Damage (OBD), is proposed for pruning weights, by using the approximated posterior. The effectiveness of this method is demonstrated with extensive experiments on various datasets and model architectures.", "questions": "Is optimizing scales with ML-II using Laplace approximation better than doing a full (approximate) Bayesian inference over scales with common sparsity-inducing priors (e.g., horseshoe) using VI or SGHMC, in terms of computation, accuracy/uncertainty given a sparsity level, etc.?", "limitations": "The authors mentioned the limitation coming from the Laplace approximation.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is novel and well-written. \n\nAlthough sparsity-inducing priors have been widely used to sparsify deep neural networks, optimizing the hyper-parameters in the prior using ML-II with Laplace approximation is novel and seems to be promising compared with using a single fixed scale to shave weights.\n\nThe experiments are comprehensive, including results with different prior structures, pruning criteria, architectures (conv and transformer), and dataset domains (images and texts).", "weaknesses": "This paper is generally well written with extensive experiment, and I only have one following concern (I'm very happy to increase my score if this is addressed):\n\nHyper-parameters in the prior often can be tackled in two ways as a Bayesian: 1. optimize the hyper-parameters with ML-II; 2. conduct Bayesian inference on the hyper-parameters by having hyper-priors on them. \n\nThis paper focuses on the first approach, but the comparison with the second approach is missing. In fact, the second approach has been widely used [1-4] to sparsify deep neural nets; for example, the node/layer-wise horseshoe prior and spike-and-slab prior can also offer different regularization strengths on different weights as well as give a more structured sparsity with good theoretical guarantees. Moreover, ML-II is known to have the risk of overfitting compared with the full Bayesian approach. So, I believe it is important to compare ML-II with the full Bayesian inference.\n\n[1] Ghosh, Soumya, Jiayu Yao, and Finale Doshi-Velez. \"Model selection in Bayesian neural networks via horseshoe priors.\" Journal of Machine Learning Research 20.182 (2019): 1-46.\n\n[2] Louizos, Christos, Karen Ullrich, and Max Welling. \"Bayesian compression for deep learning.\" Advances in neural information processing systems 30 (2017).\n\n[3] Cui, Tianyu, et al. \"Informative Bayesian neural network priors for weak signals.\" Bayesian Analysis 17.4 (2022): 1121-1151.\n\n[4] Polson, Nicholas G., and Veronika Ročková. \"Posterior concentration for sparse deep learning.\" Advances in Neural Information Processing Systems 31 (2018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your evaluation of our work and your constructive comment that shows a good understanding of the field and your readiness to increase the score if your concerns and questions are adressed. We would like to further explain our motivation behind using Laplace approximations.\n\n**Weaknesses**: \n\n> Hyper-parameters in the prior often can be tackled in two ways as a Bayesian: 1. optimize the hyper-parameters with ML-II; 2. conduct Bayesian inference on the hyper-parameters by having hyper-priors on them.\n[...] ML-II is known to have the risk of overfitting compared with the full Bayesian approach. So, I believe it is important to compare ML-II with the full Bayesian inference.\n\nThank you for your comment. The main reason not to go for the full Bayesian inference approaches is their known computational complexity and lack of scalability for large-scale problems [1] which is emphasized in the suggested work (Ghosh et al., 2019; Louizos et al., 2017; Cui et al., 2022; Polson & Ročková, 2018). With a full Bayesian approach, it is not possible to provide a solution that can be scaled across the different architectures considered in our work (e.g., ViT and GPT-2). Utilizing ASDL [2] to compute the second-order information needed for the Laplace approximation, we make use of modern scalable algebra engines and avoid the repeated sampling and complex variational optimization that tends to slow down training and reduce performance. \n\nThis makes our method easier to use for practitioners, since it scales better with large datasets and complex models and supports different types of layers. For instance, it made it possible for us to use our method starting from a \"pre-trained\" backbone (e.g., from huggingface, without adjustments to the architecture) and transfer learn or fine-tune on another task. \n\n\n\n\n**Questions:**\n> Is optimizing scales with ML-II using Laplace approximation better than doing a full (approximate) Bayesian inference over scales with common sparsity-inducing priors (e.g., horseshoe) using VI or SGHMC, in terms of computation, accuracy/uncertainty given a sparsity level, etc.?\n\nYes, full (approximate) Bayesian inference methods are likely intractable for large-scale problems due to their computational demands and the complexity of approximating or sampling from high-dimensional posteriors, as laid out in our response above. \n\nIn our case, we wanted the method to be able to scale and not be restricted by the limitations that come with full Bayesian inference that would require downscaling the experiments and not being able to perform our approach on modern architectures like vision and language transformers. BNNs are usually challenging to implement and expensive to train. With the Laplace approximation, it is much easier to use BNNs [3] and, during inference, to use a single forward pass on the pruned architecture.\nThis approach allowed us to even be able to perform OPD on pre-trained models (by computing the inverse Hessian needed for Laplace post-hoc) that are very expensive to train, like GPT-2, and as well to use SpaM on architectures like DistilBERT and ViT.\n\n**References**\n\n[1] Bai, J., Song, Q., & Cheng, G. (2020). Efficient variational inference for sparse deep learning with theoretical guarantee. Advances in Neural Information Processing Systems, 33, 466-476.\n\n[2] Osawa, K., Ishikawa, S., Yokota, R., Li, S., & Hoefler, T. (2023). ASDL: A Unified Interface for Gradient Preconditioning in PyTorch. \n\n[3] Daxberger, E., Kristiadi, A., Immer, A., Eschenhagen, R., Bauer, M., & Hennig, P. (2021). Laplace Redux – Effortless Bayesian Deep Learning. NeurIPS.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bWGrtoJDwl", "reviewer": "Reviewer_bZUn", "summary": "This paper proposes to train neural networks that are more amenable to pruning, using a Bayesian approach. In particular, they specify separate priors over individual parameters rather than a shared global prior, which allows some parameters to be regularised more than others. They maximise a Laplace approximation to the marginal likelihood to optimise these priors, deriving an approximation that allows them to utilise richer block-diagonal KFAC precision matrices rather than simple diagonal ones. Furthermore, they propose a scoring criterion based on the posterior variance and magnitude of the weights for deciding which weights to prune. The authors found that their method for pretraining sparsifiable networks generally led to much better performance after pruning than traditional MAP training.", "questions": "### Questions\n- Do the authors believe that the disappointing performance of the KFAC with non-scalar priors was due to the extra approximation that was necessarily made (Proposition 3.1)?\n- In the MAP experiments, are we using the same Laplace approximation that was used to approximate the marginal log-likelihood, and if so, how are the prior hyperparameters chosen? Please clarify if I appear to be misunderstanding this aspect.\n- In lines 165-166, could you expand upon what is meant by \"optimize the Laplace approximation to the marginal likelihood after an initial burn-in phase with a certain frequency\"?\n- Does SpaM allow for improving structured sparsifiability **during training** as well as just unstructured? This wasn't entirely clear to me\n\n\n### Suggestions\n- Expand slightly more on the difference between uniform and global pruning.\n- Remove some of the less relevant figures / discussion and include more of the experimental plots in the main text.\n- Typo in Eq. 5. $\\mathbf Q^T$ should be $\\mathbf Q^T_A$?\n- In Eq. 5, it would be clear to write $(\\boldsymbol \\Lambda_A \\otimes \\boldsymbol \\Lambda_G + \\delta \\mathbf I)$ rather than $(\\boldsymbol \\Lambda_A \\otimes \\boldsymbol \\Lambda_G + \\delta)$ as the latter implies elementwise addition of $\\delta$ rather than just the diagonal. Please let me know if this is a misunderstanding on my part.\n- Many of the figure references in the paper didn't seem to lead to the right place, e.g. on line 239. This might just be a problem with my machine, please double check.", "limitations": "The authors briefly discussed some generic limitations of Laplace approximations and additional computational cost of their SpaM training procedure over MAP. It would have been nice to comment on, for example, the remaining difficulty of structured pruning, since it is so much more beneficial than unstructured pruning.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The proposed \"SpaM\" algorithm for pretraining sparsifiable networks in conjunction with their \"OPD\" criterion appears to be an effective method for pruning a variety of network types, based on the empirical results in the paper.\n- The experiments are extensive and relevant to the contributions of the paper\n- The approximation that allows KFAC to be used with non-scalar priors is useful to the KFAC literature in general, not just in the scope of network pruning.\n- The paper is well written and concepts are introduced in an intuitive fashion.", "weaknesses": "- The more complicated KFAC with non-scalar prior approximation does not appear to be empirically better than using a diagonal precision matrix in the Laplace approximation, where the use of non-scalar priors is much simpler. However, it is still an interesting and potentially useful approximation outside of this particular application.\n\n### Nitpicks\n\n- Some details may be unclear to people less familiar with the area, such as the \"interleaved\" training of the prior hyperparameters and network parameters, or what exactly is being optimised in the MAP comparisons.\n- Some of the more interesting figures (in my opinion) are relegated to the appendix, such as the fine-tuning figure, whilst the main paper has elements that feel superfluous or irrelevant, such as Figure 1, which seems unnecessary to explain the method, or a rather lengthy discussion of the use of marginal-likelihood in e.g. section 3.2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you very much for your detailed and helpful feedback. We really appreciate your suggestions and will incorporate them in the revision.\n\n**Weaknesses:**\n> The more complicated KFAC with non-scalar prior approximation does not appear to be empirically [...] However, it is still an interesting and potentially useful approximation outside of this particular application.\n\nThank you for your comment and raising this point. You understood our motivation with this proposal. We would like to expand further by adressing your question below. \n\n**Nitpicks**\n> Some details [...] \"interleaved\" training of the prior hyperparameters and network parameters, or what exactly is being optimised in the MAP comparisons.\n\nThe optimization is conducted after a burn-in phase (number of initial epochs) and at a specific frequency (after each x epochs) referred to as `marglik_frequency` in the hyperparameter sections.\nIn is worth mentioning that after pruning (at test time), we just use the trained posterior mean (MAP solution under the optimized prior) and not the full Laplace predictive.\n\n> Some of the more interesting figures (in my opinion) are relegated to the appendix, [...] such as Figure 1, which seems unnecessary to explain the method, or a rather lengthy discussion of the use of marginal-likelihood in e.g. section 3.2.\n\nWe thank you for the suggestions, it was challenging to fit all key results in the main body of the paper.\nWe will utilize the space better in the revised version. \n\n**Questions:**\n>  Do the authors believe that the disappointing performance of the KFAC with non-scalar priors was due to the extra approximation that was necessarily made (Proposition 3.1)?\n  \nProposition 3.1 came into place to bridge the gap and to emphasize the contribution of diagonal priors in making neural networks more prunable, as it was shown to perform exceptionally well when combining the diagonal approximation with a diagonal prior. It was shown that a diagonal prior in KFAC pushes the probability of the network compared to other priors like scalar and layerwise (the approximation results are referred to as parameter-wise in red in Figure B4, compared to the KFAC with solid lines).\nFor our work, we do recommend the usage of the diagonal approximation with diagonal priors as we do not need the precision of KFAC compared to diagonal in offering a better approximation. However, other works, as you mentioned, might benefit from it all while pushing the probability of networks.\n\n> In the MAP experiments, are we using the same Laplace approximation that [...] how are the prior hyperparameters chosen? Please clarify if I appear to be misunderstanding this aspect.\n\nFor MAP, we use a standard training approach where we do not optimize the marginal log-likelihood, and thus generally do not use Laplace. In the case of using OPD with MAP, the inverse of the Hessian is computed to be able to perform Laplace post-hoc on a pre-trained network without the need for the network to be trained and optimized for the marginal likelihood.\n\n\n>[...] \"optimize the Laplace approximation to the marginal likelihood after an initial burn-in phase with a certain frequency\"?\n\nThank you for your insightful question. We will include an explanation in the revision or refer to appendix D.4. The approximation is conducted during training after an initial number of epochs (burn-in phase) and at a specific frequency (e.g., every 5 epochs) for computational considerations. For smaller architectures, the approximation can be used from the start of training and at each epoch, but for larger architectures, this would be challenging and require more ressources. \nThe table D.4 shows the choice of the parameters for each experiment and explains their significance.\n\n\n> Does SpaM allow for improving structured sparsifiability during training as well as just unstructured? This wasn't entirely clear to me.\n\nSpaM can be applied during training for the structured case, but it is more efficient to mask structures (zero them out) instead of compressing them (removing them) during training. Compression during training is inefficient due to the need to replace layers and copy weights. By masking structures, we can evaluate the smaller network before fully committing to removing any components, and this does not add much overhead, as we have everything pre-computed with each marginal likelihood update and only need to aggregate the score. After training, compression can be performed once.\n\n\n>  Expand slightly more on the difference between uniform and global pruning.\n\nThanks, we will expand on this in the revised version.\n\n**Global pruning** prunes across the entire network at a target percentage (e.g., 80%). This approach may result in some layers being pruned more heavily than others to achieve the overall target sparsity. \n\n**Uniform pruning**, on the other hand, applies the same pruning percentage to each layer.\n\n> Remove some of the less relevant figures/discussion and include more of the experimental plots in the main text.\n\nThank you, we will adress your suggestion in the revised version.\n\n> Typo in Eq. 5. should be $\\mathbf{Q}^T_A$?\n\nThank you for pointing this out, that's correct and we will add the subscript $A$.\n\n>  In in Eq. 5, it would be clear to write [...] Please let me know if this is a misunderstanding on my part.\n\nThank you for the suggestion. We will add the identity for clarity.\n\n>  Many of the figure references in the paper didn't seem to lead to the right place, [...] please double check.\n\nThank you. Indeed, some links to the appendix figures seem to be broken, where the reference is rendered correctly, but the hyperlinks seem to be broken due to a counter change. We solved this issue by setting `hypertextnames` to `false` in the `hyperref` package.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for the constructive review and feedback. We appreciate the opportunity to clarify and elaborate on the decisions made in our work. One shared comment across two reviews was the choice of the Laplace approximation and the use of the Gaussian distribution.

**Choice of Laplace Approximation and Gaussian Distributions**
The primary reason for utilizing the Laplace approximation and structured Gaussian priors in our framework stems from the need for computational efficiency and scalability. While the marginal likelihood provides a powerful tool for balancing model complexity and data fit, implementing this in a practical and scalable manner is non-trivial and indeed, the combination of Gaussian distributions with Laplace approximations are the only example in the literature known to us for which this works (Immer et al., 2021). For mean-field variational inference, for example, Blundell et al. (2015) found it to not work.

**Computational Feasibility**
Applying more complex priors such as the horseshoe or Indian buffet process priors, as suggested by some reviewers, introduces significant computational overhead. These priors often require sophisticated inference techniques, such as Markov Chain Monte Carlo (MCMC) or variational inference, which are computationally expensive and challenging to scale to high-dimensional, modern architectures. For instance, methods like those proposed by Ghosh et al. (2018) and Kessler et al. (2021) typically focus on much smaller models where computational resources are not as limiting. In contrast, our work aims to provide a scalable solution applicable to large-scale networks commonly used in practice without the need for extensive architectural changes, ensuring support for real-world use cases. This scalability is made possible thanks to the Laplace approximation (Daxberger et al., 2021), which approximates the model's posterior with a Gaussian distribution in a computationally efficient manner, leveraging second-order information of the loss landscape. As we show empirically, this approximation is good enough to yield tangible benefits.

**References**

Immer, A., Bauer, M., Fortuin, V., Rätsch, G., & Khan, M.E. (2021). Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning. ICML.

Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015). Weight uncertainty in neural network. ICML.

Ghosh, S., Yao, J., & Doshi-Velez, F. (2018). Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors. ICML.

Kessler, D., Aicher, C., & Fox, E. B. (2021). Indian Buffet Process Priors for Bayesian Neural Networks. NeurIPS.

Daxberger, E., Kristiadi, A., Immer, A., Eschenhagen, R., Bauer, M., & Hennig, P. (2021). Laplace Redux – Effortless Bayesian Deep Learning. NeurIPS.

**Adressing specific review**
To address all comments, one review in particular seems to misunderstand a few key aspects of our work, thus we would like to clarify them further.
> Novelty

We show that automatic relevance determination, which leads to a prunable neural network, is possible for deep learning models for the first time. Our method is scalable and leverages Bayesian properties to enhance the prunability of neural networks and generalizes to different architectures. Our approach includes a new provably optimal approximation for diagonal priors in the KFAC eigenbasis, confirming that diagonal priors improve prunability. We introduce a one-shot pruning approach that eliminates the need for parameter-linked pruning hyperparameters but automatically learns them, and OPD, a cost-effective pruning criterion akin to the popular OBD. OPD performs on par or better than many other criteria in practice. This method can also be applied to pre-trained networks, as demonstrated for GPT-2 on IMDB (detailed in appendix B.7.3).
> The baseline methods used are too simplistic. Baselines such as IMP, RigL, or DST should be included.

The criteria used are not baselines, as our claim is that SpaM works with any pruning criterion to achieve high sparsifiability. The only real baseline is MAP, compared directly with SpaM. We argue that SpaM could be combined with all modern pruning criteria, which is beyond our work's scope.
Our primary focus is to present a practical sparsification pipeline that aligns with various architectures with minimal hyperparameters. Methods like IMP, involving iterative retraining and weight resetting, can be computationally expensive and sensitive to initialization [1]. Even with multiple iterations of train, prune, and retrain, our results outperform theirs, as seen in Figure 14 for ResNet-18 on CIFAR-10.

DST and RigL require hyperparameter tuning for each architecture, adding to the computational cost and being impractical for pre-trained networks. RigL requires tuning sparsity level, update interval, growth method, pruning ratio, and initial sparsity, while DST involves similar hyperparameters and added growth allocation and redistribution methods(some criteria used: magnitude, gradient contribution).

In contrast, SpaM uses a fully automatic one-shot pruning approach, avoiding costly retraining. Our experiments include well-established criteria like magnitude and random pruning, and strong criteria like SNIP, SynFlow, and GraSP. Notably, GraSP, as in our work, has shown high pruning performance on various benchmarks [2,3]. These comparisons showcase our method's effectiveness in a practical setting, demonstrating that SpaM-trained networks perform better than those trained using MAP at extreme sparsities.

[1] Frankle & Carbin (2019). "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks." International Conference on Learning Representations (ICLR).

[2] Rachwan et al. (2022) . "Winning the Lottery Ahead of Time: Efficient Early Network Pruning." ICML.

[3] Lubana & Dick (2021). "A Gradient Flow Framework for Analyzing Network Pruning." International Conference on Learning Representations (ICLR).

Meta-review: This paper introduces a framework called Sparsifiability via the Marginal likelihood (SpaM) that leverages Bayesian marginal likelihood with sparsity-inducing priors to enhance the sparsification of neural networks. All reviewers praised the originality and effectiveness of Bayesian marginal likelihood as a pruning criterion, and most reviewers also appreciated the thoroughness of the experiments performed. Some reviewers questioned the choice of Laplace approximation and Gaussian approximation and asked for comparison with previous works featuring other sparsity-inducing priors like horseshoe priors; the authors mainly defended from the perspective of computational feasibility and the fact that some of the prior works require more invasive changes to the architectures/training protocols, which largely addressed the reviewers' concerns. On this point, while the AC also does not see this as a major issue post-rebuttal and acknowledges the computational concerns, the AC still thinks that some empirical comparison against these prior works, even on smaller scale setups where these baselines are indeed applicable, could still be helpful to strengthen the arguments in the paper. 

Other concerns were raised on novelty (although the AC does not view this as a major weakness given that all reviewers acknowledged the originality and effectiveness) and comparison against approaches with type-II MLE, but the authors have answered the questions satisfactorily.

Post-rebuttal, all reviewers unanimously recommend acceptance and the AC concurs with this recommendation. The authors are encouraged to incorporate all requested changes and discussions into the camera-ready version of the paper.
Decision: Accept (poster)