BMRS: Bayesian Model Reduction for Structured Pruning
Authors: Dustin Wright
Christian Igel
Raghavendra Selvan
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Bayesian model reduction
structured pruning
variational inference
efficient machine learning
deep learning
Primary area: probabilistic_methods

Abstract: Modern neural networks are often massively overparameterized leading to high compute costs during training and at inference. One effective method to improve both the compute and energy efficiency of neural networks while maintaining good performance is structured pruning, where full network structures (e.g. neurons or convolutional filters) that have limited impact on the model output are removed. In this work, we propose Bayesian Model Reduction for Structured pruning (BMRS), a fully end-to-end Bayesian method of structured pruning. BMRS is based on two recent methods: Bayesian structured pruning with multiplicative noise, and Bayesian model reduction (BMR), a method which allows efficient comparison of Bayesian models under a change in prior. We present two realizations of BMRS derived from different priors which yield different structured pruning characteristics:  1) BMRS_N with the truncated log-normal prior, which offers reliable compression rates and accuracy without the need for tuning any thresholds and 2) BMRS_U with the truncated log-uniform prior that can achieve more aggressive compression based on the boundaries of truncation. Overall, we find that BMRS offers a theoretically grounded approach to structured pruning of neural networks yielding both high compression rates and accuracy. Experiments on multiple datasets and neural networks of varying complexity showed that the two BMRS methods offer a competitive performance-efficiency trade-off compared to other pruning methods.

BMRS: Bayesian Model Reduction for Structured Pruning
Dustin Wright, Christian Igel, and Raghavendra Selvan Department of Computer Science, University of Copenhagen {av, rew,igg,,rrggar raghav) ku.dk
Abstract
Modern neural networks are often massively overparameterized leading to high compute costs during training and at inference, One effective method to improve both the compute and energy efficiency of neural networks while maintaining good performance is structured pruning, where full network structures (e.g. neurons convolutional filters) that have limited impact on the model output are removed. this work, propose Bayesian Model Reduction for Structured pruning (BMRS), a fully end-to-end Bayesian method structured pruning. BMRS is based on two recent methods: Bayesian structured pruning with multiplicative noise, and Bayesian model reduction (BMR), a method which allows efficient comparison Bayesian models under change in prior. We present two realizations of BMRS de- rived from different priors which yield different structured pruning characteristics: 1) BMRS with the truncated log-normal prior, which offers reliable compression rates and accuracy without the need for tuning any thresholds and 2) BMRS with the truncated log-unin uniformf prior that achieve more aggressive compres- sion based on the boundaries of truncation. Overall, we find that BMRS offers a theoretically grounded approach structured pruning of neural networks yielding both high compression rates and accuracy. Experiments on multiple datasets and neural networks of varying complexity showed that the two BMRS methods offer a competitive cofoommanc--fffciinne trade-off compared to other pruning methods
Introduction
Modern neural networks come with increasing computational burden, scale is often to be associated with performance [34]. The response to this has been a focus on research around the topic of neural network efficiency [] where the goal is reduce the computational cost of system while maintaining other desirable metrics. As such, selecting a method to improve efficiency comes with many tradeoffs, including how to balance compute and energy consumption with accuracy [35..
Neural network pruning seeks do this by removing parts of network which have limited impact on its output. This comes in two primary forms: unstructured pruning, where individual weights are removed, and structured pruning, where entire neural network structures such as neurons and convolutional filters are removed [27]. Structured pruning is often desirable unstructured pruning can result in sparse computations which are energy intensive on current hardware, while structured pruning can maintain more energy efficient dense operations [14] 31]. Many ways to perform structured pruning have been proposed, but the challenge how to appropriately balance accuracy and complexity in principled manner has remained.
In this work, we address this challenge by proposing BMRS: Bayesian Model Reduction for Structured pruning. BMRS is a principled method based on combining two complementary lines
wtt.:/u...ooo//aaaaaa/aaaeaaaaae Source code: http://shww...ooo////a///aaaaass
pruning/
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

E(0
Original
ElO| BMRS
20
Calculate AF
4
Reduced Prior
Prune AF
Count
Count
2
10
0
0.5 [][0]]
0.5
0.5 [1/]
Figure I: BMRS uses BMR to perform structured pruning under multiplicative noise by calculating the change in log-evidence of noise variables 9 under prior which would shrink them to O.
of work: Bayesian structured pruning with multiplicative noise 30] and Bayesian model reduction (BMR) [6]. method of efficient Bayesian model comparison under change in prior. Multiplicative noise allows one to flexibly induce sparsity at any structural level without the need use computa- tionally complex sndke-and-slab priors [6 (9). while BMR enables principled pruning rules without the need for task-specific threshold tuning. Starting with the approach from [30], we derive two versions of BMRS using different priors which offer their own benefits. BMRS is based on the truncated log-normal prior and has the benefit of achieving a high compression rate without needing to tune threshold for compression, while BMRS offers tunable compression by controlling the allowable precision of noise variables in the network. In sum, our contributions are:
BMRS: method for Bayesian structured pruning based multiplicative noise and Bayesian model reduction;
Derivations of pruning algorithms for two priors with theoretical motivation;
Empirical results range of neural networks and datasets demonstrating high compression rates without any threshold tuning, with more extreme compression achievable via a parameter controlling allowable precision.
2 Related work
The primary goal of neural network pruning is to determine the elements of network which can be removed with minimal impact on the output. Idcally, a pruning method ranks all elements in the order in which they be removed and provides criterion for truncating the resulting ordered list. Since the early works on gradient based methods for pruning 23 [2], the literature around neural network pruning has expanded rapidly, with the main lines of work exploring pruning individual weights (unstructured pruning) and pruning full network structures (structured pruning). For recent survey, (27]. The closest related works to ours those pruning methods which perform Bayesian pruning [30 28 10 16, 26). and those which use Bayesian model reduction to determine what elements remove from neural network [5 29
Bayesian pruning. Bayesian structured pruning was first explored in Kingma et al. 20 where the authors demonstrate that dropout has a Bayesian interpretation as multiplicative noise with sparsity inducing prior. The studies of (30 28 10 follow this work by explicitly modeling the
random noise in dropout with different priors, 330 using a truncated log-uniform prior and 22 TO
using horseshoe priors. Following this, the works of [5 16
33
29 26 have explored pruning
of Bayesian neural networks (BNNs) with spike-and -adee--nd--aab priors to induce both weight sparsity and group sparsity with flat and bierarchical priors, respectively. [16] demonstrate that thresholdless pruning achievable by placing an explicit spike-and-slat prior on the nodes of a BNN induce group sparsity. However, this setup requires complex and carefully constructed posteriors due to the discrete nature of spike-and-slab distributions and thus computationally inefficient 29 16
Bayesian model reduction. Bayesian model reduction, discussed in detail in $3.2 is an efficient method Bayesian model comparison which allows for analytic solutions for the model evidence under change in priors. BMR has found application across multiple scientific disciplines 9 8 [8]
and has recently been used as a method for neural network pruning (2) [.. More specifically, [ demonstrate the benefits of basebased pruning for the case of a BNN with Gaussian prior

---

the weights, and [29] demonstrate the utility of BMR for unstructured pruning of BNNs with priors inducing both weight and group sparsity.
3 Problem formulation
3.1 Structured pruning with multiplicative noise and variational inference
We approach the problem of structured pruning using sparsity inducing multiplicative noise as described [30]. In this setting, we have a dataset consisting of N i.i.d. input-output pairs, D [(xx, Mi) Vj ,,...,,. N... We consider parametric model, here a deep neural network, that maps the input data x, to their output yy using the trainable parameters W giving rise to the likelihood function PDD|O, W) = I[ P(U x, θ W). In addition to the trainable weights, W the model consists of the sparsity inducing multiplicative noise given by the random variable, θ, with prior (()). This in contrast to BNNs where the weights are random variables but aligns with the setting when using multiplicative noise for Bayesian pruning 30
The effect of the multiplicative noise 0g E O for structural element in neural network with index i, parameters w,, and input h is given as
h. 8 (w.h,-1)
0, Pa~ple p(0))
(1)
We note that w, could be the parameters of any structural element in the network, for example, single neuron or an entire convolutional filter. Given this, we would like to learn the maximum likelihood estimate (MLE) w; of the weights as well the posterior distribution over the multiplicative noise, p(O:|D, w.). when using a sparsity inducing prior p(0:) such that 0: favors values closer to 0.
Following (3). the neural network weights are learned via gradient descent as in standard deep learn- ing model optimization. The posterior distribution, p(0)D, w.) = p(D (,, w. ppp(()pp(())) however, is intractable. We resort variational approximation from tractable family of approximating distributions, (o( parameterized by & (for the sake brevity we do not indicate the dependence on w; and omit the subscript i). The parameters & obtained by optimizing the following objective w.r.t. 0:
g[ DKL (O)l(((()))) (olla,(0))l((( - B P(y X,,, 0 w)]
This is the commonly used variational free energy (VFE) or negative evidence lower bound (ELBO) 4.. Here - denotes equality up to positive constant.
The expectation Ego! [-] is approximated by a Monte Carlo estimator acting on minibatch samples from D. and reparameterization allows to backpropagate gradients through stochastic variables [19 28)
Under this reparameterization, the variational distribution, (0). becomes a deterministic function of the non- -on-mmemeteic noise ~ p(e) and the VFE is calculated as
Fíp. ql Dx. (0)ll(())
Σ (ED
log togp(u; (u, 0 = c):
(3)
where / is a function that allows us sample 0 via deterministic parameters ø and the non-parametric stochastic variable, e. Optimization of Equutton 3 Jlllows us to jointly learn w and 0. The particular choice of priors and the approximating distributions to induce sparsity are discussed in $4.1
3.2 Bayesian model reduction
Bayesian model reduction (BMR) allows one efficiently compute the change in VFE (Equation 2 under a change in prior without the need to re-estimate model parameters. To perform pruning, one start out by selecting broad prior for the original model estimation and then pick a narrower prior (i.e. reduced prior) with the density concentrated around O. Then, BMR can be used to determine the VFE is greater under the reduced model, and prune those parameters for which this condition holds. We briefly describe how this is achieved in the general case, followed by the specific realization for BMRS in $4.2 for further details see [
Consider the likelihood function p (D))) and a prior p(0) on the variable We can introduce a new
prior (0) which shares the same likelihood as the original model (i.e. p( (D)))) (0) p(D(0)) and get:
p(D ((O(D)p(D)
((0
(DD)
((D) p D ((0)
((0D)
(4)
p(0)
(())
p(0)
3

---

By marginalizing over A and taking the log, we obtain the difference in log evidence as:
log P(D) (lg) = log p(0)D) p(0) d0 so log / (0) p(0) d0 log Es
qs (0)
(5)
p(0)
p(0)
p(0)
More concisely, we call the change in log evidence AF and thus have:
(o)
AF - log (D)) - (D) Re log E,
(6)
p(0)
If the new prior, p(0), is selected so that 0 would removed, pruning can be performed when AF 2 0. Additionally, when the type of distributions between p(0). p(0). and (0) are the same or similar (e.g. Gaussian), AF can be calculated efficiently in closed form (see [7]).
We presented BMR for general likelihood function p(D 0); it holds analogously for the likelihood function (D)0 W) introduced with the multiplicative noise described in $3.1
4 Bayesian model reduction for structured pruning (BMRS)
Our goal is to derive principled structured pruning algorithm starting from the general formulation in 3 which can automatically determine which structures to prune. BMRS accomplishes this by following the multiplicative noise setup in [30] with BMR used on the noise terms. Figure Iillustrates the general approach, where AF is calculated for a model trained with multiplicative noise under a reduced prior, and elements of the model are removed if the new VFE is greater. We next describe the multiplicative noise layer trained using Equation and then derive two variants of BMRS from Equation usinn different reduced priors.
4.1 Multiplicative noise layer
The concept of multiplicative noise inducing sparsity in neural networks was first introduced with variational dropout, where [20] show that dropout has Bayesian interpretation as multiplicative noise with a log-uniform prior. One can use this interpretation of dropout in order to explicitly learn dropout parameters, 0g, as in $3, by selecting appropriate prior and variational distributions and optimizing Equation 2) direecly.. 30 propose to do so by using the truncated log-uni uniform distribution as a prior and the truncated log- tog-normal distribution the variational distribution. As such, the variational approximation can be performed using
h; = 0; (w;h;-1); (o,),) = LogN (o.M (Ollss, p(0,) LogU |0 (0.)
(7)
with bounded support between and and < a < I 1. We refer to 30 for details on how to learn 4s. which is obtained by optimizing Equation 3
The log-uniform distribution serves as sparsity inducing prior as most of its density is concentrated around 0 (see panel 2 in Figuue 1}. Additionally, it acts as regularizer on the floating point precision of the multiplicative noise terms (20) In [30] this is used to perform structured pruning by removing all structures h; where the signal-to-noise ratio of the noise term 0, falls below a pre-defined threshold. We next show how to derive principled pruning algorithms based on BMR which induce sparsity while maintaining accuracy without the need for tuning pruning thresholds.
4.2 Deriving BMRS
Our goal to use BMR in order to perform structured pruning of models trained with multiplicative noise. To do so, we must select new prior p(0) from which we can: 1) induce sparsity; 2) efficiently calculate AF and; 3) prune the network while maintaining good performance.
Selecting the reduced prior, p(0). is straightforward when the prior and approximate posterior are the same type of distributions. For example, in fully BNN where assumes a prior distribution of NV 0, I) over all the model weights with a mean-field variational approximation resulting in a factorisation over individual weights, N o²), the three criteria above can be met when one selects a Gaussian reduced prior with slight variance around 0 i.c. N (0 <),e sJ 0 However, in the case of
It can be shown that AF is calculable efficiently in closed form for this setup: see e.g.

---

multiplicative noise, our prior and variational distributions are of different types and thus the selection of the reduced prior is not immediately obvious. Here, we derive and compare the characteristics of two different reduced priors: one based on truncated log-normal distribution, which we can use to approximate Dirac delta at O (BMRSx), and one based on truncated log-uniform distribution with reduced support (BMRS;).
4.2.1 BMRS with log-normal reduced prior (BMRS,)
First, we derive AF when using the log-uniform distribution as the original prior, p(0) (o.)),,)))) and the truncated log-normal distribution as the reduced prior, p(0) [a.6 7. (p)) We select truncated log-normal distribution, it matches the variational distribution go (0), and the log-uniform prior, because it is special case of the log-normal distribution when the variance goes to infinity. Because of this, we expect that AF will have a closed form solution, and that the computation will be efficient. We briefly present the results of the derivation here; for the full derivation see A.1
We can use the specific forms of p(0) and p(0) for the truncated cog-unnformm and truncated log-normal distributions, respectively, in Equation to determine AF:
(x()) ((0)
log (log b log a)
2 2 0
1
AF log Esp
log
(8)
Zg.Za
2
0 p
with 82 =
1
+
and B4g = =
Up 5
Here, Zp = k(pp) - =((a): +(() =
53
99
b + erf( [/2]] is the CDF of the standard Normal distribution, L // 1); Op = (a - pp)
and Bp = ((b--pp) Op)/opp
As can be seen from Equation 8 the calculation for AF can be performed directly using only the statistics the priors and variational distribution. order for Equation too induce sparsity, we must select uo and og that effectively collapse 0 to O. To achieve this, we can approximate a Dirac delta at 0 by selecting jip to be close to 0 (e.g., the lower bound of truncation a), and o% to be sufficiently small. We will demonstrate in $5ttha this reduced prior results in high sparsity while maintaining performance without any need for tuning pruning thresholds.
4.2.2 BMRS with log-uniform reduced prior (BMRS,)
Next, we derive the change in VFE, AF, when using a truncated log-un log-uniform distribution as the original prior, p(f) LogU b (0), and truncated log- ong-uniform distribution with reduced support as the reduced prior, p(0) LogU (o' .1 (0). We select a reduced truncated log-uniform distribution for the same reasons as the truncated log-normal: we expect that A/ will have an efficiently calculable closed form, given that the priors are of the same type and are a special case of the variational distribution. The PDF of the reduced truncated log uniform distribution given as follows:
-00-((
{{
(Olog log #/)
a a' SOSOvs <b otherwise
p0)) LogU ja (0) 10=10
(9)
Using this, we can directly solve the integral under the expectation given in Equation 6 for AF (full details S A.2h:
(0)
log b B a LogU (a,)( log
(0)
exp OF R E,
= LogU, lar (0)
d0
Co < 0, s 5')
(10)
p(0)
where qa < 0g < B') is the CDF of the variational distribution evaluated between a' and b. We know from Equation 5 that the VFE under p(()) is greater when exp AF 1. Plugging this in:
log log & la < < 6*)
log B/ a < q (a' S 0 < 6')
1 15
(11)
log
b
Here, the left hand side of the inequality is the CDF of the truncated log-uniform distribution between a' and b'. In other words, when the new prior is log- log-uniform distribution with reduced support,
5

---

Algorithm 1: Training and pruning with BMRS
input: D; neural network with deterministic weights W and variational parameters di original prior p()); reduced prior p()); number of training epochs ey; number of fine-tuning epochs ep; number of pruning epochs P
j -
while < er and W, & not converged do
Train @ and W on DD using/Eonn Equation 3
if mod P
then forall 0g in @ do
dF + A (F(((()), ((0,), (0.))
if dF 2 0 then
Φ + Φ \ o;
W W W.
j j++1
Fine tune W and á on DD for eF epochs
the BMR pruning criterion amounts comparison between the CDF of the original prior and the variational distribution along the interval [a', 57. Additionally, [Equation 11 shows that this is generalizable to any variational distribution with support broader than the reduced prior.
BMRS pruning and connection to floating point precision. To see how BMRS, can he used for pruning, we first briefly summarize the relationship between the log-uniform distribution and floating point numbers. Floating point numbers are commonly encoded in binary as a combination of sign bit s, set of exponent bits e, and set of mantissa bits m denoting the fractional part of real-number:
(m/2p-1)- 1)- 2", where p determines the precision of the encoding. As discussed in [IT], the mantissae of "naturally observed" floating point numbers (e.g., natural constants) tend to follow a log-uniform distribution and repeated mulipplictiions/ddviiiio on a digital computer transform broad class of distributions towards a log-uniform distribution
(ollog B)= 1/B S S 1,
(12)
where B is the base of the number system. In the case where the mantissa uses p bits, there are 2 representable fractional numbers so B 2P. As such, p determines the smallest fractional value which can be represented. We can use this to have the reduced prior cover finite range of high precision values which are acceptable to prune. To accomplish this, we use a reduced log-uniform prior of the following form by selecting two integers P1. P2 where 0 S P1 < P2i
(0 log 2FT
2
1/2 S 11/1/ otherwise
((0)=
(13)
0.
Thus we can reduce the prior to range of precision between P1 and pa by selecting a' 1/22= and 6' = 1/2P:. Equuaioon 1] then has a natural interpretation comparing the probability of drawing random mantissa from the interval [1/2P², 1/2P1] to the probability of drawing a value within that interval from the variational distribution. If we select p2 to be the limit of the precision of values in the number system used (p= = 23 for dangee-pintt precision), then we can interpret p1 as determining the prunable range of precision of all variables e. The accuracy-complest
tradeoff inherent in the selection of p(0) is then controlled through Pl. the desired cutoff of the precision of the network.
4.3 Training and pruning
The details on how to train network and use BMRS for pruning are given in Algorithm | We train a model for a fixed number epochs (or until convergence) and perform pruning every P epoccs. This lends itself to either post-training pruning, where the network is fully trained followed by pruning and fine-tuning, or continuous pruning, where pruning performed during model training. In our experiments, explore both of these setups and contrast BMRS with alternative pruning methods.

---

5 Experiments
We demonstrate the pruning behavior of BMRS through several experiments with neural networks of varying complexity measured the number of trainable parameters. We use the following datasets (full details in Appendix B): MNIST (22, Fashion-MNIST (37. CIFARIO (2I], and
Tinylmagenet. For MNIST, Fashion-MNIST, and CIFARIO we experiment with both multi-layer perceptron (MLP) and small CNN (LenetS (24). Pruning layers are applied after each fully connected layer for the MI.P, and for each convolutional filter and fully connected layer for Lenet5. For CIFAR 10 and Tinylmagenet, we further experiment with pretrained ResNet-50 [13 and pretrained vision transformer (ViT) [36]. For ResNet-50, we apply pruning layers after each layer of batch normalization, and for ViT we apply pruning layers to the output of each transformer block. For the multiplicative noise layers, we set the left bound of truncation to be log a = 20 and the right bound of truncation to be log b = O. Hyperparameters for the MLPs and Lenet5 are tuned on model with no pruning performed and kept the same for each variant (see Appendix Cp. We perform experiments using the following model variants and baselines which cover both Bayesian pruning criteria (e.g. ours, SNR, and E. (8) as well as magnitude pruning (L22):
None: baseline with no compression and no multiplicative noise.
L2: A magnitude pruning baseline based on the L2 Norm of weight vectors (matrices in the case of convolutional filters) at the input of each neuron to be pruned [25]. We set the pruning threshold to the compression rate achieved by BMRS, using the same settings of given experiment.
Egg The expected value of noise variables 0. For continuous pruning, we use a set threshold of 0.1.
SNR: The signal-to-non ratio Eng [0]/ Var [0] as used in 30 For continuous pruning, we use a set threshold of 1 as recommended in (30]
X BMRS using the log-normal prior from Equation 6 In order to reduce the prior to 0, we
set uu to the left bound of truncation (a), and oz 10
BMRSu-P1: BMRS using the log- log-uniform prior from Equation 11 In our experiments, set a' to be the limit of the precision of single-poing point floats (p2 = 23 so a = 1/22h) and b' to either L
8-bit precision (t' 1/2*) or p1 4-bit (b' = 1/2*).
Post-training pruning. We first look at the behavior of BMRS when used in the post-training setting. To do so, we first train a model on a given dataset, then use each method to rank the neurons based on their pruning function (L2 norm, signal-to-noiss different compression rates, neurons are progressively removed based on their rank, and the model fine-tuned for one epoch before measuring the test accuracy. For BMRS methods, we additionally stop pruning once AF < for a given structure. The plots of accuracy vs. compression for 10 different random seeds are given in Figure 2 (Further experiments given in Appendix E).
ratio, or AF). To observe the accuracy at
First, we find that BMRS generally stops compressing near the knee point of the trade-off curve- a preferred solution of Pareto front if there is no a priori preferences - in all settings except for BMRS which only does so in 4 out of settings. Notably, BMRS,, accomplishes this with no need to tune additional thresholds as is common in pruning literature. To further visualize this, the right plot in each subfigure shows scatter plot of the accuracy at the maximum compression rate (pruning all neurons where AF > 0) along with the curve of accuracy vs. compression for SNR pruning near the knee point. We can see that the density of points for BMRS concentrated near the optimal point in all cases except for the MI. on MNIST, indicating the robustness of the proposed methods.
We additionally observe much similarity in the curves for BMRS and SNR pruning, suggesting that they may be performing similar functions. To further investigate this, we look the Spearman rank correlation coefficient [32 of the neurons based on their respective functions in FFiuure 3 (plots for additional datasets in Appendix E}. We see that BMRS tends to have high correlation with SNR, suggesting that it learns qualitatively similar function with the benefit of providing threshold for compression. B on the other hand, tends to have very low or even negative correlation. This, combined with the more rapidly declining accuracy for given level of compression, suggests that BMRS is only apt for determining a single split into elements to keep and remove, but does not provide accurate ranking of the elements..
7

---

0.5
0.6
0.50
0.4
Accuracy
Accurscy
00.0
Accuracy
0.45
L2
L2
SMR
SNR
0.40
0.2
Etheta
0.4
SNR
0.2
B.RS_U.8
B
BNRS.U.
0.35
0.3
BM
0.1
BNRS_
BMRS_U4
50
100
55
60
65
70
75
50
100
85
Comn pressisn
Compression %
Compression
Compression
(a) CIFARIO Lenet5
(b) CIFARIO MLP
0.8
0.8
0.8
Accurac
0.6
Accuracy
0.6
Accurscy
0.6
Accuracy
0
L2
SMR
0.4
Ethota
SNR
0.2
BHRS_N
BHRS
0.2
BMRS U-8
0.0
BHRS U.4
BMRS.U
BHRS U-4
BMRS.U
50
100
0.0
B5
50
95
50
100
0.0
90
95
100
Compression
Compression %
Compression
Compression %
(c) Fashion-MNIST Lenet5
(d) Fashion-MNNSY MLP
1.0
1.0
1.05
0.8
Accuragg
Accuracy
Accuracy
1.00
0.6
0.6
0.4
0.95
BHRS
BNRS
BHRSS 1--8
0.2
BMRS U-B
0.90
0.0
BMRS U-4
aMis
0.0
BHNS U-4
U-4
50
100
0.0
80
85
90
95
50
100
60
Compression
Com Com pression 9
Compression
Compression %
(c) MNIST Lenet5
(f) MNIST MLP
Figure 2: Accuracy vs. compression for post-training pruning on CIFARIO, MaSHIon-MNIST, and MNIST. The left plot in each subfigure shows the average accuracy across 10 seeds, shading shows the standard deviation. For BMRS, we mark the maximum compression rate based on when AF 0. The right plot in each subfigure shows scatter plot and kernel density estimation of accuracy vs. compression of BMRS compared to SNR accuracy. BMRS and BMRS consistently stop pruning near the knee point, preferred trade-off solution.
SKR
2,00
0.27
SNR
2:27
2.24
6:2
-041
69
Compression %
L2
6m
921
1:17
6:07
0.16
Accuracy
BMRS,N
0.65
3.03
0.32
BMRS
0.71
3:2
-0.12
40
SMRS_U
67
BMRS U-4
827
0.27
8.21
2.22
BAIKS US4
4:22
Ethda
09195 11-8
Ethda
DHRS
01005 114
66
Accuracy
BMRS
Compression
(a) CIFARIO Lenet5
(b) CIFARIO MLP
Figure 3: Average Spearman's 's rank correlation between the ranks of neurons for pruning when using different methods CIFAR10 (plots for additional datasets are given in Appendix E).
Figure 4: Accuracy and compression rate vs. Pi for BMRSu on CIFARIO with Lenet5. Results are averaged across 10 seeds with standard devia- tion indicated by the error bars.
Continuous pruning. Next, experiment with continuous pruning, where neurons pruned continuously throughout training based on either a provided pruning threshold (SNR, Eg [0) or AF (BMRS). For SNR, we prune a neuron when its SNR falls below 1 (as in (30), and for E,, [0] we set threshold of O.I. For L2 pruning, we perform post-training pruning based on the compression rate
8

---

Table 1: Parameter compression and accuracy for different baseline methods and settings of the proposed method. Standard deviations over ten runs are included. Best accuracy for the compression methods is given in bold.
MNIST
Fasb-MNIST
CIFARIO
Pruning Method
Comp. (%)
Acc.
Comp. (%)
Acc.
Comp- (%)
Acc.
MLP
None
0,00 0.00
97.43 + 0.14
0.00
88.17
0.20
0.00
0.00
44.94
L2
43.11 1. 2.06
10.39 1 0.32
87.86 1 2.27
18.23 1 10.22
42.89
2.64
10.00 1: 0.00
F(d)
52.08
96.88 +: 0.15
91.76 + 0.81
$5.59
0.26
77.99
43.39
0.46
SNR
58.57 = 2.01
96.92
0.08
99.83 + 0.00
10.00
= 0.00
75.93
1.26
43.97
0.46
BMRSW
48.86
1.32
96.95 + 0.19
93.20 + 0.66
84.99
0.35
76.36
1.08
43.59
0.29
BMRS
48.73
1.90
96.93
1 0.16
93.02
1 0.81
85.01
0.32
T7.17
0.98
43.45
0.42
-4
54.17 = 1.74
96.99 = 0.13
91.57 + 0.71
85. 79
+ 0.34
0.94 76.63
44.06 = 0.40
Lenet5
None
0.00 + 0.00
99.07 1: 0.09
0.00 0.00
89.16
0.27
0.00 + 0.00
67.62 0.77
L2
83.42 + 1.92
11.35 0.00
83.62 + 1.69
10.00
= 0.00
52.29 + 2.18
10.00 0.00
B[Ø]
88,29
0.56
50.93
+
66.19
65.83 0.90
SNR
92.66 1: 5.77
62.70 + 41.93
98.47 1 3.45
17.01 1: 21.03
2.02 70.29
87.68 1: 0.52
BMRS
86.90 1.15
95.59 1: 0.94
1: 1.00
77.90
2.44
62.87
0.70
BM
86.11 = 1.37
95.27 + 1.02
87.61 # 0.72
77.23 = 3.49
62.54
1.49
66.28 + 1.07
BMRS
87.58
1.01
96.66 = 0.59
88.72 = 0.73
81.10
68.07
1.95
67.66 +:66::00 0.59
achieved by BMRS Neurons are pruned after every epoch during training, followed by 10 epochs of fine-tuning at the very end of training.
We compare the raw performance of each variant using an MLP and Lenet5 on MNIST, Fashion- MNIST, and CIFARIO in Table First, note that using the L2 norm with the same compression rate as BMRSN results in a degenerate model; the accuracy degrades to random in all settings. Additionally, we that using the SNR as pruning criterion with the recommended threshold of 1 from 330 is also inconsistent, resulting in large drops in performance for 3 out of 6 settings. BMRS wV and BMRS result in both high compression rate and high performance in all settings. BMRS accomplishes this without the need for tuning any pruning thresholds. in one case yielding higher compression rate than BMRS, while keeping the accuracy high. BMRS, results in both the highest compression rate among the three BMRS variants in 4 out of 6 settings, and the highest accuracy out of 6 settings, with the caveat of needing to select P1 as a hyperparameter. To explore the effect of this hyperparameter further, we plot accuracy and compression vs. P1 for LenetS trained on CIFAR 10 in Figure 4] We see that compression rapidly increases after pl = 11, continuing until pi = 1. Additionally, we find that accuracy also steadily increases with a higher compression rate, indicating that BMRS reduces complexity while increasing the generalization capacity of the model.
Finally, a comparison of ResNet-50 and ViT on CIFAR10 and TinyImagenet is given in Table 2] Here, SNR and the three BMRS variants achieve similar accuracies different compression rates. BMRSNv achieves modest compression rate compared to SNR with a threshold of for each case. -4 yields a higher compression rate than BMRS, and BMRS 8 all settings. As such, we show that A.MRSN is capable of achieving high compression with no threshold tuning, while more extreme compression rate possible by selecting p1 for BMRS;
6 Discussion and conclusion
Our experimental results demonstrate the pruning characteristics of BMRS in two settings: continuous pruning and post-training pruning. One of the benefits of BMRS, and BMRS in particular, that no threshold tuning is needed, making it particularly useful for the continuous pruning setting. Here, the compression rate improves the model converges, allowing one to gradually increase the compression rate without sacrificing accuracy. The choice of BMRS vs. BMRS is then dependent on the problem, where more complex scenarios (e.g.. over-parameterizer models) are suitable for BMRS if a higher compression rate is desired. In this case, hyperparameter search over p1 in the range |e, 23] can done to select BMRS; (see Figure for an example), or one can simply use BMRS v to achieve good compression with no hyperparameter tuning. Additionally, we expect that the more over-specified the model is, the lower we can set Pt and maintain accuracy (i.e., lower bit-rate).

---

Table 2: Parameter compression % and accuracy for different baseline methods and settings of the proposed method. Standard deviations over three runs are included. Best accuracy for the compression methods given in bold.
CIFARIO
Tiny Imagenel
Pruning Method
Comp. %
Acc.
Comp. %
Acc.
Prett--Prerained
None
0.00 0.00
90.65 0.05
0.00 + 0.00
53.01 0.35
L2
+: 4.21
10.00 0.00
55.71
0.50 0.00
E[0]
89.85 0.09
16.27 1.86
83.59
1.50
21.11
SNR
91.73 - 0.16
89.24 = 0.40
77.85 + 0.14
50.54 0.59
BMRS
87.10 + 1.55
89. 62 0.41
74.98
0.16
50.56 0.33
BMRS 8
88.18 1: 0.22
89.29 1 0.20
74.99
0.04
50.84 1 0.34
BMRS
89.85 + 0.05
89.26 + 0.28
76.12 + 0.13
50.82 + 0.50
Vision Transformer
Nome
0.00 0.00
94.80 + 0.17
0.00 +: 0.00
63.14 1: 0.42
L2
57.74 0.26
54.36 1.84
47.47 0.12
7.08 0.57
E[0]
68.18 1 0.09
10.00 0.00
57.08 0.18
0.50 = 0.00
SNR
73.03 1: 0.03
94.78 0.10
62.75 0.04
64.60 1 0.07
BMRS
57.74 - 0.25
91.60 2 0.01
47.48
0.12
65.00 0.21
BMRS 8
58.14 + 0.11
94.69 + 0.04
47.34 + 0.14
65.14 + 0.13
BMRS 4
67.16 + 0.21
94.83 + 0.25
56.13 + 0.14
65. + 0.10
Limitations: We note a few of the limitations of BMRS. First, while multiplicative noise pruning allows for the flexible application of pruning at different structural levels, BNNs may offer more aggressive compression rates as they apply sparsity inducing priors at multiple hierarchical levels [2] BMR based approaches may be derived for such networks; as of this work and as far we we know, it has only been successfully applied in practice to models with flat priors for unstructured pruning [5] 29]. Additionally, multiplicative noise creates an overhead of additional parameters á which increase the training time and storage requirements. Third, more exhaustive baseline pruning criteria could be used for comparison in the future, for example classic methods based on the Hessian or gradient of weights (e.g. see Figure 5 in Appendix E) 23 [2]. Fourth, we apply multiplicative noise to linear layers and convolutional filters, while it could be useful to explore pruning more complex structures in the future. Finally, while structured pruning can reduce the inference time and energy consumption of neural networks, improvements in efficiency have been shown to have potential negative consequences in terms of energy consumption and carbon emissions based on how efficiency can affect how a model is used in practice 35
15
16 33 29..
Conclusions: In this work presented BMRS, an efficiently calculable method for threshold-free free structured pruning of neural networks. We derived two versions of BMRS: BMRS based on the truncated log-normal prior, and BMRS;, based on reduced truncated log-uniform prior. BMRS offers several key features over existing work: by basing the method off of the approach of multiplicative noise [30], the structured pruning aspect is flexible it is not dependent individual weights and can be easily applied at any structural level [16 (). Additionally, the prior and variational posterior in the multiplicative noise approach lend themselves to the derivation of BMRS using multiple reduced priors which have different pruning properties, allowing for flexibility in the compression rate when desired and threshold free pruning otherwise. Finally, our experimental results demonstrate the competitive compression and accuracy of BMRS compared to baseline compression methods on multiple networks of varying complexity and across multiple datasets. The methods presented here based on BMR could form a template for developing more aggressive pruning schemes by incorporating more complex hierarchical priors on both structures and individual weights, as well as for studying limits on the number of neural network structures required to solve a given dataset.
assuming any prior over
A Acknowledgments DW, CI and RS are partly funded by European Union's 's Horizon Europe Research and Innovation Programme under grant agreements No. 101070284 and No. 101070408. DW is also partly supported by a Danish Data Science Academy postdoctoral fellowship (grant: 2023-1425).
10

---

References
[1] L. W. Anthony, B. Kanding, and R. Selvan. Carbontracker: Tracking and predicting the carbon footprint of training deep learning models. In ICML Workshop on Challenges in Deploying and Monitoring Machine Learning Systems, 2020.
[2] J. Bai, Q. Song, and G. Cheng. Efficient variational inference for sparse deep learning with theoretical guarantee. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[3] B. R. Bartoldson, B. Kailkhura, and D. Blalock. Compute-efficient deep learning: Algorithmic trends and opportunities. Journal of Machine Learning Research, 24:122-1, 2023.
[4] M. J. Beal. Variational algorithms for upproximate Bayesian inference. University of London, University College London (United Kingdom), 2003.
[5] Beckers, B. Van Erp, Z. Zhao, K. Kondrashov, and B. De Vries. Principled pruning of bayesian neural networks through variational free energy minimization. IEEE Open Journal of Signal Processing, 2024.
[6] K. Friston, Parr, and P. Zeidman. Bayesian model reduction. arXiv:1805: 805.07090 [stat.ME], 2018.
[7] K. J. Friston and W. D. Penny. Post hoc Bayesian model selection. Neurolmage, 56(4): 2089-2099, 2011.
[8] K. J. Friston, V. Litvak, A. Oswal, A. Razi, K. E. Stephan, B. C. M. van Wijk. G. Ziegler, and P. Zeidman. Bayesian model reduction and empirical bayes for group (DCM) studies. Neurolmage, 128:413-43 2016.
[9] K. J. Friston, M. Lin, D. Frith, G. Pezzulo. A. Hobson, and S. Ondobaka. Active inference. curiosity and insight. Neural Computation, 29(10))2633-2683, 2017.
[10] S. Ghosh, J. Yao, and F. Doshi-Veles Model selection in bayesian neural networks via horseshoe priors. Journal of Machine Learning Research, 20:182:1-182:44 2019.
[11] R. W. Hamming, On the distribution of numbers. The Bell System Technical Journal, 49(8): 1609-1625, 1970.
[12] B. Hassibi and D. G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems (NeurIPS), pages 164-171. Morgan Kaufmann, 1992.
[13] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[14] Henderson, J. Hu, J. Romoff, E. Brunskill, D. Jurafsky, and Pineau. Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning. Journal of Machine Learning Research, 21(1)110039-00881 2020.
[15] A. Hubin and G. Storvik. Variational inference for Bayesian neural networks under model and parameter uncertainty. -xXiv:2305.00334 [stat.ML], 2023.
[16] S. R. Jantre, S. Bhattacharya, and T. Maiti. Layer adaptive node selection in bayesian neural networks: Statistical guarantees and implementation details. Neural Networks, 167:309-333 2023.
[17] Y. Jeon and J. Kim. Constructing fast network through deconstruction of convolution. In Advances in Neural Information Processing Systems (NeurIPS), pages 5955-5965, 2018.
[18] J. Kiebel, M. I. Garrido, R. Moran, and K. Friston. Dynamic causal modelling for EEG and MEG. Cognitive Neurodynamics, 2:121-136 2008.
[19] D. Kingma and M. Welling. Auto-encoding variational bayes. International Conference on Learning Representation (ICLR), 2014.

---

[20] D. P. Kingma, T. Salimans, and M. Welling. Variational dropout and the local reparameterizzation trick. In Advances in Neural Information Processing Systems (NeurIPS), pages 2575-2583, 2015.
[21] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[22] Y. LeCun. The MNIST database of handwritten digits. http://yann. /et.:/ynnnneecn...mm com/exdb mnist// 1998.
[23] Y. LeCun, S. Denker, and S. A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems (NeurIPS), pages 598-605, Morgan Kaufmann, 1989.
[24] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233) 1998.
[25] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. Graf. Pruning filters for efficient convnets. In International Conference on Learning Representation (ICLR), 2017.
[26] J. Li, Z. Miao, Q Qiu, and R. Zhang. Training bayesian neural networks with sparse subspace variational inference. In The Twelfth International Conference on Learning Representations, ICLR, 2024.
[27] Liang, J. Glossner, L. Wang, S. Shi, and X. Zhang. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461:370-403, 2021.
[28] C. Louizos, K. Ullrich, and M. Welling. Bayesian compression for deep learning. In Advances Neural Information Processing Systems (NeurIPS), pages 3288-3298, 2017.
[29] D. Markovic, K. J. Friston, and S. J. Kiebel. Bayesian sparsífication for deep neural networks with Bayesian model reduction. urXiv:2309.12099 [stat.ML], 2023.
[30] K. Neklyudov, D. Molchanov, A. Ashukha, and D. P. Vetrov. Structured bayesian pruning via log-normal multiplicative noise. In Advances in Neural Information Processing Systems (NeurlPS), pages 6775-6784, 2017.
[31] H. Peng, Q. Cao, J. Dodge, M. E. Peters, J. Fernandez, Sherborne, K. Lo, S. Skjonsberg, E. Strubell, D. Plessas, I. B. end Evan Pete Walsh, N. A. Smith, and H. Hajishirzi. Efficiency Pentathlon: A standardized arena for efficiency evaluation. arXiv:2307.099 097011 [cs.CL], 2023.
[32] Sedgwick. Spearman's 's rank correlation coefficient, BMJ, 349, 2014.
[33] Y. Sun, Q. Song, and F. Liang. Learning sparse deep neural networks with a spike-and-slat prior. Statistics & Probability Letters, 180:109246, 2022.
[34] N. Thompson, K. Greenewald, K. Lee, and G. F. Manso. The Computational Limits of Deep Learning. In Workshop on Computing within Limils, 2023.
[35] D. Wright, C. Igel, G. Samuel, and R. Selvan. Efficiency is not enough: A critical perspective of environmentally sustainable AI, arXiv:2309.080 02005 [cs.LG],, 2023.
[36] B. Wu, C. Xu, X. Dai, A. Wan, P. Zhang, Z. Yan, M. Tomizuka, J. Gonzalez, Keutzer, and P. Vajda. Visual transformers: Token- image representation and processing for computer vision. arXiv:2006.03677 [cs.CV], 2020.
[37] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. 1IXi:1700..0777 [cs.LG], 2017.
12

---

A Derivations
We use the following notation and distributions:
(2) 1 erf
(CDF of ( 1) evaluated at 2))
/2
a Up
0p Pp
b Hp
3, =
Op
2, ((pp) - ((pp))
exp
(log 0 Omp)=
a asosb
LogN (a.0) (1,0 (011pp oz))
2,0
2
2
2
0,
otherwise
LogU (0,00 = (Alog ")
a a otherwise
0,
(0) = LogN [a.00] (0 14g og)
5(0) I LogN (1.00 (0) ppp op
p(0) Log 10,00 (0)
A.1 BMRS
We start by finding go (0) P(0) p(A).
First we look at qs (0)) 2 Zo 09
1 ((logg 0 - -2,)²
(log 0 - -mp))
}
exp
2
0
The exponent can be rewritten as
1 2
+ (log 0 - ppp)2 (log 0- - -mn)2
02
03 p
7 (log² 2 0 log 01g + /²)) + + 0- 2 log 0ūp + x17)
2
1 (o + 52)
(log 0
0² og oo+oz
Fg
+
Up 0
o 02 + o
(0 + o2)
0303
+
0
oG+oz
2
0 p
Defining
02 =
p
2-2
+
1 -1
and
jug
:
02
F1g
02 + 02
02
p
we get
1
(log
0- )2
1
n2a
-
(0))(())
exp
exp
/
+
6 Z,Z,
0700
2
02
2
OZ /
2T
4
1 (log 0 - -2,
exp {
-
(
1
)
2 Zo pp v 2003 OZ,,
2T
exp
03
2
2 02
1
12
72
LogN (0,4) (01āg,
exp
0 2T02503
2
13

---

2. Then we divide out p(O):
(0) P(0) p(0)
(log b - log 2g v
i o
v 8
exp
Log
(a.b) (017gg 02)
ZoZp / 2TO203
3. Now we can find ((0) and AF using Equation 6
Start with ((0):
((0) (0( p(O)
cxp AF
l (0) ((t)
p(O)
J go (0)
P(O) p(0) do
(log bolg a) 2 83
A3
Z 2, 2 2
exp
2
0g
LogN [0,0) (0) 02)
(logb- log 0) Zgy
1
exp
+
LogN 07)00
2a 202
2
02
(log b- log a) 20
} LogN 0) (0) og))
2, 2,20 2
exp
2
+
(log b- log 2 /82
2, 2p 2n0 70
exp
-
2
+
LogN [0,b] (8) 82)
LogN [a,b) (0 og)
4. Finally we get AF:
Lo (0) 2(0)
A F= log
M()
(())
(log log a) 12
/v
log
exp
+
ZyZØy 2
2
log
Zg (log) b- log a)
1
log
9
ZyZr
2
2
72
2
4
F
A.2 BMRS
The PDF of the reduced truncated log-uniform distribution is given as follows:
{
log a S a' < S
b')
p(0) LogU (ar.0) Lo
(14)
otherwise
Using this, we can directly solve the integral under the expectation given in Equation for A
(0)
J." b
(
do f θ log
exp AF = R5p
LogU (0)
(0)d0
p(0)
LogU (a,0
(0)
0 log a
N6 log
a' log go (0) + f" 0d0 = log
0d0 +
(a < 0, < ~b)
log
Plugging this in to Equation 5 where exp AF > 1:
log (a S S0 S 50
log a' < (a 50
1 5
(15)
log
log b
14

---

B Dataset details
MNIST MNIST 22 is classic image classification dataset consisting of 70,000 28x28 black and white images of handwritten digits (10 classes). We use the original 10,000 image test set for testing and split the 60,000 image train set into 80% training and 20% validation images.
Fashion-MNNIST Fashion- -NNIST (37 is modernized version of MNIST using images of different articles of clothing as opposed to handwritten digits. The dataset statistics are the same as MNIST: 28x28 greyscale images, 60,000 training images, 10,000 test images, 10 classes. Similar to MNIST, we split the training set into 80% training and 20% validation images.
CIFARIO CIFARIO [21) is an image classification dataset of 32x32 color images with 10 classes. There are 50,000 training images and 10,000 test images. Again, we split the training 80% training and 20% validation.
Tiny Imagenet Tinylmagenet a reduced version of ImageNet consisting of 100,00 64x64 color images in 200 classes. We use the 10,000 image validation split for testing, and split the 100,000 image train set into 80% training and 20% validation images.
C Model details
For each model and dataset use the Adam optimizer with no weight decay. We train for 50 epochs for each experiment with an MLP and Lenet5, and for 100 epochs for each experiment with Resnet50 and ViT. Further details about each model are given as follows:
C.1 MLP
We use a multilayer pereeptron (MLP) for several experiments, with different network sizes based on a hyperparameter sweep for each dataset. Multiplicative noise for pruning is applied to every neuron the network. We sweep through the following hyperparmeters:
Number of layers {1,3,,,7,9)
Hidden dimension = {10, 30, 50, 100, 150)
Batch size = [16, 32, 64, 128)
Learning rate [0.0001, 0.1].
The final network settings for each dataset are given as follows:
MNIST: Number of layers: 7; Hidden dimension: 100; Batch size: 128; Learning rate: 8.5-10-4.
Mashion-MNIST Number of layers: Hidden dimension: 150; Batch size: 128; Learning rate:
1.5-10-3
CIFAR10 Number of layers: 5; Hidden dimension: 150; Batch size: 32; Learning rate: 6,8-10-4
C.2 Lenet5
Lenet5 [24] is an early CNN architecture consisting of 2 convolutional layers with 6 and 16 filters per channel, respectively, each followed by a ReLU activation and max pooling layer, followed by linear layers. Multiplicative noise is applied each convolutional filter map, well each neuron the the linear layers. We use the same architecture for each experiments and tune hyperparameters based the dataset. We sweep through the following hyperparameters:
Batch size | [16, 32, 64, 128) Learning rate [0.0001, 0.1].
The final settings for each dataset are given follows:
MNIST: Batch size 128: Learning rate 1
15

---

Table 3: Training and inference runtimes in milliseconds. Each runtime is averaged across 1000 forward passes of batch of 32 images. Inference runtimes are before pruning (i.e., with the full network).
MNIST
Fash MNIST
CIFARI0
Prusing Method
Train
Inf.
Train
Inf.
Train
Inf.
MLP
5.30
F[(]]
19.76
21.21
5.39
SNR
20.00
1.14
19.99
1.15
22.00
4.94
BMRS
20.02
1.14
20.14
1.15
21.76
5.57
BMRS
19.99
1.16
20.11
1.16
22.14
5.15
Lenet5
None
2.31
4.73
5.15
F[(0]
4.90
11.18
11.42
15.05
4.95
BMRS
11.39
0.94
11.47
0.94
13.34
5.07
BMRS
11.44
0.94
11.08
0.93
12.86
4.43
Fashion-MNIST
Batch size 32; Learning rate 1.4-10-3
CIFARIO Batch size 64; Learning rate 1-10-3.
C.3
Resnet50
Resnet50 [S] is a deep 50-layer CNN which uses residual connections to stabilize optimization and improve accuracy. We start with model pretrained ImageNet- then then fine-tuned the downstream dataset with pruning layers added to each output layer after batch normalization. We learning rate of 6.8e-4, batch size of 32, and train for 100 epochs for both CIFARIO and Tinylmagens
C.4 Vision Transformer (ViT)
Vision Transformer (ViT) 36 is transformer model tailored for image data based tokenizing
image as 16x16 image patches. We use ViT which is pretrained on ImageNet-21K (14M images and 21, 18.843 classes) well as ImageNet-1k(" We add multiplicative noise the output layer of each transformer block for pruning. We use a learning rate of 6.8e-4, batch size of 32, and train for 100 epochs for both CIFARIO and Tinylmagenet.
D Compute resources
All experiments were run shared cluster. Requested jobs consisted of 16GB of RAM and Intel Xeon Silver 41 10 CPUs. We used a single NVIDIA Titan X GPU with 24GB of RAM for all experiments, though utilization was generally much lower due the the average size of each network. Runtimes for each experiment ranged from approx. 7 minutes for LenetS on MNIST with no pruning layers to approx. 44 hours for ViT on Tinylmagenet with multiplicative noise trained for 100 epochs. The training of models in this work over the course of the entire project (prototyping, experimentation. etc.) is estimated to have used 3773.78 785 kWh of electricity contributing to 599,892 kg of CO2eq (as measured by carbontracker I this is equivalent to 5580.395 km travelled by car).
We additionally benchmark the training and inference runtimes of each model before pruning in Table 3 and Table (i.e., the inference runtimes are without removing structures).
httpp://groec..ooryy//sssssssaes html
http:://uuggiggfaee.oooooooooo wttp::/:ww..cooooooooooo.ecoctee www.aaee-pachi---22 2atcch16-224
16

---

Table 4: Training and inference runtimes in milliseconds. Each runtime is averaged across 1000 forward passes of batch of 32 images. Inference runtimes are before pruning (i.e., with the full network).
CIFARIO
Tinylmagenet
Pruning Method
Train
Inf.
Trii
Inf.
Res50 Pretrained
34.30
74.89
[]
329.
35.24
346.80
SNR
328.29
33.97
346.56
64.00
BMRS
329.84
32.99
343.63
64.01
BMRS
329.81
31.85
346.78
64.06
Vision Transformer
None
251.50
87.15
252.91
F]
343.51
351.19
343.50
350.93
BMRS
343.55
88.74
330.99
87.55
BMRS, 4
343.60
88.56
351.39
87.47
E
Additional plots
An additional experiment for post-training pruning including gradient- crasient-based thersholding for pruning is given in Figure 5) For gradient based pruning, we use two thresholds: one for the L2-Norm (magnitude) of structures, and one for the L2-Norm of the gradients of weights in given structure.
The neuron rank correlations for FaSHion-MNIST are given Figure and for MNIST in Figure 7
17

---

0.7
0.50
0.6
0.4
Accuracy
Accuracy
0.6
Accuracy
0.4
0.5
0.3
SMR
SNR
0.40
0.2
BHRS N
Ethets
BMRS_N
SMR
0.2
Ethela
BMR5,N
BMRS,N
SARR
BMR5_U-6
BMRS_UA
BMAS_UF
0.35
BMA L
EHRS U.4
BMRS_U4
Compression
Compression
Compression
Compression
(a) CIFARIO Lenet5
(b) CIFARIO MLP
0.8
0.8
0.8
Accuraty
0.6
Accuracy
0.6
Accuracy
0.4
SAR
0.4
0.4
SNH
0.4
Ethets
SMR
Etheta
SNR
BHRS
ENRS_U
0.2
BMRS N
BMRS_U8
BMRS N
BNAS.U
0.2
BVRS
6MRS_U-
0.0
Compression
Compression
Compression
Compression
(c) Fashion-MNIST LenetS
(d) Fashion-MNIST MLP
1.0
1.0
1.05
0.8
0.8
0.8
Accuracy
Accurscy
0.6
0.6
Gmd
0.44
SMR
0..4
0.4
SNR
Ethota
SNR
Ethtt
SNR SWILS
0.2
BMIS
BMiS
BMH5
0.2
0.0
0.0
Compression
Compression
Compression
(e) MNIST Lenet5
(i) MNIST MLP
Figure 5: Additional accuracy vs. compression results for post-training pruning including gradient- based pruning.
SNR
Etheta
0.59
0.67
..1
0.45
Etheta
1
.0.23
0.94
0.63
00.1
L2
0.59
0.59
-0.69
-0.24
L2
0.23 -0.23
0021
0.33
BMRS
BMRS_N N
BMRS_U-8
0.1
..11
0.0069
0.2
1
0.66
BMRS_ U-8
0.63
0.63
0.33
0.59
0.93
BMRS U-4
0045
0.45
0.24
-0.12
0.66
BMRS U-4 0.11
0071
0.34
0.66
0.93
SNR
Etheta
3
BMRS_M N
BMRS U-4
SNR
Etheta
4
BMRS N
BMRS_UP U-8
BMRS U-4
BMRS
(a) Fashion-MNIST Lenet5
(b) FaShion-MNIST ML.
Figure 6: Average correlation between the ranks of neurons for pruning when using different methods on -NIST..
18

---

SNR
0.77
0.94
3.016
0.36
SNR
1
0.48
0.33
0.17
0.034
Etheta
0.015
Etheta
L2
0.77
0.77
74
0.009
00.31
0.49
0.48
0.04
0.39
0.23
BMRS_N N
0.94
0.93
0.74
0.034
-0.3
BMRS N
0.93
0.93
0.44
0.22
0.033
BMRS U-8
0.016
0.089 0.034
0.65
BMRS_UA U-8
0.17
BMRS_
SNR
Etheta
L2
BMR5 N
BMRS U.88
BMRS U-4
SNR
Etheta
12
BMRS_ N
BMRS_ U-8
BMRS U-4
(a) MNIST Lenet5
(b) MNIST MI.P
Figure Average correlation between the ranks of neurons for pruning when using different methods on MNIST.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract accurately reflects the main claims of the paper, see the last paragraph of O for the primary contributions for comparison.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed after $6 paragraph marked "Limitations".
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

We provide detailed derivations and describe assumptions for each new theoretical result $4.2 and Appendix A
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The full methodology is given in $3 and $4 the experimental setting given in $ the full dataset and model details are given in Appendix B and Appendix C respectively.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The datasets we use are all open source, and details are given Appendix B The code is made available and can be downloaded/viewed at https: // //gi thub. com/
saintslab/bmrs -ttruuutrrnnnsstuuuuurreeed pruning/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
gwtp://wwiee/eeee Saaieees/oooSemmiiiiaa SonroddeummsssiooPPe.. wwriiisssiiiiiyiyioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental setting is given in $5 the full dataset and model details given in Appendix B and Appendix C. respectively.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All of our experiments bars (areas of standard deviation or confidence intervals). All tables include the mean and standard deviation across multiple seeds.
across multiple seeds. All plots include error
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Information on the compute infrastructure and resources are given in Ap- pendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttga///oourips..cpuuule/////i
Answer:
[Yes]
Justification:
We conform the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We point out in the limitations that efficiency can also have potential negative environmental aspects due to e.g. the rebound effect.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: As we propose a general method for neural network pruning, we do not see where safeguards could be put in place or that there is high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We clearly mark where any non-original code is used (see e.g. modules/utils..y in the code).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide links to the code in the paper and have documented the code for ease of use/reproducibility.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do perform crowdsourced experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not do crowdsourcing or involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
25

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "SpjDVIUTvM", "reviewer": "Reviewer_238H", "summary": "This paper works on structured pruning using Bayesian models. They try both post-training pruning and continuous pruning for MNIST and CIFAR-10 datasets.", "questions": "What is the novelty of this work?", "limitations": "See the weakness section.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The writing is easy to read and follow.", "weaknesses": "1. The novelty is quite limited:\n\nThe BMRS basically in my opinion is a naive extension as previous Variational dropout [1], or [2]. BMRS seems to simply combine these works and pruning works, and test different priors and criteria. There are no contributions from both theoretical and empirical sides. This work seems to be highly related to SSVI [3] published months ago. [3] also combines pruning and BNN training, but with fully sparse training and novel pruning criteria designs.\n\n2. Performance is quite bad.\n\nFor example, as shown in Fig 2, BMSR can only reach around 60% compression rate. This is quite bad for CIFAR-10. Previous works like [3] and most of the pruning papers can get more than 90% compression rate.\n\n3. No baselines to compare.\n\nI see no previous works as baselines. Seems that this work aims at design a pruning algorithm, then all the modern pure pruning algorithm should be compared. However, non of these are shown.\n\n4. Missing reference\n\nFor example, [3] is highly related but missing in this paper.\n\n\n[1] D. P. Kingma, T. Salimans, and M. Welling. Variational dropout and the local reparameterization trick. In Advances in Neural Information Processing Systems (NeurIPS), pages 2575–2583, 2015.\n\n[2] C. Louizos, K. Ullrich, and M. Welling. Bayesian compression for deep learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 3288–3298, 2017.\n\n[3] Li, J., Miao, Z., Qiu, Q., & Zhang, R. (2024). Training Bayesian Neural Networks with Sparse Subspace Variational Inference. arXiv preprint arXiv:2402.11025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time reviewing the paper. We would like to address the points they make in their review:\n\n**Novelty**\n\nWe respectfully disagree with the characterization of our method as a naive extension of variational dropout. We derive novel pruning criteria for a class of structured pruning models which are theoretically grounded, namely from the perspective of Bayesian model reduction (BMR). This yields a robust approach to structured pruning that performs well across our experiments/datasets without tuning of any threshold parameters. The different priors used in the paper are to attain different pruning criteria, as different realizations of BMR result from different priors; they are _not_ different priors on the model parameters. In recent work, discussed in our paper, BMR has been successfully applied in the unstructured pruning case [1,2] but not in the structured case as far as we know. \n\n**Performance**\n\nDeveloping BMR for structured pruning is the main contribution of our work. Specifically, to 1) derive theoretically grounded pruning criteria for structured pruning and 2) empirically characterize the pruning characteristics of these criteria with respect to existing criteria. Our work serves as a step towards principled Bayesian structured pruning criteria via BMR. And when comparing with methods that require tuning of importance thresholds (L2, SNR) we show that we achieve reasonable trade-offs between compression and performance. It is not expected that we would attain higher compression than, e.g., Bayesian approaches which induce both unstructured *and* structured sparsity.\n\n**Baselines and Datasets**\n\nWe would like to point out that we use MNIST, Fashion-MNIST, CIFAR10, and tiny-imagenet for our experiments (not only MNIST and CIFAR10 as the reviewer mentions).\n\nSecond, we compared to relevant baseline pruning criteria that have been used in previous work for the model class of Bayesian structured pruning that we study. Indeed in the paper referenced by the reviewer [3], their proposed pruning criteria are based on variations of $E_{q_{\\phi}}[\\theta]$ and SNR$(\\theta)$; we compare to SNR$(\\theta)$ as a baseline, and we have added $E_{q_{\\phi}}[\\theta]$ in the rebuttal PDF document, which will also be updated in the final version of the paper.\n\n**Missing citations**\n\nWe thank the reviewer for pointing out the very recent work [3], which we will cite in our paper; the two other papers on variational pruning which the reviewer mentions are already discussed in our submission.\n\n**References**\n\n[1] J. Beckers, B. Van Erp, Z. Zhao, K. Kondrashov, and B. De Vries. Principled pruning of bayesian neural networks through variational free energy minimization. IEEE Open Journal of Signal Processing, 2024.\n\n[2] D. Markovic, K. J. Friston, and S. J. Kiebel. Bayesian sparsification for deep neural networks with Bayesian model reduction. arXiv:2309.12095, 2023.\n\n[3] J. Li, Z. Miao, Q. Qiu, and R. Zhang. Training bayesian neural networks with sparse subspace variational inference. CoRR, abs/2402.11025, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "utzz2tqch0", "reviewer": "Reviewer_v8Lj", "summary": "This paper proposes a new probabilistic approach to structured pruning of neural networks. Inspired by variational dropout, the authors derive a method to learn a multiplicative noise distribution, which is encoded in a multiplicative noise layer. Pruning algorithms are derived based on assumed priors over the data and parameter distributions (specifically, the distribution of $p(\\mathcal{D}|\\theta)$, where $\\mathcal{D}$ is the dataset and $\\theta$ is the multiplicative noise). The derived pruning algorithms are then applied to some standard image classification benchmarks.", "questions": "1. What is the computational advantage, if any, of applying the proposed variational approach to structured pruning versus more classical approaches? \n2. In the post-training setting, how many samples are required to efficiently compute $\\Delta F$? \n3. Are there other choices of priors that  were considered in this work?\n\nPlease refer to the 'Weaknesses' section as well.", "limitations": "The limitations are discussed in this paper, specifically in section 6.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The variational approach to network pruning is interesting and well-motivated. \n2. The mathematical derivations seem to be correct.\n3. The writing is reasonably clear, though a specific notation section aside from the Problem Formulation in Section 3.1 would have been helpful.\n4. The paragraph on the connection between BMRS and floating point precision was interesting and highlights strong scholarship (lines 199-218).\n5. A variety of algorithms (variants) are derived, and compared empirically, both in the post-training setting, as well as pruning while training.", "weaknesses": "1. Some figures could be slightly more clear; for instance, Figure 1 should show the y-axis as well on the plot. \n2. The experimental evaluation does not show Imagenet experiments. For pruning on vision datasets, this is now a standard benchmark, and should be presented in any experimental setup.\n3. Modern architectures -including transformer architectures and ResNet models such as the ResNet50 model used in the experiments - have complex interconnections between layers, such as residual connections in ResNets (see [1]-[3]). These interconnections make pruning such models (by genuinely changing the architectures) challenging, as saliencies have to be computed for *groups of connected filters*, as opposed to individual filters. It would have been extremely interesting to see the authors apply their approach to complex interconnections instead of just individual filters.\n4. It would have been nice to see the authors use their approach to analyze \"how many filters/components are required to learn a given dataset\". The variational approach seems like it would have provided an interesting insight into such questions, which would potentially have implications in other fields as well, such as NAS and continual learning. \n5. The empirical evaluations should baseline against more than just L2, particularly in the post-training pruning. This significantly weakens the evaluation.\n\nReferences\n\n[1] Fang et al. *DepGraph: Towards Any Structural Pruning*\n\n[2] Narshana et al. *DFPC: Data flow driven pruning of coupled channels without data.*\n\n[3] Liu et al. *Group Fisher Pruning for Practical Network Compression*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and engagement with the paper. We are happy that they found the **approach interesting and well motivated**, **the writing clear**, **the math sound**, the **connection to floating point format interesting**, and **that we derive a variety of pruning algorithms**. We address their weaknesses below:\n\n**W1. Figure clarity**\n\nWe will revise the figures to be more clear, including the suggestion for Figure 1 which has now been updated in the rebuttal PDF. \n\n**W2. No ImageNet**\n\nThe ResNet and ViT that we use in the paper are pre-trained on ImageNet, and then further fine-tuned on Tiny-ImageNet, which consists of 100,000 ImageNet images from 200 classes. While not as large scale as ImageNet, we do not expect to see radically different compression vs. accuracy if we perform further fine-tuning on the full ImageNet dataset. That being said, we are happy to perform experiments and include results on ImageNet in the final version of the paper, which could not be completed within the rebuttal period due to the size of ImageNet. \n\n**W3. Compare pruning characteristics when applied to complex structures**\n\nThis is indeed an interesting point, and well worth looking into. The scope of our work is focused on pruning criteria for Bayesian structured pruning, while this question is slightly broader (relating to the Bayesian pruning literature in general). We can point this out in the paper as a useful future direction to explore.\n\n**W4. How many filters are required to learn a given dataset?**\n\nThis is also an interesting research question! Figures 2, 5, and 7 (the post-training pruning experiments) show us this empirically. The knee-points of the SNR curve show us where accuracy begins to drop for higher compression. We see here that BMRS gives us a **lower bound** on the number of prunable structures in a network i.e. slightly higher compression can be achieved without significant drops in accuracy, but the accuracy will quickly decrease with further compression.\n\nIt should also be noted that different sparsity inducing priors will lead to different answers to this question. For example, [1] introduce a hierarchical prior design to induce both unstructured and structured sparsity, which can lead to a more sparse solution (and thus fewer parameters required to learn a given dataset), but is more complex to train and BMR cannot be directly used for group sparsity.\n\n**W5. More Baselines**\n\nWe agree on the importance of relevant baselines. In our case, the baselines are different pruning criteria for multiplicative noise with the log Uniform prior, so we include L2 norm, SNR, and a no-pruning baseline. \n\nAdditional baselines that are comparable are:\n- Expected value, $E[\\theta]$\n- Magnitude of gradient\n- Hessian of gradients\n- Magnitude of activation \n\nGiven the time constraints we have not been able to finish running all of these baselines, but we have submitted updated versions of Figure 2 and Table 1 with the finished results for $E[\\theta]$ in the supplemental material (the rest will be included in the final version).\n\n**Q1. Advantage of variational pruning vs. classical methods**\n\nThis depends on what the reviewer mean by \"computational advantage\"; the variational approach can lead to **higher sparsification** than classic approaches, which provides a net gain at inference time, but can be **more costly to train** due to the overhead of the variational parameters.\n\n**Q2. How many samples required for $\\Delta F$?**\n\nOne of the benefits of our proposed method is that the calculation of $\\Delta F$ does not require any sampling; it is calculated analytically using the statistics of the variational distribution, original prior, and reduced prior, making it quite efficient (see Eq. 8 and Eq. 11).\n\n**Q3. Are there other choices of prior considered?**\n\nThis depends on if the reviewer means the prior for the original model, or the reduced priors used to calculate $\\Delta F$. The former is selected based on previous work that we build upon [2]. The latter are selected in order to be able to calculate $\\Delta F$ analytically.\n\n**References**\n\n[1] D. Markovic, K. J. Friston, and S. J. Kiebel. Bayesian sparsification for deep neural networks with Bayesian model reduction. arXiv:2309.12095, 2023.\n\n[2] K. Neklyudov, D. Molchanov, A. Ashukha, and D. P. Vetrov. Structured bayesian pruning via log-normal multiplicative noise. In Advances in Neural Information Processing Systems (NeurIPS), pages 6775–6784, 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "feBAbz9IOz", "reviewer": "Reviewer_L897", "summary": "The paper introduces Bayesian model reduction for structured pruning (BMRS), an efficient method for structured pruning of neural networks. It improves Bayesian structured pruning with multiplicative noise by combining it with Bayesian model reduction, enabling a principled pruning strategy based on efficient Bayesian model comparison. Two variants with different priors are proposed, threshold-free BMRS_N and BMRS_U with a hyperparameter that allows for more aggressive compression. The experimental evaluation on a range of data sets and neural network architectures demonstrates the usefulness of both approaches.", "questions": "- As the main figure of the paper, Figure 1 could profit from a more explicit description both in the graphs themselves (e.g., axis labels) and the label (e.g., a more detailed description of the steps taken or that it uses the log-uniform prior).\n- Why is a pruning threshold defined for L2 but not SNR at the start of Section 5? Isn't a major drawback of SNR the need for choosing a threshold?", "limitations": "Several limitations are discussed.", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper tackles an important problem with fundamental societal impact, especially given the massive resource requirements of current large language models.\n- The paper is well-written and introduces all required concepts clearly, also to readers like me who are less familiar with the field. It focuses on addressing a single gap in the existing literature clearly and comprehensively.\n- An anonymous code repository with clear structure and dependencies is provided upfront.\n- The experiments are thoughtfully designed, featuring a range of data sets and neural network architectures, aggregation across a sufficiently high number of random seeds, clear presentation of the main results, and mostly convincing evidence for the usefulness of the proposed method.", "weaknesses": "- The performance of the proposed BMRS method is better than SNR for simple architectures (see Table 1), but this advantage vanishes for more complex architectures (see Table 2), where SNR achieves higher compression at similar accuracy for 3 of 4 settings. Unlike the thorough analysis of the results for the rest of the experiments, possible underlying reasons are not discussed here. Ultimately, the economic and environmental impact of the proposed pruning method scales with the size of the neural network architecture, so I believe these results to be crucial for extrapolating to bigger architectures that are not computationally feasible to evaluate experimentally here (e.g., LLMs). These concerns currently prevent me from giving a higher score to this otherwise excellent work.\n- The computational overhead of introducing multiplicative noise is discussed in the Limitations Section, but not considered in the experiments. I understand that this is not related to the novelties of the BMRS method itself, but it would still be helpful to provide runtime information in the tables so that the difference to the baseline can be factored in.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and thoughtful review! We are happy that they noted that **the paper tackles and important problem**, it is **well-written** and **clearly and comprehensively addresses a gap in the literature**, and that **the experiments are thoughtfully designed**. We address their concerns and questions below.\n\n**W1. Differences in performance advantage for different settings**\n\nThe reviewer is referring to the continuous pruning experiments. For MNIST, Fashion-MNIST, and CIFAR10 with an MLP and LeNet, BMRS clearly outperforms the next best baseline in terms of compression vs. accuracy, while for ResNet50 and ViT on CIFAR10 and Tiny-ImageNet, SNR provides slightly higher compression at comparable accuracy. We explain this as follows:\n\nSNR is dependent on selecting a pruning threshold, which we set at a fixed value (1.0) as used in previous work [1]. We show that this results in radically different pruning behavior for different model sizes and datasets. In contrast, BMRS$_{\\mathcal{N}}$ does not use any explicit threshold, and its solution is dependent on how good our variational approximation is. We chose to end training at 200 epochs for the larger models, but with further training we expect the compression gap to close.\n\nAdditionally, we show two settings of $p_{1}$ for $BMRS_{\\mathcal{U}}$ (8 and 4), but with lower values of $p_{1}$ we would expect to see a higher compression rate (e.g., see Figure 4).\n\n**W2. Runtime information**\n\nThis is an excellent point! We have included the average training runtime for a sample of datasets and models in the additional supplement. We will update these numbers for all experiments in the final version. Also note that this is the average *total training time*, which can vary depending on the load on our compute cluster and which nodes the models are run on. We can provide fair training and inference benchmarking in the final version of the paper.\n\n**Q1. Improve Figure 1**\n\nThank you for the suggestions! We have incorporated the changes to Figure 1 and included it in the additional PDF. \n\n**Q2. No pruning threshold for SNR**\n\nThat is correct, and it is an oversight on our part; the pruning threshold is set to 1.0 for SNR in order to compare with previous work. This is mentioned at line 275 as the threshold is only used for continuous pruning, but we can additionally mention this at line 243.\n\n**References**\n\n[1] K. Neklyudov, D. Molchanov, A. Ashukha, and D. P. Vetrov. Structured bayesian pruning via log-normal multiplicative noise. In Advances in Neural Information Processing Systems (NeurIPS), pages 6775–6784, 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their time and their reviews. We are glad that they generally found the **approach interesting**, the **math sound**, and the **problem important**. We are also happy that they all found the writing clear and the paper presented well. To contextualize the reviews, we would like to echo a point made by reviewer L897: _that the paper addresses a single, clear gap in the literature on Bayesian structured pruning_.

To the best knowledge of the authors, Bayesian Model Reduction (BMR) for structured pruning has been successfully explored for the first time in this work.  Our key contribution shows how to derive principled and theoretically motivated pruning criteria for Bayesian structured pruning using BMR. We empirically show that our derived pruning criteria work on four benchmark datasets (MNIST, FashionMNIST, CIFAR-10, Tiny-ImageNet) across different architectures (MLP, LeNet5, ResNet-50, Vision Transformer).

Given this, we address the concerns of each reviewer in the individual responses. In addition, we include the following in the additional PDF as part of our rebuttal:

 - An updated Figure 1, including more clear x- and y-axis ticks and labels, as well as an updated caption
 - Additional baseline results using $E[\theta]$; this includes an updated Table 1 and Figure 2 (other figures and tables will be updated in final version)
 - Average training runtimes for Lenet5 and ViT on all of the datasets in the paper

We are looking forward to engage in the author-reviewer discussion.

Meta-review: The paper proposes a new approach to structured pruning of neural networks using Bayesian model reduction (BMR). The authors derive a method to learn a multiplicative noise distribution, which is encoded in a multiplicative noise layer, and use this to prune the network. The paper presents a thorough experimental evaluation on several datasets and architectures, demonstrating the effectiveness of the proposed approach.

The reviewers generally found the paper to be well-written, clear, and well-motivated, with a good presentation of the mathematical derivations and experimental results. However, some reviewers raised concerns about the novelty of the approach, the performance of the method on more complex architectures, the computational overhead of introducing multiplicative noise, and the lack of baselines and ablation studies. The authors responded to the reviewers' comments, addressing the concerns and providing additional information and clarification. They also provided updated results, including additional baselines and runtime information, which helped to alleviate some of the concerns.

After the rebuttal and discussion, all reviewers lean towards acceptance. Overall, I would thus recommend to accept the submission. However, I would further recommend that the authors take the reviewer feedback into account for the camera-ready version.
Decision: Accept (spotlight)