Optimal Brain Apoptosis
Authors: Mingyuan Sun
Zheng Fang
Jiaxu Wang
Junjie Jiang
Delei Kong
Chenming Hu
Yuetong Fang
Renjing Xu
Venue: ICLR 2025 Poster (2025)
Keywords: Network Pruning
Efficient Maching Learning
Hessian Matrix
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We directly calculate the hessian-vector product term in Taylor expansion for accurate importance score estimation in network pruning.

Abstract: The increasing complexity and parameter count of Convolutional Neural Networks (CNNs) and Transformers pose challenges in terms of computational efficiency and resource demands. Pruning has been identified as an effective strategy to address these challenges by removing redundant elements such as neurons, channels, or connections, thereby enhancing computational efficiency without heavily compromising performance. This paper builds on the foundational work of Optimal Brain Damage (OBD) by advancing the methodology of parameter importance estimation using the Hessian matrix. Unlike previous approaches that rely on approximations, we introduce Optimal Brain Apoptosis (OBA), a novel pruning method that calculates the Hessian-vector product value directly for each parameter. By decomposing the Hessian matrix across network layers and identifying conditions under which inter-layer Hessian submatrices are non-zero, we propose a highly efficient technique for computing the second-order Taylor expansion of parameters. This approach allows for a more precise pruning process, particularly in the context of CNNs and Transformers, as validated in our experiments including VGG19, ResNet32, ResNet50, and ViT-B/16 on CIFAR10, CIFAR100 and Imagenet datasets. Our code is available at https://github.com/NEU-REAL/OBA.

Published as a conference paper at ICLR 2025
OPTIMAL BRAIN APOPTOSIS
Mingyuan Sun', Zheng Fang Jiaxu Lang",Junjii Jiang', Delei Kong,, Chenming Hu', Yuetong Fang" & Renjing Xu NNotheesttern University Thhe Hong Kong University of Science and Technology (Guangzhou) HHnna University
ABSTRACT
The increasing complexity and parameter count of Convolutional Neural Net- works (CNNs) and Transformers pose challenges in terms of computational ef- ficiency and resource demands. Pruning has been identified as an effective strat- egy to address these challenges by removing redundant elements such as neu- rons, channels, or connections, thereby enhancing computational efficiency with- out heavily compromising performance. This paper builds on the foundational work of Optimal Brain Damage (OBD) by advancing the methodology of param- eter importance estimation using the Hessian matrix. Unlike previous approaches that rely on approximations, we introduce Optimal Brain Apoptosis (OBA), a novel pruning method that calculates the Hessian-vector product value directly for each parameter. By decomposing the Hessian matrix across network lay- ers and identifying conditions under which inter-layer Hessian submatrices are non-zero, we propose a highly efficient technique for computing the second- order Taylor expansion of parameters. This approach allows for more pre- cise pruning process, particularly in the context of CNNs and Transformers, as validated in our experiments including VGG19, ResNet32, ResNet50, and ViT- B/16 on CIFARIO, CIFAR 100 and Imagenet datasets. Our code is available at https:/ / thub. com NEU-REAL/ /OBA.
INTRODUCTION
With the rapid development of deep learning, neural networks have become deeply integrated into all sectors of our daily life. Convolutional Neural Networks (LeCun et al., 1998; Krizhevsky et al., 2012; He al., 2016; Ma al., 2023) and Transformers (Vaswani al., 2017; Dosovitskiy al., 2021) are two typical structures used most widely. As researchers continuously innovate, the perfor- mance of neural networks improves, but the number of parameters and the computational complexity also increase significantly. Therefore, how to efficiently reduce the parameter size and computational overhead of neural networks while maintaining their performance as much as possible has become a crucial problem.
Extensive research (LeCun al., 1989; Molchanov al., 2016) demonstrates that pruning a powerful tool for dealing with this issue. Generally speaking, pruning can be divided into two main streams: unstructured pruning and structured pruning. Unstructured pruning (Guo et al., 2016; Han et al., 2015; Dong et al., 2017) involves the removal of individual weights or neurons from neural network. The key advantage of unstructured pruning lies in its flexibility and the fine- fne-grained control it offers over the model's architecture. This method often requires specialized hardware software to exploit the resultant sparsity for computational efficiency. Structured pruning (Anwar et al., 2017; Yeom et al., 2021) removes entire neurons, channels, or layers from a neural network, which is more frendly to software since parameters of neural networks are mainly structured data, such as tensor, matrix, and vector. Removing entire neurons directly corresponds to slicing or selecting operation on the structured data, which is easy to implement and more compatible with standard hardware accelerators, such as GPUs and TPUs.
Hanson & Pratt (1988) is one of the earliest works to explore structured pruning. The underlying idea that significant weights typically possess greater magnitudes, they need to process and transmit
"ooresponning authors.

---

Published as a conference paper at ICLR 2025
more information to be influential in determining the network's accurate output. However, pruning under this guidance may sometimes incorrectly remove important neurons with small magnitude and reserve unimportant neurons with large magnitude. Optimal Brain Damage (OBD) (LeCun et al., 1989) and Optimal Brain Surgeon (OBS) (Hassibi & Stork, 1992) propose to leverage the Hessian matrix of loss w.r.t. parameters estimate the importance of each parameter. The Hessian matrix contains second-order partial derivatives of the loss function w.r.t. all pairs of parameters, which is very computationally expensive to compute. Thus, OBD approximates as a diagonal matrix by assuming the loss of deleting several parameters is the sum of deleting these parameters individually. OBS views the pruning as an optimization problem and solves it with the Lagrange multiplier. They either discard approximate the second-order partial derivatives between all pairs of parameters, which capture the change of loss on one parameter when deleting another parameter.
Our Contributions In this paper, we follow the idea of OBD, leveraging Hessian matrix for pa- rameter importance estimation, Instead of approximating Hessian matrix, we calculate the Hessian- vector product element Σj & 50,50; for each parameter in the network. To achieve this, we first 39,00; separate the Hessian matrix of the whole network into Hessian submatrices between layers. Then, in the context of widely used network structures including convolutional neural networks (CNNs) analyze the conditions where the Hessian submatrices between two layers
and Transformers, are nonzero. Finally, we propose highly efficient method to capture these conditions and obtain the Hessian-vector product element on each parameter. Stepping from approximating the Hessian matrix with the Fisher matrix to directly computing the Hessian-vector product, we propose Optimal Brain Apoptosis (OBA), novel pruning method that efficiently calculates the second-order Taylor expansion for each parameter and is applicable to both structured and unstructured pruning tasks.
2 RELATED WORK
Model Compression Model compression is an area that focuses on creating smaller, faster, and more efficient models suitable for deployment in environments with limited resources, like mobile devices embedded systems. There are several typical fields within this area, including quanti- zation (Courbariaux et al., 2015; Rastegari et al., 2016; Pouransari et al., 2020), knowledge distil- lation (Hinton et al., 2015; Chen et al., 2021; Zhou et al., 2021), neural architecture search (Liu et al., 2018: Zoph & Le, 2016; Pham et al., 2018), and network pruning (Molchanov et al., 2019; 2016). Quantization, outlined in works like Hubara et al. (2018) and Jacob et al. (2018), focuses on reducing parameter precision to accelerate inference and decrease model size, enabling deployment on devices with limited resources. Knowledge distillation, as introduced by Hinton al. (2015); Romero et al. (2014), leverages a smaller "student" model to mimic a larger "teacher" model, effec- tively compressing the knowledge and achieving high performance with less computational demand. Neural Architecture Search (NAS), with seminal contributions from Zoph & Le (2016), automates the discovery of optimal architectures, often outperforming human-designn and accuracy. Pruning techniques, highlighted in work by Han et al. (2015), remove non-essential weights or neurons, significantly reducing model complexity and enhancing inference speed with- out major accuracy losses. Together, these techniques represent the forefront of model compression research, addressing the balance between performance and computational efficiency necessary for advanced AI applications.
models in efficiency
Network Pruning Network pruning, initially recognized as an importance estimation prob- lem (Molchanov et al., 2019; Chauvin, 1988; Yu al., 2018; He et al., 2020), has been prompting researchers to focus on finding accurate criteria that reveals the importance of parameters or neu- rons in neural networks. Molchanov et al. (2016) operated under the assumption that each layer in feed-forward networks held equal importance and introduced a heuristic for global scaling normal- ization. However, this approach did not prove effective in networks incorporating skip connections. Additionally, the method relies on using network activations to calculate its criterion, resulting in in- creased memory demands. In contrast, pruning methods that focus on batch normalization (Gordon et al., 2018; Huang & Wang, 2018; 8; Liu et al., 2017; Ye et al., 2018) bypass the need for sensitivity analysis and are applicable on a global scale. In intricate network architectures (Liu et al., 2021; Luo & Wu, 2020; You et al., 2019; Zhang al., 2021), parameter interdependencies often require their joint pruning. This collective pruning of interlinked parameters has been an area of focus in structured pruning research since its early stages. Fang et al. (2023) proposes build dependency
2

---

Published as a conference paper at ICLR 2025
graph which captures the interdependencies between parameters and prunes parameters belonging to a graph together to achieve structured pruning for neural networks with complicated structures. In pruning setting, view the pruning vidual parameter (unstructured) or parameter group (structured) obtained from Fang et al. (2023). Parameters with low importance are pruned in each pruning step.
importance estimation problem for each indi-
Hessian Matrix Estimation The structure and computational aspects of the Hessian matrix in feedforward neural networks have been extensively studied since the early 1990s (Buntine & Weigend, 1994; Wille, 1997). The Hessian matrix was first utilized in neural network pruning by LeCun al. (1989) to calculate the importance score of each neuron, leveraging diagonal approxi- mation is used to estimate the Hessian matrix:
8LOBD (07) H
Building upon this, OBS (Hassibi & Stork, 1992) views the importance estimation an optimization problem, and aims at finding a set of weights that yields least change on the loss:
min min 60
5000
50 H80 s.t. e, 0, 0 =
Early research such as that by Buntine & Weigend (1994) provided an extensive review of how to compute second derivatives in feed-forward networks, and Wille (1997) examined the Hessian ma- trix's structure and second derivative techniques. More contemporary efforts, such as EigenDam-- eigenbasis for network reparamcterization and
age (Wang et al., 2019), utilize Kronecker-factoonn approximate the Hessian matrix using the Fisher matrix, as discussed by Martens (2020). Recent studies (Wu et al., 2020; Singh et al., 2021) have thoroughly investigated the common structures and rank properties of the Hessian in neural networks. Pearlmutter (1994) initially introduced an efficient method for computing the Hessian-vector product in feedforward neural networks. Our research ap- plies this idea to the pruning of modern network architectures including CNNs and Transformers.
PRELIMINARY
Consider a feed-forward neural network with parameters 0 and L layers. Similar to OBD (LeCun al., 1989), when we add small perturbation 50 on 6, the second-order Taylor expansion of the perturbation on the objective function is given by
Oc T50 SC(0) a8
50 5 H50 + o(1150113).
2
where H is the Hessian matrix that represents the second-order derivatives between all parameter pairs. It is usually infeasible to compute the Hessian matrix due to its complexity of (n?)). where is the number of parameters in the network (LeCun et al., 1989; Hassibi & Stork, 1992). By expanding the first and second term of eq. (3), we can define the perturbation of the loss caused by 50;
ac 80, + Σ 00,
LC 00,00, 80,50, + o(1660,13).
SC((,)
(4)
The first term of eq. (4) is leveraged to estimate the improtance of neurons Molchanov et al. (2016). Same to prior works, we ignore the higher order term o(1)50,113) Current works (Hassibi & Stork, 1992; Yu et al., 2022; Benbaki et al., 2023) that leverage the second-order Taylor expan- sion term approximate the Hessian matrix with Fisher information matrix. This approximation, if aL dL 80,505. However, this approxi- applied to eq. (4), would change its second-order term to Σ; 00; 00, mation is not accurate enough to capture the second-orded loss perturbation caused by 80, and 50,- Therefore, we focus on theoretical analysis on how calculate the original second-order term &
ΣS 30,30; 50,50,-
METHOD
4.1 DEFINITION
We derive from a general form of linear layers in modern neural networks. For layer E [1, L), we denote the weight parameter W()) E Reom Xa Pasigt and bias parameter as ((0) Rlon, We denote
3

---

Published as a conference paper at ICLR 2025
the input of layer 1 as X E R and output of layer as Y E Xpur, Pweight Pout. and Pin are the length of flattened weights, output values, and input values that contribute to the connections between every pairs of input neurons and output neurons, for example Pweight 1, Pom 1, Pin
in the context of fully connected layers and Pweight = s "kernel' Peernel-Pout Pout = h ((())()))
(1)
out Pmn //20 win in the
context of convolutional layers. The forward propagation of layer / given by
y- = >>' Wilh x M + (
(5)
acd
cde
where M E {0, 1 Pesigte x Pie, determined by the layer itself, represents the connections among all input values, weight values, and output values. We denote the flattened vector of X W yO, and M as x0()) E Riie w E Rlorrwooour y()) Rleu and (!)) E {0, P Poogee Pe,
[w(() respectively. the parameters g(t) of layer can be expressed as Q()) (())
We represent all
partial derivative terms with variables in their vector forms to ensure these terms are with the same definition of Jacobian matrix.
In certain layer defined by eq. (5), the gradient of the loss w.r.t. the indexed parameters and ay!!! Ee x!! Ml and Es ac It clear that the condition for the bide.
and ( are respectively Eab ac
ay Hessian matrix entries between layer and any distinct layer I' being exclusively zero hinges on the &c and x()) with respect to the parameters of layer l', which depends
differentiability of terms Tyto on the connection type of two layer types. We observe that the connectivity types which introduce nonzero Hessian submatrices between any two layers in a neural network can be divided into two cases: series connectivity and parallel connectivity, as shown in fig. la. We next analyze these two 8 cases and calculate the Hessian-vectt vector product element Σj 80,00, 80,50, in eq. (4) for the two cases respectively.
4.2 SERIES CONNECTIVITY
Definition 4.1 (Series Connectivity). a neural network at layer I, if there exists a layer such that there is differentiable function mapping the output of layer l' to the input of layer we say layer and are in series connectivity. Specifically:
layer l' is in lower series connectivity to layer L.
layer l is in upper series connectivity to layer I'.
In fig. la, there are differentiable functions from ((4) to (122 and (1ss,, respecively, so layer 1 is in series connectivity with both layer l2 and layer lg. Without loss of generality, we take layer l and layer la as an example. Then
OC 0y
OC
yyyl)) dxll2)
drl(s) 0y
differentiable glig)
also (12) is differentiable to g(4)) According to our analysis in the beginning of this section, a nonzero Hessian submatrix exists between layers and l2.
Theorem 4.2. For layer in a neural network where layers lup E Lupp and layers Lio E Liow are in upper and lower series connectivity to layer respectively, then for weight parameter wil) and bias parameter 199 of layer we have
Σ
E
JCC
50
8w
Σ
OL
((a) (hol (w((
Swi
(6)
(0.0))n fwl())
By
Owe()
UL
L
OL ()
Tyll) Dwil)
X
in which ( is given by
(x (1)
X1)
Σ Σ
a0 50flum)) (lme))
Lowe
4

---

Published as a conference paper at ICLR 2025
Seres Cornectvvty
WF-
weight
Parameter Layer
VV
Fomard Inpul
Forward expal
Fonerrs
Panalel Connectivy
-
JuPF Feature
Seriss Connectvioy
ocoblan Matrix
weight
Nonparameter Layer
Figure 1: (a) An illustration of conditions where the Hessian matrix between parameters of two layers are nonzero. (b) An illustration of Jacobian- Vector Product Forward Propagation. Two for- ward propagation processes are needed for parameter layers and one forward propagation process needed for nonparameter layers. For nonparameter layers leverage Jacobian- -eecto product to conduct the forward process and do not need to calculate the Jacobian matrix explicitly.
and
OC V'EnuUiis 00 ane
Σ Σ
00 o 56
dL {( dr(w)) Sy(lw) (W O60) @ 51
(9)
EL
where (1) Swo; E (ale ma) is the jacobian matrix of (0) with respect ( taking (W()) as the weights, and (y Owilt &() is the jacobian matrix of (D)) w.r.. w taking X() as input.
The proof is provided in appendix E.I. Note that the sets (!) and Loo (l) are dependent on layer I, and abbreviated for simplicity. With theorem 4.2, the second-order term of the Taylor expansion the loss with respect to each individual parameter can be computed, taking into account parameters belonging layers that in series connectivity network structures such as convolutional neural networks and fully connected neural networks, there only exist series connectivities between layers. Thus, we can directly apply theorem 4.2 to calculate eq. (4) for each individual parameter.
specific layer. For classical neural
4.3 PARALLEL CONNECTIVITY
For recent novel neural network structures such as Transformer (Vaswani et al., 2017), matrix mul- tiplication plays a crucial and effective role in achieving their impressive performance. It also in- troduces parallel connectivity to layers and lead to the nonzero Hessian matrices between these connected layers.
Definition 4.3. In a neural network, if there exist two layers and such that there are differentiable functions respectively mapping the outputs of layer I and I' to the inputs ( and X tright of matrix multiplication operation (mul)) X (let)) xiright i.c. layers and are both in lower series connectivity the matrix multiplication operation, we say layer I and l' in parallel connectivity. Denote the multiplication operation as layer 4m. then
layer is in left parallel connectivity to layer P!.
layer I' is in right parallel connectivity layer 1.
In fig. la, layer la and layer l3 are in parallel connectivity. Similarly, the gradient of loss w.r.. By (4) (52)))
(4) differentiable to g(ls) and
parameters of layer 1z is ag Ty a2) ag01)) in which dz
vice versa. Therefore, the nonzero Hessian submatrices resulting from parallel connectivity should also be considered.
Theorem 4.4. For multiplication operation (imll) (1e)) (right) in neural network, where X(lef)) E Blrww xhyss and (right) Reeaa layers I E Lug and I, E L are in lower series connectivity to this multiplication operation, respectively. Consider surrogate weight matrices
5

---

Published as a conference paper at ICLR 2025
X(lel) E RRlow xlasa and (
E R
Rlasixlent given by:
X
Σ
Σ
50(1))
(10)
)0(")
X
(right)
Σ
50
(11)
a0fi>
Then we have
ΣΣ GCC
OC (yfmmu)
right))
50 q
o 500))
3 5001.)
(12)
: oga.)
drf(mul) (ight)
A (g(,))
and
OC
Ory O
E 1e L. a0 o011))
(0(+))
c 6004)
+ 8004)
(13)
Ory(mull Ozllee))
Ktriako
Proof can be seen in appendix E.2. Now we can analytically caleulate the Hessian-vector product element of each parameter specifically focusing on the interaction between its layer and any other layers that are parallel connectivity to it. Combining theorem 4.2 and theorem 4.4, we can caleu- late the -essian-vector product element of each parameter in neural network. Next, we focus on an efficient calculation the Hessian-vector product element of each parameter.
4.4 VACOBIAN-VECTOR PRODUCT FORWARD PROPAGATION
From a computational overhead perspective, the main calculation part within theorem 4.2 and the- orem 4.4 focuses on (0), the gradients of X w.r.t. the parameters of all layers that are in lower series connectivity (1), as we need to separately back-propagate each entry of ( usually a large matrix in attention modules. To address this issue, we introduce Jacobian Vector Product Forward Propagation (JVPF), method capable of computing (1) for all layers with acceptable computational expense.
Let's look at a layer in lower series connectivity to layer / and denote it as layer how The derivative of (0) w.r.t. the parameter (1amm a single layer low multiplied with 50(mm) can be expressed as
(1)
(x(())
(hn+ Ox (g
( (0(lm)
Oz(1-1] ax(lun)) 00(
indicating that we can calculate it in layer-by-layer manner. Further, let group layers in lower series connectivity to layer I into several groups, each of which contains both parameter layers and nonparameter layers that are in series connectivity to each other. We denote the ith group as G; = (41442 ... ,N,, J. where N; is the number of layers in group G; and layer 4(j+1) is subsequent to layer lyy. Then we have
0x
Ox(l))
Σ Σ
Σ
fllined flan, 19) o .... ((l)((0)
y(lx
Em
where
50(l)s)
+
Oy (15)
lig has parameters,
(x))
(14)
Bn Dz (2y x
else.
(l1), y(lss), and (54es) are the input, output, and parameters of layer 4g. Once we replace the forward function each layer with eq. (14), we could calculate ((() for each layer / through one forward propagation process, and all series connectivity groups be caleulated in parallel. An intuitive demonstration of JVPF is shown in fig. 1b.
4.5 PRUNING STRATEGY

---

Published as a conference paper at ICLR 2025
Utilizing theorem 4.2 and theorem 4.4 along as our proposed JVPF, which offers an effi- cient choice to obtain the essential intermedi- ate values, we can calculate the Hessian-vector product element of each parameter their im- portance scores with several batches of traning data. Please refer appendix B for the de- tailed importance score acquisition. This im- portance score can be leveraged conduct un- structured pruning for each parameter individu- ally, or conduct structured pruning for each pa- rameter group.
Upper
Upper Waght insertance Score
Upppar Gathennn
Lower
Lower Winght
Gutloring
insa
impersanc
Reurtes of Lowerlage Lager
Lawer
Figure 2:
Importance score of each neuron in group is gathered from parameters of lower layers and upper layers.
Structured Pruning Following Fang (2023), parameters from in-layer and inter- layer connections can be organized into several groups
The importance scores belonging to each group is defined as Ia [ (gs E G}. For each group G, importance scores are summed on every neuron over parameters upper layers L, C gi with length Nr, and lower layers L C 9i with length Nt, as illustrated in in fig.
Unstructured Pruning The importance score unstructured learning is more straightforward. By eliminating the process of gathering importance, considering each parameter layer as group, and each parameter as neuron, the definition of importance score becomes similar to what described in structured pruning scenarios. Experimentally we found that the gradients of some parameters are zero due to gradient vanishing, making us hard to judge the importance of these parameters. This would have little influence on structured pruning since neurons' importance scores are gathered through multiple weights. However, unstructured pruning, the importance score of each weight only depends itself. By adding the magnitude of the corresponding weight in the importance score term, the problem is resolved.
In each pruning step, calculate every importance score I of jth neuron in ith group over traning data of batches, and normalize them within each group. Then we rank them from lowerst to highest. The lowest p percentage of parameters pruned. We can gradually increase p to iteratively prune the model to a specific FLOPs or parameter percentage.
5 RESULTS
We empirically study the performance of OBA on CNNs where only series connectivity exists, and attention networks where both series connectivity and parallel connectivity are present. We focus on pruning towards as small FLOPs as possible reduce the computation overhead of model inference. For structured pruning, we primarily compare our methods with those that leverage Hessian matrix information (Hassibi & Stork, 1992; Wang al., 2019). the context of unstructured pruning, evaluate approach against the state-of-the-an unstructured pruning method, CHITA (Ben- baki et al., 2023). Classical importance acquisition methods Weight (the magnitude of weights), OBD (LeCun et al., 1989) and Taylor (Molchanov et al., 2016) are also added into comparison for both structured and unstructured pruning tasks. our experiments, we choose 50, = 8, The implementation details can be found at appendix A.
5.1 STRUCTURED PRUNING
Current structured pruming workflows be roughly divided into one-shot pruning and iterative pruning. The former prunes the fine-tuned model towards sparsified network and fine-tunes it after pruning, whereas the latter prunes the model iteratively and fine-tunes the model after each pruning step. One-shot pruning is more efficient than iterative pruning, but the latter is more effective. We evaluate our method on both of these two workflows.
5.1.1 IMPORTANCE SCORE RANK CHARACTERIZATION
We first measure the ability of our method to characterize the importance of each neuron for struc- tured pruning. Intuitively, the importance score of neuron should he positively correlated to the

---

Published as a conference paper at ICLR 2025
Table 1: The Spearman correlation of the importance Table 2: Comparison on Accuracies (%) scores of each method with the importance rank of and speed up up ratios with recent pruning works on ImageNet.
neurons on ResNet32.
Method
OBA
Weight
Taylor
OBS
OH1>
HigenlDunage
Method
Pruned
Speed Up
all layers
CIFAR100
Max
0.500
0.115
0.457
0.248
0.222
-0.077
Resnet50
Standardization
0.496
0.335
0,450
0.292
0.474
-0.047
Weight
75.12 1.0
1.99x
Norm
0.192
0.433
0.441
0.407
0.541
-0.012
C-OBD (Wang et al., 2019)
74,86
1.99x
None
0.444
-0.247
-0.157
0.222
0:275
0,046
C-OBS (Wang et al., 2019)
76.48-067
2.01x
per layer
0,443
0.292
0.304
0.287
0.018
EaganDamage (Wang ct al., 2019)
75.30
2.00x
Taylor (Molchanov et 2016)
2,00x
all layers
CIFARIO
OBA(Ours)
75.62 p.Gs
2,00x
Max
0.446
0,092
0,413
0.373
0.426
0.165
Standardization
0.382
0.049
0.444
0.412
0.166
VIT-B/16
dy-marm
0.402
0.443
0.433
0.367
0.186
0.192
Weight
77.03 -.04
1.32X
None
0.379
-0.187
0.080
0.360
0.342
0,172
Taylor (Molchanov et al., 2016)
77.65-3.42
1.31x
per layor
0.421
0.289
0.409
0.362
0.347
0.056
OBA(Ours)
79.64
1.30x
change of loss when removing a neuron. This can implemented by masking the neuron's output in the evaluation process. Specifically, we mask the output of each neuron in pre-defined layers and compare the change of loss between the masked model and unmasked model, which refered to as ground truth importance. Then calculate the Spearman's 's rank correlation of the importance scores with estimated importance scores from different methods. The higher the Spearman's 's rank score, the better the method captures the importance. We select the output neurons of the first con- volutional layer and the three following residual blocks of ResNet32 (He et al., 2016) as candidates. For per-layer case, Spearman correlation is calculated within each layer and averaged. For all-layer case, before calculating Spearman correlation, we first normalize the importance scores of each layer and concatenate them together. Different layer-wise normalization methods are considered in the evaluation and are detailedly introduced in appendix C.
As shown in table 1, with evaluation within each layer, OBA yields the best rank similarity to the ground truth importance among all methods. In the all-layer importance condition, our method also yields good performance on different normalization methods. is noteworthy that methods such as Weight, Taylor, and OBD, although potentially exhibiting higher correlation scores in importance under specific normalization schemes, may also yield notably low correlation scores under certain normalization techniques. In contrast, OBA consistently maintains relatively high correlation scores irrespective of the normalization method applicd. This proves our method's 's capability of importance capturing, along with its robustness to different normalization methods.
5.1.2 ONE-SHOT PRUNING RESULTS
Next, we conduct experiments to evaluate the performance of OBA on pruning towards a specific FLOPs percentage in one-shot pruning workflow. As the FLOPs and number of parameters are not linearly dependent w.r.t. the number of neurons in the neural network, it's hard to calculate pruning ratio under which the network is pruned into a predefined FLOPs or parameter number. Thus prune the network for several steps with increasing pruning ratio. FLOPs and number of parameters are calculated after each step to check whether the network satisfies the target. We first evaluate our method on ImageNet with ResNet50 (He et al., 2016) and ViT-B/16 (Dosovitskiy al., 2020). As shown in table 5, our method realizes a 2x speed up at ImageNet on ResNet50 with an accuracy decrease of only 0.53%. Our method outperforms other methods on both ResNet50 and ViT-B/16, which demonstrates the effectiveness of our method on large-scale datasets. On ViT-B/16, our method achieves a 1.30x speed up with an accuracy decrease of 1.43%, which is far smaller than those achieved by Taylor criteria and Weight criteria, demonstrating our proposed criteria's 's superiority over Weight criteria and the first-order Taylor criteria.
Next we evaluate our method on CIFARIO 10 and CIFAR100 datasets. The results are shown in table 3. Our method achieves the best results on ResNet32 model across the both datasets. When pruning network towards small 6% target FLOPs, our method on ResNet32 surpasses other methods a lot, with only 0.79% accuracy loss on CIFARIO datasets. On CIFARIO with VGG19 the performance of OBA is slightly worse, but still comparable to other methods, with relatively lower FLOPs. We observe an interesting phenomenon that the parameter reduction of models pruned with OBA are slightly higher than other methods under similar FLOPs, which indicates that method tends to prune neurons with relatively higher FLOPs, and is more suitable for applications of lower FLOPs requirements.
8

---

Published as a conference paper at ICLR 2025
Table The accruacies (%), weights reduction (%), and FLOPs reduction (%) of different methods across CIFARIO and CIFAR100 under one-shot pruning with ResNet32 and VGG19.
Method
CIFART0
CIFAR100
Weights
FLOP%
Acc
Weights
Acc
Weights
Acc
Weights
VCG19/
94.17
13.34
NN Slamming (Lib et al., 2017)
92.84
80:07
42.65
85.01
97.85
97.89
11.89
14.60
38.33
58.69
97.76
94,09
C-OBD (Wang 2019)
94.04
82.01
38.18
92.34
97.68
77.39
72.23
77.03
33.70
58.07
97.97
77.55
(Wang al. 2019)
34.73
12.27
13.83
58.87
97.61
Krop- OBD (Wang et al.. 2019)
94.00
$0.40
38.19
92.92
97.47
81.44
72.29
77.24
37.90
60.70
97.56
82.55
Krom OBS (Wang et al., 2019)
91.09
79.71
36.93
92.56
97.53
90.39
72.12
14.18
36.50
60.66
97.48
83.57
EigenDannge (Wang
2019)
93.95
78.18
37.13
92.29
97.15
86.51
72.90
76.61
37.40
65.18
97.31
88.63
Weight
96.93
50
Taylor (Molchanov et al. 2016)
94.11
62.29
38.28
92.29
93.89
86.29
71.82
55.58
37.76
66.65
93.12
$8.28
OBA (Ouns)
93.9
56.63
38.06
92.48
91.89
86.27
72.36
53.33
37.62
66.72
92.74
$5.80
Reseei33(B
95.3
78.17
C-OBD (Wang all...2299)
97.30
C-OBS (Wang at, 2019)
76.75
90.04
95.
97.39
Kron-OBD (Wang et al., 2019)
95.11
63.97
63.41
92.57
96.11
94.18
75.96
63.92
62.97
62.42
96.42
95.85
Kron OBS (Wang et al., 2019)
95.14
64.21
61.89
92.76
96.14
94.37
75.98
62.36
60.41
63.62
93.56
95.65
RigenDumage (Wang
70.25
93.05
94.74
95.21
94.62
Weight
71.99
92.07
91.61
75.72
68.
66.09
95.44
Taylor (Molchanov et al. 2016)
95.06
06.57
71.40
93.32
94.88
94.05
76.13
64.27
70.11
64.56
90.35
94.99
OBA (Ours)
95.19
63.90
71.13
93:45
93.68
91.51
76.47
64.34
70.02
67.81
91.43
94.40
CRA1100
CEARS0
(398115
5.75
8:10
333
E Centage
Chitage
Wrigh
Wright
24
25
Figure 3: Iterative pruning results CIFAR10 and CIFAR100 with ResNet32.
5.1.3 ITERATIVE PRUNING RESULTS
In this subsection, we evaluate the performance of OBA under iterative pruning workflow on ResNet32. Specifically, we set a list of target FLOPs for each dataset and model, and iteratively prune the model from largest FLOPs to smallest FLOPs. Fine-tuming is conducted after each prun- ing step. In terms of FLOPs, the accuracy loss for each iteration of our method is lower than other methods, as shown in fig. 3. This validates the effectiveness of our proposed criteria. As for param- eter reduction, our method yields less advantage over other methods, which is consistent with the results of one-shot pruning.
5.2 UNSTRUCTURED PRUNING
We also evaluate the performance of OBA on unstructured pruning task. We follow a similar setting of the multi-stage pruning in CHITA (Benbaki et al., 2023) to have a fair comparison. Since Tay- lor (Molchanov et al., 2016), Weight, and OBD (LeCun et al., 1989) all obtain the importance scores can ignore their importance gathering steps and add them into comparison. The results are shown in table 4 and table 5. Given the varying performance of the initial unpruned networks, we directly compare the accuracy ratio relative the raw accuracy of all methods. The main version of CHITA, i.c. CHITA-CD, has very large computational time and memory cost on Resnet50, making itself infeasible for such huge networks. Thus we implemented the more effi- cient CHITA-
from each parameter,
for comparison. It can be seen that Taylor and OBD fails on this task as their accuracies rapidly fall into 10% in the first pruning step. OBA's results on high sparsities surpass CHITA+ by a huge margin, proving itself to be effective in unstructured pruning task.
5.3 RUN TIME ANALYSIS
Here, empirically study the time consumption of OBA. Table 6a shows the time consumption of each part of OBA, and the time costed by regular training. It can be seen that the time cost of computing parallel connectivity is the most time-consuming part of OBA, nearly same to the time series connectivity. In network structures that do not contain multiplication operations, the time cost of OBA would be much lower. In table 6b, 200 batches of data with batch size of 64

---

Published as a conference paper at ICLR 2025
Table 4: The unstructured pruning results on CIFAR-10 dataset with ResNet-20.
Spursity
Taylor (915))
OBD (9158)
Weight (91%)
OBA (918)
CHITA (91.3696)
Aecuracy (%)
Ratio (1)
Accurscy (5)
Ratio (%)
Accuracy (5)
Ratio (3))
Accuracy (5)
Ratio (%)
Accuracy (98
Ratio (16)
0.1
11.00
14.45
10.03
13.17
90166
99.63
90.85
99.81
11:00
14.45
10.03
13.17
90.90
99.89
0.5
10.00
13.14
10.01
13.15
90.79
99.77
90.57
99.53
91.04
99.65
0.6
8.20
10.27
10.00
13.14
90.23
99.15
90.69
99.66
90.78
99.37
10.00
13.14
10.00
13.14
88.83
89,94
98.81
98.93
97.11
Table 5: The unstructured pruning results on Imagenet dataset with ResNet-50.
Sparsity
Taylor (76.13%))
OBD (76.13%)
Weight (26.13%)
OBA (76, (3%)
CHITA-BSO (77.01%)
Avcuracy
Accuracy
Aucuracy
(3)
Accuracy
Aucuracy
77.00
0.11
0.14
0.10
0.13
75.23
98.82
15,40
99.04
76.91
99.87
0.3
0.10
0.13
0.10
0.13
74.57
97.95
74,93
98.42
76.87
99.82
0.1
0.11
0.14
0.10
0.13
73.54
96.50
74,50
97.86
76.59
99.46
0.13
0.10
0.13
20.73
73.42
96.44
76,01
98.70
0.10
0.13
0.10
0.13
48.09
63.17
65.57
86.13
64.38
83.60
0388
0.10
0.13
0.09
0.12
16.08
21.12
47.14
61.92
26.21
34,03
0.9
0.10
0.13
0.10
0.13
0.50
1.05
5.65
7.43
0.43
0.566
Time tsipruning Iteration)
(s/pruning Iterationn
Resuct32
VIT-B/16
300
Regular Traning
0.326
0.252
Upper Series
0.881
0.799
200
150
Parallel
1.944
100
Total
2.072
4,027
50
(a)
Taylor
080
065 EigenDamage 0BA
CHITA 850 CHITA-CO CIITA-CO
OBA
(c)
Table 6: The running time (s) of OBA different models and datasets. (a) The specific running time (s/iteration) of each part of OBA on ResNet32 and ViT-B/16. (b) The running time of OBA on ResNet32 with CIFAR100. (c) The running time of OBA ResNet20 with CIFAR10.
are leveraged in each pruning iteration. In each pruning stage we conduct pruning iterations to gradually prune the network, and the overall computation time of pruming would be around 4.2 hours, which is acceptable in the actual pruning scenarios. In table 6c, we samely use 200 batches of data calculate gradients. The mainly proposed method in CHITA, CHITA-CD, would require twice as much time compared to our approach. However, the performance of both CHITA-CD and CHITA-BSO under high sparsity worse than OBA, showcasing the efficiency of our method.
CONCLUSION AND LIMITATION
In this paper, we propose Optimal Brain Apoptosis, novel method for pruning neural networks. We first provide theoretical analysis on modern neural network structures to figure out nonzero Hessian submatrix conditions between layers. Then we propose an efficient approach that directly calculates the Hessian-vector product values for each parameter in the network, thereby calculating the second- order Taylor expansion for each parameter without any approximation. We empirically demonstrate the efficacy of our method on both structured pruning and unstructured pruning.
Limitation OBA, in its current form, can be applied to network structures including MLPs, CNNs, and Transformers. For networks with more complex architectures, like RNNs and State Space Mod- els that handle time-series data, computing the Hessian matrix becomes more difficult and necessi- tates additional research. This is interesting that warrants further exploration in the future.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
We sincerely appreciate the dedication and valuable feedback provided by all anonymous reviewers. This research partially supported by the National Natural Science Foundation of China under Grants 62073066 and 62405255, as well as the Fundamental Research Funds for the Central Uni- versities under Grant N2226001. Additionally, it received support from the 111 Project under Grant B16009 and the Intel Neuromorphic Research Community (INRC) Grant Award (RV2.137.Fang).
REFERENCES
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems (JETC), 13(3):1-18, 2017.
Riade Benbaki, Wenyu Chen, Xiang Meng, Hussein Hazimeh, Natalia Ponomareva, Zhe Zhao, and Rahul Mazumder. Fast as chita: Neural network pruning with combinatorial optimization. In ICML, pp. 2031-2049. PMLR, 2023.
Wray L Buntine and Andreas S Weigend. Computing second derivatives in feed-forward networks: A review. IEEE transactions on Neural Networks, 5(3):480-488 1994.
Yves Chauvin. A back-propagation algorithm with optimal of hidden units. Advances in neural information processing systems, 1, 1988.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge re- view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5008-5017, 2021.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Train- ing deep neural networks with binary weights during propagations. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural In- formation Processing Systems, volume 28. Curran Associates, Inc., 2015. htpps://proceedii 3e1 5cc1 Ssccl1f979 (t
URL
neurips. wwtppppr_ailleeeeeeeeeeeeeeepppe ptses/pppe//1////11 Foweer5000000d0eeeee pf
Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in neural information processing systems, 30, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929,, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. URL https: tttetts:/oppenee wwt.//wwww.....eeeeeeeeee n
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In CVPR, pp. 16091-16101, 2023.
Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi. Mor- phnet: Fast & simple sssource-constrannee structure learning of deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1586-1595, 2018.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. NeurlPS, 29, 2016
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. NeurlPS, 28, 2015.
Stephen Hanson and Lorien Pratt. Comparing biases for minimal network construction with back- propagation. NeurIPS, 1, 1988.
11

---

Published as a conference paper at ICLR 2025
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain sur- geon. In NeurIPS, 1992. URL https::/pproceeddgggg neursis cttperippsccccpapeeeeeo ccppaper.ppppep.p/ppppeo files/ paper/1992/ wttee/eeeeeeeeee http:///a......eecaaaaaaaaaaaaaa
pdf.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, June 2016.
Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter prun- ing criteria for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2009-2018, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1rXiv:1503.022 02531, 2015.
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pp. 304-320, 2018.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations, journal of machine learning research, 18(187):1-30 2018.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arihmneti-onnly inference. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713, 2018.
Alex Krizhevsky, lla Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. NeurlPS, 25, 2012.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In NeurlPS, http/ppper_f.llspppe///////
1989. URL https::/prroceed.ngs..t neurips http:///8222aacc1001000011111110
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2288-2322, 1998.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceed- ings of the European conference on computer vision (ECCV), pp. 19-34, 2018.
Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen. Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learning, pp. 7021-7032. PMLR, 2021.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pp. 2736-2744, 2017.
Jian-Hae Luo and Jianxin Wu. Neural network pruning with commecal-oonntttions and limited-data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1458-1467, 2020.
Xinyin Ma. Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. arXiv preprint arXiv:2305. 2023.
James Martens. New insights and perspectives on the natural gradient method, 2020.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In ICLR, 2016.
Pavlo Molchanov, Arum Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1 11264-112 1064-112777 1272, 2019.
12

---

Published as a conference paper at ICLR 2025
Barak A Pearlmutter. Fast exact multiplication by the hessian. Neural computation, 6(1):147-160, 1994.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pp. 4095-41044 PMLR, 2018.
Hadi Pouransari, Zhucheng Tu, and Oncel Tuzel. Least squares binary quantization of neural net- works. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 698-699, 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks, In European conference on computer vision, pp. 525-542. Springer, 2016.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: 2014.
Sidak Pal Singh, Gregor Bachmann, and Thomas Hofmann. Analytic insights into structure and rank of neural network hessian maps. Advances in Neural Information Processing Systems, 34: 23914-23927, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. Eigendamage: Structured pruning in the kronecker-factorred eigenbasis. In ICMI,, pp. 6566-6575. PMLR, 2019.
Jörg Wille On the structure of the hessian matrix in feedforward networks and second derivative methods. In Proceedings of International Conference on Neural Networks (ICNN'97), volume 3, pp. 1851-1855. IEEE, 1997.
Yikai Wu, Xingyu Zhu, Chenwei Wu, Annic Wang, and Rong Ge. Dissecting hessian: Understand- ing common structure of hessian in neural networks. arXiv preprint arXiv:2010.04261, 2020.
Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the smaller-ormm-les-inornatiii assumption in channel pruning convolution layers. arXiv preprint arXiv:1802.001124 2018.
Seul-Ki Yeom, Philipp Seegerer, Sebastian Lapuschkin, Alexander Binder, Simon Wiedemann, Klaus-Robert Müller, and Wojciech Samek. Pruning by explaining: A novel criterion for deep neural network pruning. Patter Recognition, 115:107899, 2021.
Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter prun- ing method for accelerating deep convolutional neural networks. Advances in neural information processing systems, 32, 2019.
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9194- 9203, 2018.
Xin Yu, Thiago Serra, Srikumar Ramalingam, and Shandian Zhe. The combinatorial brain surgeon: Pruning weights that cancel one another in neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18666-25683., PMLR, 17-23 Jul 2022. URL https: http://///ceeggeeeeeeeeeee mmlr. press/vis v1 62 f22.f...... html.
Yulun Zhang, Huan Wang, Can Qin, and Yun Fu. Aligned structured sparsity learning for efficient image super-resolution. Advances in Neural Information Processing Systems, 34:2695-2706 2021.
13

---

Published as a conference paper at ICLR 2025
Sheng Zhou, Yucheng Wang, Defang Chen, Jiawei Chen, Xin Wang, Can Wang, and Jiajun Bu. Distilling holistic knowledge with graph neural networks. In Proceedings of the IEEE/CVF inter- national conference computer vision, pp. 10387-10396, 2021.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint arXiv:161..0157,, 2016.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A IMPLEMENTATION DETAILS
In our structured pruning experiments, we align our settings to EigenDamage (Wang et al., 2019), which is a good Hessian based pruning method. For each pruning step, we obtain the importance score from 200 batches data to calculate the importance score. We set the batch size 64 and the learning rate 0.001 for the fine-tuning process. We use the SGD optimizer with momentum of 0.9 and a weight decay of x 10- The learning rate is divided by 10 at the 80th and 120th epochs. We set the maximum epochs to 150 for CIFARIO and CIFAR I00 datasets. For ImageNet experiments, we use the same settings as Fang et al. (2023) of ResNet50 and ViT-B/16. We set the maximum epochs to 100 for ImageNet experiments.
Algorithm Importance Score Acquisition of OBA
Input: model m and its parameters 0, batch of data D Output:
parameter importance I
Initialized importance dict
2: Conduct one forward propagation and back propagation process on model m with data D and for each layer
record the output gradient Byy
3: for parameter layer in m do (...bckward(( 0c (()
4:
0g kwa))
5: end for
6: for parameter layer / in m do
7: I[[] + 50(1) @ (0)0_((ra) grad
8: end for
9: Conduct JVPF according to eq. (14) and record (1) for each layer
m.zero grad()
11: for parameter layer I in m do
12: Z[4 + I[4 + ac dy
(y 20τт [ 3 500)
13: if layer is attention module then
$ softmax (( R
Qth (()TT v(2))
14:
softmax (etiaxx((11) (Q") (1)) d())
Ô SV + SV SV ac
17:
O. b.bacwward( 80
end if
19: end for
for parameter layer in m do
I[|] + I[| + 50() g00) grad
22: end for
23: m.zero m.zero.grade grad()
B ALGORITHMIC DETAILS
As can be seen in algorithm 1, We first conduct forward propagation process on the network and record the output gradient the loss w.r.t. the output of each layer (line 1-2). Then we back- propagate these gradients for each parameter layer with corresponding weight 50 to obtain the term 8'c (!') (line 3-5).
EVeT Ej D0.""a000
1. Upper Series Connectivity: These terms recorded in the gradient of the corresponding parameters, so that we can obtain Evet. E3 a²c 20- @ 80()) in eq. (6) by multiplying the aaoi
parameters with their gradients (line 6-7).
15

---

Published as a conference paper at ICLR 2025
Next, We obtain X = Eim Σn
ax ((an) for all parameter layers through the JVPF,
these values are useful for calculating Lower Series Connectivity cases and Parallel Connectivity cases (line 9).
3. Lower Series Connectivity: We obtain the latter term of eq. (6) and add them into the importance scores (line 12).
4. Parallel Connectivity: In the meantime, for all attention layers that induce parallel connectivity, back-propagate the gradient with the surrogate inputs X and Xright according to eqs. (12) and (13) (line 13-17), and multiply the parameters with their gradients that are Lower Series Connectivity with the attention layer (line 20-21).
C NORMALIZATION METHODS
In our implementation, We leverage these normalization methods on importance scores for each group, including:
No Normalization (None) When the normalizer is set to None, the original values of importance scores are returned without any modification. This means that the data is used as-is, with all its original properties (such as scale and distribution) intact:
yoormaliedd Ig-
Standardization (or Min-Max Normalization) This method scales the data that it fits within a specific range, typically 0 to 1. This is achieved by subtracting the minimum value of the data and then dividing by the range of the data:
I min(T) max(1) min(7)"
pnormalized
Max Normalization In this approach, every importance score is divided by the maximum im- portance score of corresponding group to ensure that all the normalized values fall between 0 and
1:
I
I porralliedd
max(I)
Normalization This method normalizes the importance scores by dividing it by the l2 norm (Euclidean norm) of the importance scores belonging to the same group. The l2 norm is calculated the square root of the sum of the squared values:
1
Inormatized
11/112
ResNet32
VGG19
10P
10a
CBA
Tayfor
10a
109
CIC
CED
FLORS
107
107
10"
Taylor
085
o3n
10a
DownDamage
15
20
25
Layer Index
Layer Index
Figure A1: The layer-wise FLOPss for all parameter layers from models pruned by different criteria.
16

---

Published as a conference paper at ICLR 2025
D PRUNED LAYERS VISUALIZATION
We visualized the layer-wise FLOPPs of pruned models by OBA and other methods for ResNet32 and VGG19 on CIFARIO0 with target FLOPs of 6%. It can be seen that compared with other methods, the differennce of OBA between FLOPs of different layers is smaller, resulting a smoother model in terms of number of heurons across layers. This significantly helps to improve the model's performance across various datasets and provide useful guidance for researchers to design and prune neural networks.
E PROOFS
E.1 THEOREM 4.2
For any two layers how and hup in series connectivity, where lup is upper than how- Note occ that Pwllumabbiig is a zero matrix because parameters w(hw) and 6(hp) are independent of We first calculate term
each other. With this prior, 00 d (w) is actually
Oc uppt 00 ow,
Σj aof/um) Breghne
AC 50/4m) swofh) for the lower layer how
Lemma E.
O
sw 8w((m)) thi
OC (lw)
Ox(l
SW and
(15)
aw the thi thi aed
aw!h.
acd
0y((m)] sW
ow acd
and
O2L (m.))
OL ((a) @x!hor
Σ
Sw (ho)
fhi 5b(1em)
((lmm))
(16)
Thi ow (ha) ) shi E (hw ma) is the jacobian matrix of yow) with respect to alwe) taking
Ty(l (w
ab (are)
where (he) sw (he) as the weights.
Proof. Let J (1p)) Wlla) Oull
The element-wise derivative of loss w.r.t. ( is given by
OC
Σ OL
Wille) (up) the
(17)
(x(1w)) hj
oy fg
fo
ay Jup' fax!1
and the element- wise derivative of loss w.r.t.
((w)) is given by
dc
Σ
Oc
x (up)
M Mllly)
(18)
Uhy ow uup
0Y!lu))
OY
(pa))
hj
fhi
Applying chain rule, we have
(hl ow!b Ox!(aa)
(bap
Σ
JC
Sw ow Shi and
Σ
OC
Σ
Thi aw!lu Uhe) oww) fhi acd
ay (ww)
giz
shi
aw acd
aed
f9
hij
dC
((l) (xp)
5w Owml acd
(19)
(yy (wp)
and
and
Σ Mllly) sw (ww)
OL Σ Cnp (law) fhi
(we Stllmm))
Σ
dc
shi 2D (mm) Bllm)
Thi aw aw
shi
19 ig
hāj
OL ((p)
(20)
wr) (w)) (l
17

---

Published as a conference paper at ICLR 2025
Next we obtain term Ei
8
50, 50! 8uu(lo) for the upper layer lup
00 (we) aw,
(hup)
Lemma E.2.
OC C
ow (bx) ths
50
Oc
Oylly)
oW (w))
(21)
ow (op) Hii
Tylly) ow
shis top
fhi
where hagd is given by
8 Σ
X (llddon
hj (
50(1an)
(22)
bj
Proof.
Ox (heg
OC
sw((mn) thi
dL
Σ
M M gij
Σ
hj a0!h
50 Mo
aw!lwa) thi 81mm
ay (ww)
fhi
19
ax
(tw) hJ
OC
Muwy
50
sw
(ag fhi
Oy Bug)
9i3
ag!h
19
C
50 @g( (wl))a) (1m)
(23)
aw
(up)
)0! shi
fhi
Proof of theorem 4.2. Sum the results in lemma E. for upper series connectivity cases with results in lemma E.2 for lower series connectivity cases and we get
Σ Σ
OC (1)
50l")
Σ
ac (ww) Oy(hip sw
ow
11
Oc
0y
Swl (
ow?
Ow: (
dy
ow (1)
ULn
ELp
(24)
with
ox((!)
x
Σ
(lidko))
Σ Σ
50 (lum))
hi
bim Eai
Inn Eliime
and
Σ
Σ
JL
(00))
86
Σ
Oc
J ((l)] dxll
86
(25)
!0!
2b!")
Ty(hip uw
06
PELLUUL
Rewrite eq. (24) and eq. (25) as vector forms and we obtain eq. (6) and eq. (9).
Appendix
E.2 THEOREM 4.4
Here follow the notations in definition 4.3 to denote each value of the layers.
Lemma E.3. Let ((leRt E Xlhess and X (right) E IR RRlnaalloo such that
be two surrogate weight matrices
(x
f kn
6014).
(26)
X
friigh
ao(l_) 50(1.).
(27)
18

---

Published as a conference paper at ICLR 2025
Then have
82
ac (yfmml))
60(1-) (00())
601
(28)
a0.1))
00111)
4
Dyfmmul) righigh
& a0
and
O2C
dL dylmmu)
Oxfi 50())
50
50
(29)
ael
00
U1)
Dyfmmul) Oz(((ft
&immr 08, (4)
9
Proof. The second-order partial derivative of loss w.r.t.
of!h)
and (()))
can be written as
OC
dc
ax x a a
Σ
Σ
ko
(30)
- (0!1)
(1)
ay-iuui
x kn
ox
(rihh
a0d
an
ko
up
po
Since the multiplication can be expressed as (mul)
En
(right which means
ko
kn
y2yr(mul)
1 n P.
(31)
0x
0 X
0 n +
With this, eq. (30) can be rewritten
82
Oc
X x
(32)
aytmull
00.
00
ko
n
By expanding the right-hand side eq. (28) we have
OC ay(amull
(right) ao(i.)
Σ ko
Σ
(g(L.)
kn
50(1)
(33)
Oc
@x
(rightt
ax kn
Σ
80(())001)
mull)
an
ko
UC
00
(h)
80
(-)))
9
(34)
Expand the right-hand side of eq. (29) and we can see it also holds.
Proof of theorem 4.4. According to lemma E.3, we have
Σ Σ OC
50
Σ ΣΣ Oc
ax-(ight
x
50(())800 80(()
-0 20
q
ay-coul
al((.)
1TT4 9
1TTT
9
OC
@x
(lef))
Σ
58(4)
ay/amul) ko
oo(l-)
og(h)
4
La L
4
OC
(yy)
(ight
58((.) (35)
aylmul) tarrtight
Encron En
(0
00((.)
0e
Apply similar operations on E Enng Er -0 80
ofc (+) (0( and we can get eq. (13).
19


Reviews:
[{"review_id": "3WUnrl8AKW", "reviewer": "Reviewer_rbue", "summary": "This paper presents Optimal Brain Apoptosis (OBA), which appears to be a novel pruning method for neural networks that builds upon the principles of Optimal Brain Damage (OBD), which date back to 1980s and 1990s. Recognizing the limitations of previous pruning methods, OBA leverages the full Hessian-vector product to compute the importance of each parameter in the network accurately, moving beyond previous methods that relied on approximations. The authors first analyze the conditions under which Hessian submatrices between layers are nonzero and develop an approach to calculate the second-order Taylor expansion for each parameter (enabling precise pruning in both structured and unstructured settings). Empirical tests demonstrate that OBA effectively reduces computational overhead while maintaining model accuracy. The authors acknowledge that while OBA works well for architectures like CNNs and Transformers, extending it to more complex models like RNNs or State Space Models will require further research.", "questions": "Please see the weaknesses section. Providing feedback on those comments during rebuttal will be appreciated.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Unlike previous methods that approximate the Hessian matrix, OBA calculates the full Hessian-vector product, providing a more accurate measure of parameter importance and leading to more precise pruning.\n- OBA supports both structured pruning (removing entire neurons, channels, or layers) and unstructured pruning (removing individual weights). It's compatible with a wide range of architectures.\n- The approach optimizes the calculation of the Hessian-vector product (reduced computational complexity).\n- Showing adaptability to widely-used architectures in deep learning.\n- The authors provide a solid theoretical basis for their approach, with clear proofs.", "weaknesses": "- It seems like calculating the full Hessian-vector product, even with optimizations, can still be computationally expensive for larger and more complex networks.\n- Extending this method to newer architectures seems to require additional work. \n-The method was tested on a specific set of architectures, and its generalizability across a wider range of tasks or domains is yet to be fully established.\n- The experiments make the paper feel somewhat outdated, as the evaluations and datasets used are reminiscent of those from 7-8 years ago.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k26lhzC6pN", "reviewer": "Reviewer_v9aH", "summary": "This paper introduces a tractable way of computing the Hessian of the loss function in feed-forward networks. This finding is very interesting, as it may have a wide range of applications, in particular, the one of pruning CNNs and transformers. The results are very promising, as they show some improvements, but these improvements do not seem to be very large.", "questions": "Considering that recurrent neural networks use back-propagation through time, for a given time window, how tractable would be to extend your method to such networks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "An interesting new method for computing the Hessian of the loss function in feed-forward networks in a tractable way.", "weaknesses": "The improvements in accuracy and speed do not seem overwhelming.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rnck1Jg5cn", "reviewer": "Reviewer_1PJY", "summary": "The authors propose a new method for neural network pruning called Optimal Brain Apoptosis (OBA) inspired by the prior work Optimal Brain Damage (Lecun et al., 1989). This method calculates the Hessian-vector product for each parameter and identify the conditions under which inter-layer Hessian submatrices are non-zero. The proposed method is able to prune models to increase model efficiency on 3 convolutional backbones and one vision transformers backbone on CIFAR10, CIFAR100 and ImageNet datasets.", "questions": "1.\tPlease refer to Weakness 1.\n2.\tMinor suggestion. For LaTeX notations, the subscripts (especially subscripts of superscripts) can be wrapped in text format. What I mean is instead of $\\mathbb{R}^{l_out}$, using $\\mathbb{R}^{l_\\textrm{out}}$ might give you a better looking symbol.\n3.\tWhat do a, b, c, d, e respectively mean in Equation 3? Could the authors define them or point to the text where they are defined?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tPruning performance looks fairly promising. The method achieves consistent improvement on various datasets, using the most commonly-used backbones (ResNet and ViT), and on unstructured vs structured pruning.\n2.\tThe results shown are quite comprehensive: pruning performance (accuracy, parameter reduction, FLOPs reduction, throughput increase), pruning cost (training and pruning time). Surprisingly, the pruning cost was not as high as I originally expected knowing that the method involves computation of the Hessian-vector product.", "weaknesses": "1.\tSince pruning is not my area of expertise, I am unsure whether the authors used fair baselines for comparison. The proposed method is mainly compared against 7 methods from 3 papers, respectively in 2016, 2017 and 2019. A simple literature search gave me a few methods that claim to have achieved better pruning performances and are fairly well cited and fairly highly stared: https://arxiv.org/abs/2203.04248, https://arxiv.org/abs/2208.11580, https://arxiv.org/abs/2210.04092, https://arxiv.org/abs/2112.00029. I would encourage the authors to either compare to some of the latest works, or provide a justification on not considering these more recent works --- meanwhile, I would resort to reviewers with more experience in pruning on their opinions regarding this matter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6I6M1U3u56", "reviewer": "Reviewer_fJfY", "summary": "The paper concerns the importance of pruning to reduce computational burden of neural network inference. The structure of the Hessian matrix is analysed with a productive derivation of structure for  feedforward nets with so-called series and parallel connectivity. Empirical results are presented for standard (vision) CNNs.", "questions": "1)\tEmpirical evidence: While the results for CNNs are impressive, please consider transformers. There is much interest in pruning and redundancy in transformers e.g.  \na.\tMen, X., Xu, M., Zhang, Q., Wang, B., Lin, H., Lu, Y., Han, X. and Chen, W., 2024. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853.\nb.\tLad, V., Gurnee, W. and Tegmark, M., 2024. The Remarkable Robustness of LLMs: Stages of Inference?. arXiv preprint arXiv:2406.19384.\n\n2)\tL652 Consider setting the scene, motivation and related work as an integral part of the paper. With due refence to early work on pruning and Hessian structure\n\n3)\tL058-9 The OBS method is only different from OBD when the non-diagonal Hessian is used, i.e they do not ignore the off-diagonal terms in OBS\n\n4)\tL095 You mention that the Gauss.Newton approximation is in-sufficient what is the evidence?\n\n5)\tL087 Do you keep track of the first order term (which is assumed zero in OBD/OBS)?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper’s most prominent strength is the set of Hessian structures derived for series and parallel connectivity. The empirical results are promising showing significant reductions in FLOPS for virtually unchanged performance.", "weaknesses": "1)\tThe field is very busy with many results going back to the early 90s. Computational results and simplifying structure of the Hessian for feedforward network are found in many papers including\n•\tWille, J., 1997, June. On the structure of the Hessian matrix in feedforward networks and second derivative methods. In Proceedings of International Conference on Neural Networks (ICNN'97) (Vol. 3, pp. 1851-1855). IEEE.\n•\tBuntine, W.L. and Weigend, A.S., 1994. Computing second derivatives in feed-forward networks: A review. IEEE transactions on Neural Networks, 5(3), pp.480-488.\n•\tWu, Y., Zhu, X., Wu, C., Wang, A. and Ge, R., 2020. Dissecting hessian: Understanding common structure of hessian in neural networks. arXiv preprint arXiv:2010.04261\n•\tSingh, S.P., Bachmann, G. and Hofmann, T., 2021. Analytic insights into structure and rank of neural network hessian maps. Advances in Neural Information Processing Systems, 34, pp.23914-23927.\n\nThe paper presents a reduced complexity calculation of Hessian x vector, this is well-known - see work by Barak Pearlmutter for example; Pearlmutter, B.A., 1994. Fast exact multiplication by the Hessian. Neural computation, 6(1), pp.147-160.\n\n•\tIn general the context of related work is not sufficient. I am not a fan of placing context related work in an appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)