DPaI: Differentiable Pruning at Initialization with Node-Path Balance Principle
Authors: Lichuan Xiang
Quan Nguyen-Tri
Lan-Cuong Nguyen
Hoang Pham
Khoat Than
Long Tran-Thanh
Hongkai Wen
Venue: ICLR 2025 Poster (2025)
Keywords: Prunning at Initialization
Sparsity
Neural Architecture Search
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: Our paper proposed a differentiable pruning at initialization methods that achived significantly better performance on pruning at initialization tasks.

Abstract: Pruning at Initialization (PaI) is a technique in neural network optimization characterized by the proactive elimination of weights before the network's training on designated tasks. This innovative strategy potentially reduces the costs for training and inference, significantly advancing computational efficiency. A key factor leading to PaI's effectiveness is that it considers the saliency of weights in an untrained network, and prioritizes the trainability and optimization potential of the pruned subnetworks. Recent methods can effectively prevent the formation of hard-to-optimize networks, e.g. through iterative adjustments at each network layer. However, this way often results in large-scale discrete optimization problems, which could make PaI further challenging. This paper introduces a novel method, called DPaI, that involves a differentiable optimization of the pruning mask. DPaI adopts a dynamic and adaptable pruning process, allowing easier optimization processes and better solutions. More importantly, our differentiable formulation enables readily use of the existing rich body of efficient gradient-based methods for PaI. Our empirical results demonstrate that DPaI significantly outperforms current state-of-the-art PaI methods on various architectures, such as Convolutional Neural Networks and Vision-Transformers. Code is available at https://github.com/QuanNguyen-Tri/DPaI.git

Published as a conference paper at ICLR 2025
DPAI: DIFFERENTIABLE PRUNING AT INITIALIZATION WITH HODE-PATH BALANCE PRINCIPLE
Lichuan Xiang 14,,, Quan Nguyen-Tri Lan-Cuong Nguyen 2.3 Hoang Pham',
Khoat Than', Long Tran-Thanh', Hongkai Wen Universitty of Warwick, Hanoi University of Science and Technology, SPPT Software Al Center, "Collov Labs
ABSTRACT
Pruning Initialization (Pal) is technique in neural network optimization char- acterized by the proactive elimination of weights before the network's s training designated tasks. This innovative strategy potentially reduces the costs for training and inference, significantly advancing computational efficiency. A key factor leading to Pal's effectiveness is that it considers the saliency of weights in an untrained network, and prioritizes the trainability and optimization potential of the pruned subnetworks. Recent methods can effectively prevent the formation of tard-to-optimize networks, e.g. through iterative adjustments each network layer. However, this way often results in large-scale discrete optimization prob- lems, which could make Pal further challenging. This paper introduces a novel method, called DPal, that involves a differentiable optimization of the pruning mask. DPal adopts dynamic and adaptable pruning process, allowing easier optimization processes and better solutions. More importantly, our differentiable formulation enables readily of the existing rich body of efficient gradient-based methods for Pal. Our empirical results demonstrate that DPal significantly out- performs current state-of-the-arr Pal methods on various architectures, such Convolutional Neural Networks and Vision-Treee Tranon-Traaffooess...... Code is available at httpss//gihhub.com//uunneae -ww.anagenennnnnnaaaa......
INTRODUCTION
The Lottery Ticket Hypothesis (LTH) (Frankle & Carbin 2018 Chen et al. 2020 202 la) serves as a foundational concept our research, revealing the potential of sparse subnetworks that can be trained from scratch to achieve the performance levels of their dense counterparts. However, a critical challenge with LTH its resource-intensive nature, involving an iterative process of pruning and retraining that often exceeds the cost of training a dense network. This challenge presents an intriguing research question: Can we identify sparse, trainable subnetworks the initialization phase, eliminating the need for pre-training? Specifically, method capable of effective pruning before training could significantly reduce memory and computational costs without substantially impacting performance (Wang et al. (022). Such a breakthrough would increase the adaptability of neural networks in
environments (Alizadeh et al. 2022 Yuan et al. 2021a).
To address this, various Pruning at Initialization (Pal) methods have been proposed (Lee et al. 2019b
Tanaka et al.) 2020) de Jorge et al. 2021 Wang et al., 2020; Alizadeh et al. 2022: Liu et al. 2022b). These methods often rely on gradient information (Lee et al. 2019b Wang al. 2020) and focus on assessing the importance of network parameters. Nonetheless, their effectiveness in reducing computational load and maintaining accuracy is limited. This limitation arises from a singular focus on parameter importance, neglecting the broader network topology, leading to sub-optimal, hard-to-optimize narrow networks during retraining.
Recent studies Franklee et al. 2021 Su et al. 2020) suggest that the overall topology of the final sparse network may be more crucial than individual weight-wise importance. Building on these
"Equal Contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2025
findings, (Pham et al. 2023) have introduced Node-Path Balancing Principle (NPB) that shows the balance between the number of effective nodes and the effective path is essential for generating extremely sparse networks with good performance. However, to do so, the current NPB principle requires addressing the complexity of an underlying sequence of discrete optimization problems, which necessitates iterative solutions at each layer, often resulting in sub-optimal outcomes for the final pruning mask.
In this paper, we introduce novel approach that resolves the mask optimization issue at initialization in learnable manner, directly targeting the overall network metries and enhancing performance. In particular, our approach makes NPB differentiable (and thus, making the NPB principle more compatible with standard neural network training processes, a.k.a. learnable) by replacing the current discrete (i.e., non-differentiable) optimization component in the NPB principle with a new differen- tiable module. A key challenge here is relax the underlying set of non-linear integer programs into a continuous version, which still provides valid solutions. We introduce DPal (Differentiable Pruning at Initialization), novel differentiable continuous optimization approach to identify good final pruning masks. In summary, our contributions are as follows:
DPal is the first differentiable pruning at initialization method that takes into account net- work topology. specifically the Node-path Balancing Principle (NBP), a recently identified essential concept to achieve good sparse neural networks.
It extends NPB into a differentiable formulation, making its integration with the training process of sparse neural networks more natural. Different from NPB, our DPal enables readily use of the existing rich body of efficient gradient- based methods.
Extensive experiments on diverse datasets show that the proposed DPal can discover superior sparse sub-networks across multiple sparsity levels.
2 RELATED WORK
Neural Network Pruning. Traditional neural network pruning methods, as described in seminal works by LeCun et al. (1989). Hassibi al. (1993). Han al. (2015), Dong al. (2017), and Molchanov et al. (2017), ), primarily focus on trimming trained models based on specific pre-defined criteria. Subsequently, these pruned models, or subnetworks, undergo fine-tuning to ensure conver- gence. However, recent empirical studies by /Frankle & Carbin (2018), Frankle et al. (200)), Chen (et (2020), and Chen al. (2021a) have revealed the existence of 'lottery tickets' subnetworks with random initializations that, when trained from scratch or in the early stages training, can perform comparably to their unpruned, dense counterparts. Despite these findings, the process of identifying these 'lottery tickets' remains computationally intensive due to the necessity of repeated training and pruning cycles, as highlighted by Frankle et al. (2020) and Frankle & Carbin (2018).
In response this challenge, gradual pruning approaches such as those proposed ZhZhu & Gupta (211)) and Gale et al. (2019). intertwine the processes of pruning and training. These methods typically incur lower computational costs compared to post-training pruning. Nonetheless, they still require initial training phases to determine the most effective sparse subnetwork configuration. In contrast, other techniques, such as those introduced by Chen al. (202Tb) and Chen al. (2023). implement one-shot pruning during the training process to further decrease computational demands. Alternatively. dynamic sparse training methodologies, as explored by Mocanu al. (2018). Evei et al. 2020)) Liu et al. (2021b), and Liu et al. 20) initiate with a randomly sparse network and adaptively update network connectivity throughout the training.
Moreover, certain approaches (c.g. Lee et al. (2019b), Wang et al. (2020). Patil & Dovrolis (2021),
Tanaka et al. (2020), and Alizadeh et al. (2022) focus determining subnetworks based on network initialization, gradient information, and network topology, essentially pruning before training. However, as Frankle et al. (202T) and (Su et al.) (2020) have demonstrated through their experimental results, the existing criteria in these Pal methods may not always yield subnetworks with optimal performance.
Network Shape and Gradients During Pruning, In Pruming at initialization (Pal) methods, where training data is often unused (Tanaka et al., [020 Patil & Dovrolis 2021) or used minimally for gradient information (Lee 2019b;; Wang et al. (020),, network topology becomes critical to optimize pruned network performance. Various Pal techniques affect network topology. SynFlow (Tanaka al. 2020) aims to maintain input-output paths but often increases isolated neurons.
2

---

Published as a conference paper at ICLR 2025
Strategies by Patil & Dovrolis (2021) and Gebhart et al. (2021) preserve network efficiency through
path kernels, focusing on subnetwork structure. PHEW (Patil & Dovrolis 202T) uses random walks
to increase effective nodes but reduces input-output paths. Other researchers have examined 'effective
nodes' and 'effective paths' in pruned subnetworks (Wang al. 2020: Naji et al. 2021).
Recent advancements in Neural Architecture Search (NAS) by Xiang et al. (023) and Sun et al. (2023) integrate gradient signal-to-noise ratio in zero-cost network performance evaluations, This method balances gradient distribution across layers, ensuring uniform spread and avoiding gradient concentration, which can lead to narrow layers and poor performance. This approach aligns with node-path balance, considering the quality of gradients along various paths for more effective evaluation of network trainability.
Extreme Sparse Networks. In the context of extreme sparsity (Cho et al. 2021| Price & Tanner 2021). the network density is less than 1%. Cho et al. (2021) associate two works from Leee et al. (2019b); Zhou et al. (2019) to learn masking during training. Tanaka et al. (2020); de Jorge et al. (2021): Vysogorets & Kempe (2021) leverage iterative pruning to prevent subnetworks from layer collapse in super sparsity cases. Price Tanner (202T) only requires an extremely small number of trainable parameters associated with a freeze fully connected network, which helps the model performing well on extreme sparsity settings. This suggests preserving information flow through network connections is crucial in extremely sparse networks.
Differentiable Neural Architecture Search. Liu
(2019) introduced the concept of using continuous architectural parameters (a) for searching neural network architectures in a differentiable manner, optimizing a with V.C., (w vV (wr, a)). This method involves constructing a supernet that encompasses all architectures within the search space and optimizing both a and the supernet weights (w). The final architecture derived by retaining operations with the highest values. Despite reducing search time, DARTS has stability and generalizability issues, often favouring trivial models with excessive skip connections (Zela et al. 2020). To address these issues, SDARTS (Chen & Hsich 2020) smooths the loss landscape, and SGAS (Li et al. 2020) uses a greedy algorithm for operational selection and pruning. Recently, DARTS-PT (Wang et al. (021)) proposed a perturbation- based operation selection method, evaluating operations based on their impact on the supernet's validation accuracy upon removal. However, while these approaches derive dense subnetworks from the supernet by learning and optimizing a during training, they may not be suited for identifying sparse subnetworks at initialization.
3 METHOD
In the following section, we first describe the NPB principle (Section 3.1), which forms the basis of our solution. We discuss the novel formulation of differentiable Node-Path Balancing (d-NPB) in Section 3.1, and then present our DPal algorithm in Section 3.3, followed by convergence analysis for d-NPB methods in Section 3.4.
3.1 THE NODE-PATI BALANCING PRINCIPLE
In sparse network, it is intuitively clear that one should arrange the connections into a configuration that neither too thin nor too spread out to have good information propagation during training. Evidence of this understanding is the Node-Path Balancing (NPB) principle introduced by Phhm et al. (2023). In what follows, first start with the key definitions, then introduce the NPB concept; the detail of metrics implementation can be found in Appendix
Effective Path. We define path to be effective if it connects input node to output node without interruption. Metrics based on paths are mentioned in Tanaka et al. (2000); /ebbarr et al. (2021) as and l2 path norms, respectively. In this paper, we only consider the number of paths.
Effective Node/Channel. node/channel is effective if at least one effective path goes through This concept is also considered in |Patil & Dovrolis (202T); FFannle et al.) (202I). For convolutional layers, we consider a kernel as a connection and channel node and then convert the convolutional layer into a fully connected layer.
The key objective of the NPB principle is generate a sparse network with the maximal number of effective nodes and paths simultaneously, given the sparsity constraint. To achieve this, the principle aims solve the following discrete optimization problem:
3

---

Published as a conference paper at ICLR 2025
Given an architecture A with parameter W E RN, where N is the total number of parameters. and sparsity ratio т. Denote Rp as the total number of input-output paths, R as the number of activated nodes, and consider the mask for parameter M = {0, 1j as the variable to solve. For some I a S
Maximize RNPB= == NRN + Rp s.t. | S N(1 -)
This large-scale discrete optimization problem NP-hard, and thus, Pham (2023) have proposed a heuristic to produce sparse neural network architectures. However, this heuristic is often suboptimal, and the underlying discrete optimization concept cannot be easily integrated into the standard neural network training pipeline, which may limit the usage of the NPB principle.
3.2 DIFFERENTIABLE NODE-PATH BALANCING (D-NPB) OPTIMIZATION
To overcome this issue, this section introduces novel way of converting the discrete NPB principle into differentiable form. To so, we introduce a differentiable parameter s()) R.h x
adjust the importance of parameters in layer I, in which A!4) the number of neurons in layer 1. The mask for the parameters in layer can be obtained by: mi = Topa (1s2))) where
Top& (x) = [1
[O if pruned k:01) can be the if in & largest elements and we set m { otherwise"
otherwise
desired density level for layer / (e.g. given by the Erdōs-Rényí Kernel (ERK) (Liu et al. 2022a) )
Due to the extremely large number of paths, especially in denser networks, there is an imbalance with other objectives, like the number of nodes, which is significantly smaller. To address this, we applied a logarithmic scale to all objectives to prevent this imbalance. The optimisation of the NPB principle now becomes:
Maximize RNPB: = log loogg R + (1 a:) log Rp
|| S ((1-))
Differentiable Calculation of the Effective Path: Denotes P( the number of incoming paths
node 2 The number of effective paths is the number of incoming paths to the nodes in the last layer L:
A8-1)
Rp = P( P(, = Σ m
) =
(I)
1=1
The derivative of 8ij can be estimated via the Throughh Estimator (Bengio et al. derivative goes "straight-thruugh" (.)):
(!)
2013 (the
8 8log Rp 8R 8P(0)) (1)
8 log Rr
1
3Rp (11)) 8P(1 (3)
$1
P(
(2)
5Rp
(P(15))
Rp
The number of outgoing paths from node vfth to the nodes in the last layer L can be estimated by the following derivative:
ORp 8P( (1+1)
8RP
Σ
Σ
8Rp
(++1) j.k
8P 8P( (+1))
8P(1,))
8P(c
m
(3)
= Σ
m (1) (1) (T- m
(111) j,k
(1) (1-11 m m m
(1+1)
8P(wn (L)
n.P.g....k
n.p.q.....k
Differentiable Calculation of the Effective Node/Channel: A node u/ is activated if there is path connecting the input nodes to it P(u)") > 0, and there is a path connecting to the output nodes 6R (5) > Therefore, node of) considered effective when
(1
)P())))
8RP Oー
N( D))
8Rp (1) 0)
Σ
m P(
(1-1
x Σ
8 log Rp
(1) m
>0 (4)
i=1
4

---

Published as a conference paper at ICLR 2025
We tanh(z) <---- differentiable activation function for the number of effective nodes 6= counts, resulting in tanh (v))
when ug) is an effective node, and y is a sufficiently large constant. The objective of maximizing the number of effective nodes can be written follows:
1. 1.0
RN ΣΣ tanh (22V(190))
(5)
==
We can write the derivative of log RN as follows:
log RN
(1) RN (1 (1-tane tanh? (2N(c191)) 8 log Rp
ds;j
(1)
(6)
Differentiable Calculation of the Effective Kernel/Connection In pruned network, want to maximize the number of weights that receive gradient for the update. [ham et al. (2023) also integrates this idea in terms of regularization that aims to encourage activating many kernels as possible. We transfer this concept into our differentiable method via the concept of Effective (!)
Kemel/Weight. A kernel/comnection m.j is effective if there is an effective path pass through it:
AR ((m P(u!- mmb SP
(1)
8log Rr 5el
( 0. The objective for kerne//connection is:
mi.j
RC Σt= En Ene tanh (7:((m
(m))
The derivative of log Re computed as follows:
log
2 Re
tanh? (Nmm))) &log Re. Note that ! represents a single connection in 6e
de
fully connected layer or the whole kernel in convolutional layer.
Then, maximize the overall objective RDPat ax) log Rp + a[[11 (1-3) 3) log RN + 3 log Rcl by repeatedly computing the following rule until convergence, where v/ is the learning rate:
19 + + (11 (1-a)
8Rp (1)
8RN
RRC
$ 0 = 8
+
B)
(1)
8813
8813
3.3 CONVERGENCE ANALYSIS OF DIFFERENTIABLE NODE-PATH BALANCING
(!) Assuming that after an update, edge m replaces mp.g. and the rest the sub-network remains fixed. Therefore, before the update we have: m 0 and m 1. and after the update we have:
(1)
(1) 1 and mp.g O. We can compare the score parameters corresponding edge m!! and
m. (1)
Mm.. before the update as:
log R
8 log R
(
P.Q
88
(1)
p.q
(1)
+ 1
8sp.
(8)
where R can be R.p or R or Re. We have 8..
BlogT
x
BlogR 8.3
(t)
8P(u)")
8R
)
P( (1-1)
((
2 0.
hence the derivative of the new edge must be higher than the old ones:
5log 5u
For the Path objective log Rp, We have:
1 8R,
P(v (1-1) ((o
>
8R,
P P(u
(1-1))
18p.0
(1)
(9)
R. (1)
Rp SP( !1)
(2)
$
Sp.4
8Rr P(u )
8Rp
P(ufl11)
(10)
)P() (1)
)P(u!!1)
(1)
5

---

Published as a conference paper at ICLR 2025
where P((z)
!1) 6R represents the total number of effective paths containing edge mi.j (1) mm 6P(2)
After the update, the number of effective paths is changed by: A = P(u!-1)) 8P(
8RP
This indicates that updating parameters concerning the path objective guarantees an increase in the number of effective paths. For the Node Objective log Ry or the Kernel Objective log Rg. we both consider group of parameters (Node or Kernel) as N(). In the following, take example of the Node Objective:
P(up P(up-1) () BRp
sP(es")
R.v (1 (1-tanh²
tann? (T:((o)))) log Rp
1 RN (1 (1-tanil) tanh? (IN(1(1))) 8 8log Rp
(11)
88
08p.4
(1)
When is sufficiently large we have:
8
Blog
if N(o N(u
88p.4
log Rr
0
if N(u,") 0. N(u 0
8x
log R.P
if N(o,tD)) > 0, N(v. 0
(12)
0
88p.a
update
>>0
if N(w,)) > 0, N(u
no updale
As result, the update only occurs when the node u/ is ineffective, and after the update, it becomes effective because update, the number of effective nodes increases by one, and the number of effective paths grows by ARp P(vl111)
foo RP
(1)
holds (since 5log 0). If og is ineffective before the Bsp.o
8ss.ght mij
&Rp
If ( is already effective before the update, the change in SRP If mp is the SP(127)"
number of effective paths is given by A = P(u!- (1-1) SRP P(up 11-1), (P()
last connection node ( the node becomes ineffective, leaving the number of effective nodes unchanged; otherwise, the number of effective nodes increases by one. The Kernel Objective behaves
similarly, where N (v.")) is replaced with (m))))
When combining the Node (Kernel) Objective with the Path Objective, obtain:
1 1-a a tanh? (y)((.)) 8log Rr (1)
1 1 + (1 tanh2 (+:(())) 5 log Rp (I)
>
(13)
RN
88
RN
8sp4
here we combine y with a as the factor for the node objective. With sufficiently large 7, we have:
log
llog Re Axpa
if =
or N(o)") 00,((o >
>
log Ry !x
x
5 log Rr
if
(i) N(u, 0. N(v
(1 )(x0)>
(1)
log Rg 66 (9)
log Rg
if
(ii N(u > 0, N (1) =0
66p.9
(14)
The combined objective continues to activate ineffective nodes after the update while ensuring that the number of effective paths always increases. When is high, it increases the likelihood of (i) occurring and decreases the likelihood of (ii) occurring, thus the objective focuses more on activating more effective nodes. Conversely, when is low, it decreases the likelihood of (i) and increases the

---

Published as a conference paper at ICLR 2025
Algorithm 1 Differentiable Pruning at Initialization (DPal)
1: Input: network fær W), final sparsity p. iteration steps T, hyperparameter a, ,,
(0)
2: Initialize the score parameters: $ij ~ (N(,
3: Obtain layer-wise sparsity k:(1) from p by using the ERK method
4: for E ,, do
5: Binarize the mask on each layer: mij + Top (1
6: Compute the number of effective paths: R.p s- MM
7: Compute the derivatives with respecting to each objective:
6R SRx ARc 50
8: Update the score parameters: 3 - 5, + (
a
RR
+ a (1 3) a
+
3
58,
68
9: end for
10: Output: pruned network far, Mo W)
likelihood of (ii), shifting the focus towards moving connections from ineffective nodes effective nodes, thereby promoting more effective paths. Thus, by adjusting a, we can control the behavior of the objective to obtain the desired sub-networks.
3.4 DIFFERENTIARLE PRUNING INITIALIZATION (DPAI) ALGORITHM
Pruning at Initialization (Pal) methods are designed to remove weights from neural networks be- fore training, thereby reducing training costs. Existing approaches typically involve progressively increasing pruning sparsity or employing layer-wise solutions. However, these methods often result in sub-optimal masks the overall pruned network has never been appropriately evaluated during the pruning process. As proposed in the previous section, we introduce a novel pruning mechanism to address this challenge. This mechanism updates all masks concurrently and optimizes them directly towards a predefined target objective based on the masked parameters at a given sparsity.
Layer-wise Sparsity Ratios. A significant challenge in existing Pruning at Initialization (Pal) research the distribution of overall sparsity across each network layer. Current gradient-based methods tend to allocate lower weights to layers with more parameters, leading to an unreasonable pruning ratio in these larger layers. This can result collapse, in which a whole layer is pruned, rendering all nodes and paths ineffective Frankle et al. (20TT. However, recent research by Liu et al. (2022a) demonstrates that the Erdös-Rényi Kernel (ERK) method is highly efficient and effective, surpassing most contemporary iterative or dynamic approaches in assigning layer-wise sparsity ratios.
catastrophic phenomenon known layer
The concept of Rénös-Rényi (ER) topology was initially applied by Mocanu et al. (0018) to introduce sparsity in Multilayer Perceptron (MLP) networks. This method employs random topology that imposes higher sparsity on larger layers. Evei et al. (2020) further extended this approach to convolu- tional networks, creating the ERK, which scales the sparsity of convolutional layer proportionally to the number neurons/channels in that layer.
In our study, we devised method to determine the sparsity level for each convolutional layer. This is accomplished by scaling the sparsity proportionally as follows:
n + n' + w + ht
(15)
1n' w
where re' denotes the number of neurons channels in the tt layer, while w' and h represent the width and height of that layer, respectively. This formula designed to adjust the sparsity in relation to the layer's 's structural dimensions, ensuring balanced and effective sparsity distribution across different layers of the network.
Differentiable Mask Updating. In light of the objectives previously outlined, we propose novel differentiable algorithm designed update masks gradient- based manner. This approach enables direct optimization of specified metrics for targeted sparse networks at given sparsity level. The algorithm operates depicted in Algorithm T In all our experiments, we say that the algorithm has converged when i) it runs for 3000 steps or ii) the objective (e.g. the number of effective nodes, paths, and kernels does not change significantly).
7

---

Published as a conference paper at ICLR 2025
Nunber of edfectte patte 101g 5034
Spansit 68.38%
Sparsity 80.00%
Sparsity 96.91%
Sparsity 29, 00.00%
RosNet20 on CIFAR-10
2012
26
20
54
Narber of offective nodes Cn scaln)
Narber of offective rodes Cn ecale)
Number offective rodes (In ocale)
Number of officttv nodes ih s0d00
NuRteN of dteetoo porks Do teato
Spunsis 68 58%
Spersity 90.00%
Spersity 96.81%
40.3
Sparsity 50,000s
VGG19 on CIFAR- 100
72.5% Mi
to
VIIS
ALET
20-3
30
264
20.5
Naubbrr of ciliciire auder du scalc)
Nunterr of cflectiv MAde Gassele)
Nunder sdlodiie undoo Cla seule)
Nustice ol etlod.vv uaddo lla SAND0)
ResNet18 on Thyy Iry--nannnett
Nirilet of cntoortee thi scatts
Spansite 68 10.38%%
Sparsit 90.00%
Sparsity 96.84%
Sparsity 99, 0098
50.3
50.0
A443
45.0
11
36.2
40.0
50.00
66
37.5
$5.22
23
32.5
61
24
7.4
24
8.2
6.75 244 7,25 7:00 375
Namber of offection moder Do araln)
Nanbber of affection rodes scale)
Naniberr affection modee meale))
Nonoher wftvd vn nodeee (It wokm))
Figure 1: Accuracy of different Pal methods on three datasets with their corresponding number of effective nodes and paths across sparsity levels. The best accuracy of each setting set in blue.
EVALUATION
In this section, we evaluate the pruned sparse network by re-training the network on different tasks (CIFAR-10. CIFAR-100, and Tiny-ImageNet, respectively) align experiment setting present in previous satte-of-the-ar((( (he-ar((SoTA) works; the training details are provided in the Appendix. We also perform experiments on ImageNet-1K, (Deng et al. 2009) to verify our methods work on large-scale dataset tasks.
4.1 COMPARING DPAI WITH PREVIOUS SOTA
We now turn to evaluate the performance of DPal. In alignment with Equation 7 grid search strategy to optimize the hyper-parameters o and (3. This approach was instrumental in refining our final objective for the DPal method. Subsequently, we compared DPal with previous state-of-tt the-art (SoTA) methods in the Pruning at Initialization (Pal) task. Our observations reveal that DPal consistently and significantly outperforms all prior SoTA methods across various ResNet-basee tasks (see Fig. for more details). This particularly evident at higher sparsity levels (96.8 99,.8%, 99.00%), where we noted the most substantial improvement in accuracy (up to 4.6%). with most cases showing improvements greater than 2%. DPal only underperforms NPB and PHEW on the VGG19 network at 99.00% sparsity. We argue that those methods bias their algorithms towards weight magnitudes, which gives them an advantage in this specific case of high sparsity. However, DPal, which relies solely on the Node-Path Balance principle, still outperforms all baselines at other sparsity levels on VGG19, achieving significant gap of 1% to 2%. We can also observe how DPal achieves substantial improvements in accuracy by comparing the number of effective nodes and effective paths identified by DPal with those found by other baselines. In ResNet-like architectures, the subnetworks discovered by DPal consistently have higher number of effective nodes and paths, outperforning all baselines across all settings. Although NPB addresses the same objective using a discrete optimizer, must decompose the full objective into layer-wise objectives due intractability. This approach results in their method producing sub-optimal solutions for architectures with complex connections between layers, such ResNet-like architectures with skip connections. In contrast,
we employed a
8

---

71.8 Published as a conference paper at ICLR 2025
Table Comparison of Avg and Best Acc(%) between Synflow and DPal Methods on ImageNet-
Avg Acc(%)
Best Acc(%)
Synflow DPal
71.4 = 0.29
72.2 + 0.25
72.5
Sparsity 68.38%
Sparsity 90.00%
Sparsity 96, 8..44%
Sparsity 99 99.00%
0fffccun Tutte (99 (206)
a1.
81.0 125
90.6
85. 200
20.5
90.22
79.5
50.5
75.0
89.8
75.5
78.0
15.72 16.33 26.55 36.3
23.0
255
135
243 184
Sparsity 68.38%
Sparsity 90 000%
Sparsity 96. 84%
Sparsity 59 000%
75.0
13.0
+0heeie on CIFAR-30
74.
725
72.0
74
70.
70.0
60.,0%
Sparsity 0000
Sparsity 84%
Sparsity 99 000
RezNette on Tinps Inngennneee
65.0
64.5
00.0
Accurtt
59.5
00
96.4
Difocto Nodre Pus Dn a0000)
Dfficetiio Nedos Plas On scalo)
Efflcctive Nedoo Pras (In scalo)
Offecttie Nodes Plus On scale)
Figure 2: Results for different hyperparameter settings across various architectures and datasets, along with the number of Effective Paths (y-axis) and the combined number of Effective Nodes and Kernels (x-axis), reported. The "# Effective Node Plus" refers to our extended concept of effective nodes, which now includes effective kemels,, calculated as log RN + log Rc.
DPal can seamlessly adapt to any complex neural network architecture due to its differentiable nature. In the VGG experiments, both NPB and DPal yield a comparable number of effective nodes and effective paths. However, simply displaying the number of effective nodes and effective paths does not fully capture the results of our objective, as it also includes the goal of activating effective kernels/connections.. We present extended view that combines nodes and kernels to analyze their impact on accuracy in Section 4.2 and Appendix H
To verify approach in more practical setting, also perform ImageNet- 1K experiments on EfficientNetB0 in Table [ We evaluated our DPal methods against the baseline approach, Synflow, in pruning the EfficientNetBO architecture under sparsity constraint of 0.3. The original EfficientNetBO model has 5.29 million parameters; our pruning target was to reduce this to 3.72 million; more experiment settings can be found in the Appendix [F] prresent the average and best accuracy percentages of two methods, Synflow and DPal, accuracy of Synflow is 71.4% with a standard deviation of +0.29, and its best accuracy is 71.8%. The DPal method shows a slightly higher average accuracy of 72.2% with standard deviation of +0.25, and best accuracy of 72.5%.
given experiment. The average
4.2 ABLATION STUDY
In Fig. 2 we analyze the effect of the hyperparameters and on DPal's performance. Each value of a and 3 corresponds to a specific point in the figure, representing different numbers of effective nodes, paths, and kernels. We found that these hyperparameters highly impact DPal's

---

Published as a conference paper at ICLR 2025
Pruning Time- Resnet18
Pruning Time- VGG19
Pruning Time- Resnet20
Pruning Time (s)
4000
5000
Syn
Pruning Time
6000
Synfiise FIE
Pruning Time
60
3000
NFO
CPul
40.00
MPD
DPi (ours)
40
SNIP
2000
2000
20
Systt
1000
MP5
70
80 50
100
70
80
90
100
70
80
90
100
Sparsity (%)
Sparsity (9%
Sparsity (%6)
Figure 3: Pruning time of various pruning methods (SNIP, Iter-SNIP, SynFlow, PHEW, NPB, and DPal) on different neural network architectures (Resnet18 on Imay-em, VGGI9 on CIFAR-100, and Resnet20 on CIFAR-10) 10) across different sparsity levels.
effectiveness. However, even in the worst cases, DPal still outperforms most baselines (such as Random, SNIP, SynFlow, Iter-SNIP) across the majority of settings. The trade-off between effective paths and effective nodes (kernels), in relation to different hyperparameter values, creates Pareto front (Van Veldhuizen et al. 1998) in the multi-objective optimization of Node-Path-Kern point on this front represents a sub-network that is competitive with the baselines. From Fig. 2 we can identify trends that help in more quickly locating the best sub-networks on the Pareto front. The optimal sub-networks typically lie between the middle of the node-path balance point and the section with higher number of effective nodes and kernels. A very high number of effective paths often leads to worse performance compared to having a high number of effective nodes and kernels. In conclusion, the dependency on hyperparameters is major drawback of the Pal method, including our proposed DPal. However, unlike most baselines (such as NPB, PHEW, SNIP, SynFlow), which bias their algorithms based on the initial weight magnitudes, DPal is entirely data-agnostic and independent of initial weights. This makes it easier to reuse the pruned sub-network across different datasets once it has been properly pruned on a single example dataset.
Each
4.3 PRUNING TIME OF RESULTING SUBNETWORKS
We observed that DPal has achieved significant performance gain compared to existing pruning approaches. Fig. 3 shows the pruning cost in terms of wall clock time (seconds) of the proposed DPal as well as existing algorithms. We can observe that DPal provided consistent and relatively low pruning time across different initial architectures and sparsity, showing our efficiency is robust across different pruning settings. Compared with previous methods like NPB, in which initial architectures significantly influence the pruning time, PHEW strongly correlated with initial architectures and the level of sparsities. For our DPal method, we update differentiable scores using layer-specific statistics. Although implemented sequentially for simplicity, these computations can be parallelized as they are inyee-indeeennett, potentially reducing pruning time. Even without parallelization, DPal is able to discover better subnetworks without significantly increasing pruning time. The detailed statistics of pruning time results are shown in the Appendix. B
CONCLUSION
In this paper, introduce DPal, novel differentiable method for pruning at initialization (Pal). This approach extended the Node-Path Balancing (NPB) principle, addressing the challenges of developing a continuous gradient for the NPB optimization problem. Extensive experimental results demonstrate that DPal outperforms the state-of-the-arn without incurring significantly higher computational costs. Due to its differentiabilityy DPal's key advantage over the current best solution NPB, is its seamless integration into standard neural network training pipelines. This capability opens up potential applications for DPal in areas like neural architecture search and sparse training. Future work will focus on customizing DPal for these applications and exploring parallelization to further improve efficiency.
ACKNOWLEDGMENTS
This work was conducted while Quan Nguyen- Tri was funded by the Vingroup Innovation Foundation (VINIF) under project code VINIF2021. A.S...02I.ThS.B.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Milad Alizadeh, Shyam A. Tailor, Luisa M Zintgraf, Joost van Amersfoort, Sebastian Farquhar, Nicholas Donald Lane, and Yarin Gal. Prospect pruning: Finding trainable weights initialization using meta-gradients. International Conference on Learning Representations, 2022. URL
https:
net m
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: /30.33444444 3432, 2013.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33:15834 15846, 2020.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and superviud pre-training in computer vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16306-16316 2021a.
Tianyi Chen, Bo Ji, Tianyu DING, Biyi Fang, Guanyi Wang. Zhihui Zhu, Luming Liang, Yixin Shi, Sheng Yi, and Xiao Tu. Only train once: A one-shot neural network training and pruning framework. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021b. URL TtRLhttps:// htttps:://peerevieer net. /ortt/fmmm? 1d=p5rMP jrcCzq
Tiayyi Chen, Luming Liang, Tianyu DING, Zhihui Zhu, and Ilya Zharkov. ОTOv2: Automatic, generie, user-friendly. In The Eleventh International Conference Learning Representations, 2023. URL htt.tp:://openreiee...... www.ee//oouummmmmm htt.::/eeeoooouaaaaaaoooooo..... iPMt
Xiangning Chen and Cho-Jui Hsieh. Stabilizing differentiable architecture search via perturbation- based regularization. In International Conference on Machine Learning (ICML), pp. 1554-1565. PMLR, 2020.
Minsu Cho, Ameya Joshi, and Chinmay Hegde. Espn: Extremely sparse pruned networks. In 2021 IEEE Data Science and Learning Workshop (DSLW), pp. 1-8. IEEE, 2021.
Pau de Jorge, Amartya Sanyal, Harkirat Behl, Philip Torr, Grégory Rogez, and Puneet K. Dokania. Progressive skeletonization: Trimming more fat from a network initialization. In International Conference on Learning Representations, 2021. URL holLhtps::/// htt.t:::///penzeeeew.eeetttttt m id 9GsFOUyUP1]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in Neural Information Processing Systems, 30, 2017.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952. PMLR, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprini arXiv: 180300360 2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning. pp. 2259-3266 PMLR, 2020.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? In International Conference on Learning Representations, 2021. URL https: http://pppeeeveeeeeeeeee eeweeeeewwww.. net wtrrmoormmmm// Ig I--
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: (9l.::9902002252 09574, 2019.
11

---

Published as a conference paper at ICLR 2025
Shangqian Gao, Feihu Huang, Yanfu Zhang, and Heng Huang. Disentangled differentiable network pruning. In European Conference on Computer Vision, pp. 328-345. Springer, 2022.
Thomas Gebhart, Udit Saxena, and Paul Schrater. A unified paths perspective for pruning at initialization. ArXiv, abs/2101.10552 2021.
Song Han, Jeff Pool, John Tran, and William Dally, Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pp. 293-299. IEFF, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
N Lee, T Ajanthan, and P Torr. Snip: single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations. Open Review, 2019a.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. International Conference on Learning Representations, 2019b. URL
net Forumriruriaareee (tte:/foooooo--BB222)e))
Guohao Li, Guocheng Qian, Itzel C Delgadillo, Matthias Muller, Ali Thabet, and Bernard Ghanem. Sgas: Sequential greedy architecture search. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 1620-1630. 2020.
Shaohui Lin, Wenxuan Huang, Jiao Xie, Baochang Zhang, Yunhang Shen, Zhou Yu, Jungong Han, and David Doermann. Filter pruning for efficient enns via knowledge-die -rivee differential filter sampler. arXiv preprint arXiv:2307.00198 2023.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations (ICLR), 2019.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems, 34:9908-995 1999908-9922, 2021a.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense ? in-time over over-paramenee paramtterizzaioonn in sparse training. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 6989-7000. PMLR, 18 2--24 Jul 2021b. URL htt.pss://pooceeeeiiiiii
mlr. wttp.ppee.//..1////.e.... .hmml
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Shen, Decebal C Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. In International Conference on Learning Representations, ICLR 2022,
2022a.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. In International Conference on Learning Representations, 2022b.
URL
wtt.:/iwwweeeeeeeeeeeeeeeeeeeeee net. forum? id= 3tz-t
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. In International Conference on Learning Representations, 2018.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):1-12, 2018.
12

---

Published as a conference paper at ICLR 2025
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Repre- sentations, 2017. URL http://ssw//eeeeeeeee..em. net/ http:///eaneeeeeeeeeeeeeeeeoaaaa fo
Seyed Majid Naji, Azra Abtahi, and Farokh Marvasti. Efficient sparse artificial neural networks. ArXiv, abs/2103.07674 2021.
Shreyas Malakarjun Patil and Constantine Dovrolis. Phew: Constructing sparse networks that learn fast and generalize well without training data. In International Conference on Machine Learning, pp. 9432-8442. PMLR, 2021.
Hoang Pham, The-Anh Ta, Shiwei Liu, Lichuan Xiang, Dung D. Le, Hongkai Wen, and Long Tran-Thanh. Towards data-agnostic pruning at initialization: What makes a good sparse mask? In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL https //openreview .waleew.neeeeee net X
Ilan Price and Jared Tanner. Dense for the price of sparse: Improved performance of sparsely initialized networks via a subspace offset. In International Conference on Machine Learning, pp. 8620-8629. PMLR, 2021.
Jingtong Su, Yihang Chen, Tianle Cai, Tianhao Wu, Ruiqi Gao, Liwei Wang, and Jason D Lec. Sanity-checking pruning methods: Random tickets can win the jackpot. Advances in Neural Information Processing Systems, 33:20390-20401, 2020.
Zihao Sun, Yu Sun, Longxing Yang. Shun Lu, Jilin Mei, Wenxiao Zhao, and Yu Hu. Unleashing the power of gradient signal-to-noics noise ratio for zero-shot nas. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5763-5773, 2023.
Hidenori Tanaka, Daniel Kunin, Daniel L. Yamins, and Surya Ganguli. Pruning neural networks with- out any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33:6377-63389, 2020.
David A Van Veldhuizen, Gary B Lamont, et al. Evolutionary computation and convergence to pareto front. In Late breaking papers at the genetic programming 1998 conference, pp. 221-228. Citeseer, 1998.
Proceedings of the European conference on computer vision (ECCV), pp. 3-18, 2018. Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In
Artem Vysogorets and Julia Kempe. Connectivity matters: Neural network pruning through the lens of effective sparsity. arXiv preprint arXiv:2107.023006 2021.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020. URL htttps://cpenreiii
net/ forum poroumiidee wtw.ummmddskksssKCC..
Huan Wang, Can Qin, Yue Bai, Yulun Zhang, and Yun Fu. Recent advances on neural network pruning at initialization. In Luc De Raedt (ed.), Proceedings of the Thirty-Firs! International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pp. 5638-5645. ijcai.org, 2022. doi: 2.44963/jaai.22221177 URL https: doi 111.:/0...99911111 /0L 2022 786
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsieh. Rethinking architecture selection in differentiable NAS. In International Conference on Learning Representa- tions (ICLR), 2021. UtL/atts:///ppeeeeeee httppp/ewwweiiiieeeeeeeeaaaaaaaa
net http://ooooo...ddaaa..aeeeee.e
Lichuan Xiang, Rosco Hunter, Minghao Xu, Lukasz Dudziak, and Hongkai Wen. Exploiting network compressibility and topology in zero-cost nas. In AutoML Conference 2023, 2023.
Yujia Xie, Hanjun Dai, Minshuo Chen, Bo Dai, Tuo Zhao, Hongyuan Zha, Wei Wei, and Tomas Pfister: Differentiable top-k with optimal transport. Advances in Neural Information Processing Systems, 33:20520-200 205311 2020.
13

---

Published as a conference paper at ICLR 2025
Sparsity
Architecture
68.38%
90.00%
96.84%
99.00%
ResNet20- CIFAR-10
1.0/1.0
0.99/0.8
0.99/0.5
0.99/0.5
VGG19 CIFAR-100
1.0/0.1
0.99/0.9
1.0/0.5
1.0/0.5
ResNet18 Tiny-Imagenet
1.0/0.1
1.0/0.1
0.99/0.5
1.0/0.5
Table 2: The best hyperparameters a/3 for each experiment.
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory-economic sparse training framework on the edge, Advances Neural Information Processing Systems, 34:20838-20850,
2021a
Xin Yuan, Pedro Savarese, and Michael Maire. Growing efficient deep networks by structured continuous sparsification. In 9th International Conference Learning Representations (ICLR), 2021b.
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search. In International Conference on Learning Representations (ICLR), volume 3, pp. 7, 2020.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. Advances in neural information processing systems, 32, 2019.
Michael Zhu and Suyog Gupta. To prune, not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv.1710.0187., 2017.
A HYPER-PARAMETERS OF DPAI
Like other Pal methods, DPal's hyperparameters need be tuned for optimal performance. We observed that a sufficiently large learning rate and an adequate number updates can help DPal converge effectively. In all experiments, we utilized the Adam optimizer optimize the score parameters and masks, setting the learning rate to 0.005 and the number of update steps to 3000. We performed grid search to find the best values for and B. Specifically, in the set 0.1, 0.5, 0.9, 0.99, 1.0, 0.0 and / 0.1, 0.5, 0.8. 0.9, 1.0, 0.0. The following table P presents the best hyperparameters and 3 identified for each experiment. As seen in Figure 2 the best hyperparameters tend to prioritize the Node and Kernel Objectives.
searched for
B DETAIL RESULTS FOR PRUNING TIME AND NETWORK FLOPS
Table 3: Pruning time and FLOPs of subnetworks for different pruning methods and compression ratios on Resnet18- ImageNet.
Pruning time (seconds)
FLOPs (108)
Sparsity (%)
68.38
90.00
96.84
99.00
68.38
90.00
96.84
99.00
SNIP
5.14
4.95
5.55
5.64
11.35
5.77
3.04
1.55
Iter-SNIP
229.16
235.34
233.19
231.23
10.73
7.05
3.98
1.97
SynFlow
108.17
96.18
91.15
92.60
14.71
8.91
4.24
1.50
PHEW
5511.20
1342.03
471.23
324.78
14.29
8.35
3.92
1.50
NPB
380.52
375.65
384.32
387.89
14.37
5.21
1.74
0.59
DPal (ours)
88.77
88.77
88.77
88.77
14.37
5.20
1.74
0.60
For DPal method, compute our loss function to update differentiable scores based statisties from each layer. For the d-NPB component, we need to compute effective nodes and paths. No- ticeably, in our implementation, calculate those statistics sequentially over each layer for simple
14

---

library and Published as a conference paper at ICLR 2025
Table 4: Pruning time and FLOPs of subnetworks for different pruning methods and compression
ratios on VGGI9 CIFAR-100.
Pruning time (seconds)
FLOPs (10")
Sparsity [%)
68.38
90.00
96.84
99.00
68.38
90.00
96.84
99.00
SNIP
5.15
4.96
5.12
4.55
17.952
7.806
3.686 1.816
Iter-SNIP
115.91
115.52
116.83
117.60
18.465
9.479
4.951 2.529
SynFlow
96.55
100.33
101.90
104.67
22.998
12.702
6.306 2.605
PHEW
6928.59
1699.80
605.65
417.25
22.108
11.746
5.611 2.340
NPB
430.52
438.20
412.16 16
425.33
22.035
8.773
2.874 1.046
DPal (ours)
71.88
71.88
71.88
71.88
22.035
8.773
2.873 1.046
Table 5: Pruning time and FLOPs of subnetworks for different pruning methods and compression ratios on Resmet20- CIFAR-10.
Pruning time (seconds)
FLOPs (10%)
Sparsity
(9))66833
90.00
96.84
99.00
68.38
90.00
96.84 99.00
SNIP
1.42
1.40
1.87
1.68
17,952
8.323
3,470 1.709
Iter-SNIP
58.21
54.60
52.02
57.61
18,465
9.698
4.510 2.022
SynFlow
57.46
56.24
53.43
54.13
22,998
11.549
4.263 1.633
PHEW
78.31
18.09
4.78
2.58
22,108
10.690
4.110 1.640
NPB
20.10
23.91
21.51
21.13
22,035
7.642
2.645
1.122
DPal (ours)
77.12 77.12 77.12
77.12
23.683
7.642
2.645 1.114
implementation, but those processes be parallel computations in each layer independent. We expect a further reduction in pruning time if parallel acceleration is employed. However, despite the lack of parallel acceleration, we can already that DPal does not significantly increase the pruning time while obtaining significantly better subnetworks.
Besides pruning time, find that the FLOPs reduction of subnetwork after pruning is more important than pruning before training. We have measured the FLOPs of subnetworks produced by different methods. The result indicates that our DPal benefits from the node-path principle that can produce subnetworks with lower FLOPs as NPB than other baselines. the same time, DPal outperforms the existing Pal methods.
C EXPERIMENT DETAILS
We describe our experiment settings on architectures and datasets. We use Pytorch conduct experiments on single A5000.
Datasets. Our main experiments are conducted with CIFAR-10 10, CIFAR-100, 100, and datasets, where:
CIFAR-10 is augmented by normalizing per-channel, randomly flipping horizontally.
CIFAR-100 is augmented by normalizing per-channel, randomly flipping horizontally.
-iny--Im is augmented by normalizing per channel, cropping 64x64, and randomly
flipping horizontally.
Architectures. We three different networks:
VGG-19 is a CIFAR-100 network used SynFlow (Tankaa et al. 2020 We choose a
batch-normalizatioo version.
htpps//pyyorch.oogg
15

---

Published as a conference paper at ICLR 2025
ResNet-20 is 20-layer CIFAR-10 10 version of ResNet created by He et al. (2016). This version has added batch normalization layers before each activation function.
ResNet-18 8 is ImageNet version with 18 layers adapted from SynFlow (Tanaka al. 2020
The first convolution has kernel size 3x3 (instead of 7x7) and max-pooling layer that follows has been removed.
We treat all of the weights from convolutional and linear layers of these networks are prunable parameters, but we do not prune the biases nor the weights in the batch normalization layers. The weights in convolutional and linear layers are initialized with Kaiming normal, while biases are initialized be zero.
Training details For training on final sparse network, the hyperparameters are chosen as follows:
Table 6: Summary of the architectures, datasets, and hyperparameters used in experiments.
Network
Dutasel
Epuchs
Batch
Optimizer
Momentum
LR
LR Drup, Epoch
Weight Decay
VGG-19
CIFAR-100 100
160
128
SGD
0.9
0.1
10x, [60.120]
0.0001
ResNet-20
CIPAR-10
160
128
SGD
0.9
0.1
10x, [60, 120]
0.0001
ResNet-18
Tmny-mmageNet
100
128
SGD
0.9
0.01
10x, [30,60,80]
0.0001
D LAYER-WISE EFFECTIVE NODES AND EFFECTIVE PATHS
We conducted experiments comparing ERK and Uniform layer-wise sparsity using DPal on ResNer18 as shown in Table. 7 The results indicate that ERK consistently performs better than Uniform, particularly in extreme sparsity settings. When the network reaches 99.90% sparsity, the network with Uniform layer-wy wise sparsity collapses, whereas ERK still produces a trainable network. Additionally, with ERK layer-wise sparsity, the effective nodes of the learned sub-network layer-wise sparsity results in several bottleneck layers
are distributed more harmoniously-unifor with very small number of effective nodes.
Table 7: Analyzing the impact of layer-wise sparsity, employing DPal on ResNet18 8 trained on the ImageNn dataset. "Log effective paths" refers to the logarithmic scale of the number of effective paths. "Test acc" indicates test set accuracy. "Layer-wise effective nodes" represents the count of effective nodes per layer.
Spursity 99.00%
Layer-wise
lug effective paths
acc
ERK
64-64-64-664-44 -2---6---8888- 128- 12-- 118-8-28-25662-80
62.47
44.93%
Unitarm
3.37.5051 61- 156 126
64,94
41.01%
99.68%
ERK
64 128-128-252 256 22226661112222222111111111133330
44.98
30.88%
Uniform
6- 28-22-16 21 32-66 56447.14 21
61
210
468
503
48
66.11
16.33%
99.90%
ERK
49.99
16.33%
Uniform
E EXTREMELY SPARSE NETWORKS
The extreme sparsity settings pose greater challenges due to the larger search space for the remaining parameters. Other methods often struggle to balance the number of effective nodes and effective paths. Our DPal method consistently outperforms NPB, especially with extreme sparsity (see the results in Table D. NPB's ' discrete optimization and approximate node-path balancing objectives for each layer make it difficult to find the optimal number of effective nodes and paths. In our observations, NPB could only find maximum of /516/1749/860 effective nodes for the 99.0099..0. 1tt.00%%99..6.. 668%/99 9.%9 sparsity settings, respectively. In contrast, DPal can discover sparse networks with higher numbers of effective nodes and effective paths, and this gap becomes more significant higher sparsity regimes.
F PRUNING EFFICIENTNET ON IMAGE-NET 1K DATASET
We employed Stochastic Gradient Descent (SGD) with Nesterov momentum for the training pipeline. Each model was trained for 150 epochs with a batch size of 256. The initial learning rate was set to 0.035 and decayed by factor of 0.99 for each epoch. The optimizer was regularized using I.2
16

---

Published as a conference paper at ICLR 2025
Table Results from experiments conducted on ResNeetB-Tiny-Imagehh under extreme sparsity settings. All experiments are conducted with random seeds (0-4) and results are reported as mean and standard deviation. "effective nodes" refers to the total count of effective nodes in the network. "Log effective paths" denotes the number of effective paths on logarithmic scale. "test acc" represents the accuracy of the test set.
Method
Sparsity
effective nodes
log effective paths
test acc ( (%)
DPal
99.68%
4974+7
45. 45.1240.22
30.73+0.27
99.90%
34964203
49.9620.93
15.6940.37
NPB
99.68%
500423
76.15=0.90
24.33+0.19
99.90%
626+32
64.06+1.58
11.7340.62
regularization with coefficient of 4e-5, Each model was trained using three random seeds (0, 1, 2) to ensure robustness, and the model was trained on Nvidia A100.
G EXERPIMENTS ON VIT
We also add further experiments with larger architectures, such ViT-B/16 (85 million params). We trained those networks using Tiny-lmageNet, and the results are shown in the following table:
Table Results on Vision Transformer (VIT) architecture: We compare Pal with baselines such as Synflow and Random ERK on ViT-B/16 at 99% sparsity, trained from scratch Tiny-ImageNer We use the ViT-B/16 architecture, following the implementation https://gittub.com/// how..b/////cccdii///iiitt. pytorch, which has 85 million parameters, and train it using the SGD optimizer with Warmup Cosine Schedule.
Method
eff nodes
log eff paths
test acc(%)
Random ERK
83844
186.54
17.40
Synflow
10465
219.27
29.55
DPal (ours)
82772
211.91
35.61
Experiments demonstrate that our approach effective in the linear layers of transformer architectures. However, we recognize that, in its current form, our method has not been fully explored for adaptation self-attention layers transformers. Adapting it function with those layers presents a promising direction for future work.
H NODE-PATH BALANCING PRINCIPLE IN PAI
Here, we investigate the relationship between the number of effective paths and effective nodes (kernels) in relation to performance. When consider only nodes (log R kernels (log Rc) with paths (log Rp), the results do not adhere to the node-path balancing principle, as some points with a higher number of effective paths and effective nodes (kernels) still underperform. However, when we compare performance using the new Effective Nodes Plus metric (log R + log Rc), the results demonstrate greater consistency. This suggests that incorporating effective kernels/connections as an extended definition of effective nodes into the Node-Path Balancing principle is natural progression.
METRIC DETAILS
Effective path. To exactly compute the number of effective paths, we remove the batch nor- malization layers and initialize all the remaining parameters to 1. Then, we put the input vector the network, and the number of effective paths is the sum of logits the output layer R ( we))1.
More precisely, we face problems with pooling layers in convolutional neural networks. With the max pooling layer, do modify its output. At that time, the result is the maximum number
17

---

21.0 Published as a conference paper at ICLR 2025
Sporsity 68.38%
Spirsity 90.00%
Sparsity 96 96.84%
Spariity 99, 00.%
Sthecbie Path Cogg 40380)
F 6
60.22
65.00
MA.5
ACCUSSC
60.3
64.50
-4-
5
9.5
5.50
8:00
9.43
830
84
15
83
70
Restiert Innggnnggeet
60.00
64
Efrestte Eerta (nd
Acouncs F
62.00
16
60.3
56-
00
60
153 354 11
16o 15.2
13.0
11.5
11.5
12.0
12.3
11.3
11.
11000tin Datte 360 60300
63.00
60.3
Accurey
69.77 1%
36
5
21.50 31.15 22/09 22 35 22 30.35
26,5 22.0
23.5
22.0
15.0 16.5
25.0
26.5
Figure 4: The accuracy for different hyperparameters settings of experiments ResNet18 8 Tiny- ImageNet along with the number of Effective Paths (y-axis) and the comparison between number of Effective Nodes, Kernels and Nodes Plus (x-axis) across sparsity levels
of paths in subnetworks. With the average pooling layer, since all inputs of this layer contribute to the output, we change the average operator to the sum operator to precisely compute the number of effective paths. We all use ReLU activation functions compute this metric since this function does not affect the calculations' results.
Effective parameter. We follow Frankle et al. (2021) when identifying which is the effective parameter. Similar computing effective paths, make further steps. After having the sum of logits, we compute the gradients of this sum concerning weights V R. Then, an unpruned weight has non-zero gradient, is effective and vice versa. Effective parameters dense edges that connect two effective nodes
Effective node/channel. With fully connected layers, if all connections to one node or out of one node pruned, this node is pruned. If connections exist to a node, but all of these connections ineffective, then this node becomes ineffective.
In convolutional layers, instead of nodes, we have channels. We consider kernel connection and a channel node and then convert the convolutional layer into fully connected layer. If all parameters in the corresponding kernel are removed, the connection is pruned. Finally, identifying the effective nodes/channels similar the process in fully connected layers.
J COMPARISON ON PREVIOUS METHODS
J.1 COMPARING WITH SPARSITY TRAINING AND POST-TRAIN PRUNING
While the top-K operation with the Straight-Through Estimator (STE) and the Erdös-Rényi Kernel (ERK) technique are well-established and have been adopted in prior works Gao al. (2022) Yuan
18

---

Published as a conference paper at ICLR 2025
et al. (2021b): Louizos et al. (2018); Liu et al. (2022a). our contribution lies in their application and
adaptation to the Pruning at Initialization (Pal) domain- - context distinct from the pruning methods
used during or after training Gao et al. 2022); Yuan et al. bb) Louizos et al. (2018) Liu et al.
(2022a): Lin et al. (2023): Veit & (2018)
Unlike traditional pruning techniques that optimize sparse masks after or during training Gao ct al. (2022); Yuan al. (202Tb): [ouizos et al. (2018): Liu et al. (2022a); Lin et al. (2023):
Veit & Belongie 2018) using task-specific loss functions (most commonly mean squared error in image classification) with regularization terms, our proposed method, Differentiable Pruning at Initialization (DPal), introduces differentiable masks specifically for Pal, enabling the identification of effective subnetworks before training commences. This novel application integrates network topology considerations, such as the Node-Path Balance (NPB) principle, into initialization sttggeeonn-ttteeeeeeee optimization. By leveraging the differentiable Node-Path Balancing (d-NPB, framework, our method balances effective nodes and paths to achieve superior trainability and performance by directly optimizing the differentiable NPB objectives and facilitating task aggnostc pruning.
Furthermore, while techniques like Gao et al. (2022); Yuan et al. (2021b); Louizos et al. (2018);
Liu et al. 2022a); Lin et al. (2023); : Veit & Belongie (2018) focus on pruning trained models or dynamically adjusting during training, DPal eliminates the need for iterative pruning and retraining cycles, significantly reducing computational overhead. Regarding ERK methods, we directly adopted
approaches fromLiu et al. (2022a), as they are standard techniques for NPB and PHEW; our method
is designed based on those findings.
J.2 COMPARED TO PREVIOUS PAI METHODS
Here, we briefly introduce the baseline methods used for comparison with our proposed method. The fundamental principle of Pruning at Initialization (Pal) methods is to calculate an important score for each parameter in the network and then prune the network by removing parameters with the lowest scores until the desired sparsity level is achieved.
SNIP (Lee et al. 2019a) determines importance scores by evaluating the sensitivity of each connection to the training loss. Iter-SNIP (de Jorge et al., (202T). an iterative extension of SNIP, uses the same importance scores but incrementally prunes the network from its dense state to the target sparsity level. Both SNIP and Iter-SNIP rely on data and task-specific training loss, which makes them fall under dettaddeendent Pal methods. SynFlow (Tanaka et al.) 2020). on the other hand, is an iterative, data-agnostic Pal method designed to maintain network connectivity even at extreme sparsity levels. Similarly, PHEW (Patil & Dovrolis (022]) is an iterative, data-agnostic Pal method that uses random walk biased toward higher weight magnitudes to identify and preserve critical input-output paths. NPB Pham al. (2023) introduces the node-pat balancing principle, suggesting that networks with a higher number of effective nodes and paths tend to achieve better performance. They also utilise a suboptimal discrete optimization approach to identify subnetworks with a high number of effective nodes and paths.
K DISCUSS PRUNING TIME WITH MODEL SIZE INCREASING
The proposed DPal method exhibits a complexity comparable to Iter-SNIP and Synflow, as all these methods iteratively update the scores of each parameter over multiple iterations. Each iteration involves a forward pass and a backward pass of the neural network. In contrast, PHEW employs random walk strategy biassed towards higher weight magnitudes to identify input-output sets to be preserved, continuing until the subnetwork reaches the predefined sparsity. Consequently, the complexity of PHEW depends on the number of parameters in the base network and the target sparsity level. Similarly, NPB relies on a discrete optimizer. with complexity scaling based on the number of parameters in the base model. To mitigate this, NPB partitions the parameters of each layer into chunks of no more than 16,384 parameters and solves the discrete optimization for each chunk, though this partitioning often results in suboptimal solutions.
Among the architectures compared, ResNet20 has the smallest number of parameters (272.474), while ResNet18 8 and VGG19 have significantly larger parameter counts at 11,271,23 1,22 and 00,0086,62.. respectively. Consequently, DPal, SNIP, Iter-SNIP, and Synflow demonstrate considerably lower pruning times than PHEW and NPB on ResNet18 and VGG19, underscoring their scalability for large-scale models. Moreover, since DPal and Synflow are data-agnostic methods, they further
19

---

Published as a conference paper at ICLR 2025
reduce pruning times when applied to larger-scale datasets such as Imae-mmae This highlights the superior scalability DPal and Synflow not only for large-scale models but also for extensive datasets.
In Figure 3, you may notice that our pruning method takes longer compared to some other methods. This is because our approach involves updating the pruning masks simultaneously in differentiable manner during optimization. While this differentiable updating introduces additional computational overhead upfront, it allows us to maintain a relatively consistent pruning time across different model sizes.
Most previous Prune-at at- Initialization (Pal) methods tend to show a significant increase in pruning time the model size grows. This is due to their iterative layer-wise pruning procedures, which become more time-consuming with larger models. In contrast, our method's ' simultaneous and differentiable mask optimization scales more efficiently with model size, ensuring that the pruning time does not increase substantially as the models become larger. As we presented in the appendix, pruning time for different sizes of the models (ResNet18, ResNet20, VGG19) under different sparsities (from 68.38 to 99%) are in the range of (70-90)s, while NPB can increase from the 20s to 430s, and PHEW can increase from 78.31s is to 6928.59s.
Therefore, although our method may take more time initially esspccallly compared to simpler methods on smaller models- it offers better scalability and efficiency for larger models. We believe this trade-off results in a more practical and effective pruning approach for models of varying sizes.
L DETAIL. ABOUT OUR OBJECTIVE DESIGNING
DPal updates the importance scores iteratively in differentiable manner, employing STE to estimate the gradients the importance scores relation to the discrete optimization problem. The mechanism by which DPal updates the importance scores to maximize the number of effective nodes and paths can be observed in the derivatives associated with each objective.
Path Objective (Equation Th: derivative with respect to the importance score st.f of a connection
shows that its score increases based on the number of incoming paths P(o and outgoing paths
8Rp it can connect. Specifically, P(u!-1) 6R represents the number of effective paths the )P(e)))
(P(u)")
connection can contribute to creating. This encourages the selection of connections that can form the
highest number of effective paths.
Node Objective (Equation T)e derivative with respect to the node objective indicates that the importance score of connection increases if it belongs to an ineffective node. Additionally, the score influenced by how many effective paths the connection can contribute to creating. This dynamic promotes the selection of parameters from diverse nodes, encouraging more nodes to become effective.
These mechanisms demonstrate that the optimized importance scores effectively result in subnetwork that maximizes the number of effective nodes and paths. Furthermore, through the convergence analysis in Section 3.3, we establish that each step of the iterative update can result in an incremental increase in the number of effective nodes and paths.
To better understand the convergence of the proposed method, we conducted empirical observa- tions through ablation studies, examining how the number of effective nodes, paths, and kernels changes during the optimization of each objective. Table 10 shows the number of effective nodes, paths (in logarithmic scale), and kernels in the subnetwork obtained by optimizing each objective independently.
The results when pruning ResNer20 at 99.68% sparsity indicate that:
The maximum number of effective paths (in In scale) 69.22, achieved by optimizing the path objective alone. (Visualization during training: 5a
The maximum number of effective nodes is 570, achieved by optimizing the node objective
alone. (Visualization during training: (bb)
The maximum number of effective kemels 860, achieved by optimizing the kernel
objective alone. (Visualization during training: 5c)
20

---

Published as a conference paper at ICLR 2025
Objective
Eff. Nodes
Eff. Kernels
Eff. Paths (In scale)
R
75
211
69.22
RN
570
740
22.30
Rc
168
860
47.97
Rp + RN Rc
321
822
42.16
Table 10: Number of effective nodes, kernels, and paths under different objectives.
Optimizing the overall Node-Path Balancing (NPB) objective must strike balance among these three objectives. (Visualization during training: 5d
nunthr effectivn kemol
22.5
ottecctvee
600
Namber dftediee sode/kenel
600
21.5
350
-1
300
200
20.5
Aunber
a00
number of ettective rodes
offective
100
20
mimber of ettective patts
0
500
1000
1500
2000
2500
3000
500
1000
1500
2000
2500
3000
(a) The optimisation of R.P.
(b) The optimisation of R.N-
1001
-00
Nunber of effectire oodes wndee Rcsees
Nimthe of effecttre pubtr scal))
700
40
-3
600
memter of otfecctvve modes
rumber effective modes
effective
number of effection kemal
mmather of otfective paths
offfectvve palls
000
Nimber
300
0
300
1000
1500
2000
2500
3000
500
1000
1500
2000
2500
3000
Step
Step
(c) The optimisation Re-
(d) The optimisation of R
Figure 5: Visualizations of effective paths, nodes, kernels, and convergence during training.
L.1 RATIONAL OF CHOSEN LOGARITHMIC
The main purpose of using the logarithmic scale for the number of effective nodes and effective paths in our differentiable objective is to address the imbalance between these objectives, discussed in Section 3.2, lines 177-183. The number effective paths typically far greater than the number of effective nodes connections. For example, in ResNet18 8 under 68.38% sparsity, the number of effective paths can reach up 1065 (see Figure ablation study on hyperparameters), while the number of nodes is only around 5000. Mathematically. the logarithmic function is an optimal choice for handling such large numbers, as log z grows the slowest approaches infinity.
We kindly point out that the theoretical justification for employing the logarithmic scale embedded in the computation of the derivative for each objective under the logarithmic scale. The derivative
21

---

Published as a conference paper at ICLR 2025
with respect to the path objective in equation (2) is proportional by the number of effective paths that may contain iig which itself is very large number similar to the total number of effective paths. However, with the logarithm scale, the derivative now divided by the total number of effective paths, making it relatively smaller.
L.2 RATIONAL OF CHOSEN STE
The principle of pruning revolves around identifying the importance score for each parameter and removing those with the lowest scores. Applying the Top-K function to these importance scores is natural fit for this task, especially when pruning the network achieve specific sparsity level. DPal determines the importance score in a differentiable manner by applying the Top-K function on these scores compute the NPB objective, updating the scores using the Straight-Throuugh Estimator (STE). While there exist soft and differentiable alternatives to the Top-K K function(Gao et al. (2022): ;
Xie et al. (2020), the hard Top-K K function with STE appears to be computationally more effictent and aligns well with our objective, which focuses solely on maximising the number of effective nodes and paths rather than their specific importance scores. Additionally, we provide both theoretical analysis on the convergence of the proposed objective and experimental validation, demonstrating that the method with the straightforward STE effectively optimises the number of effective nodes and paths the pruned subnetworks.
To the best of our knowledge, DPal is the first method to apply the Top-K K function with STE to address a non-differentiable optimization problem, specifically Pal. While previous works have utilised the Top-K function with STE for non ri-ddferrentialle optimization problems in pruning during or after training, their focus has been optimising the training loss along with regularisation term for reducing model FLOPs. However, these approaches require extensive training of the neural network, making them unsuitable for Pal settings. contrast, our proposed method focuses solely on optimising the network topology using the NPB objective, which is data-independent and significantly more computationally efficient.
L.3 RATIONAL FOR TANH FUNCTION
The tanh(-) function naturally aligns with our objective since it only counts the number of ef- fective nodes. Given that (-)) O represents the number of effective paths passing through node/channel or kernel/connection, the tanh(-) function outputs tanh(N(-)) = for effective nodes and tanh(NV()) 0 for ineffective nodes.
Alternatively, any activation functions f(x) with similar characteristics can be employed. For instance, the sigmoid function (Ix) - can be adapted for this purpose using f(x 20(xx) This modification with sigmoid function yields derivatives computed as 2o (x) (1 o (x)), which exhibits behaviour consistent with the analysis presented in Section 3.3.
22


Reviews:
[{"review_id": "Xda4SZzDD9", "reviewer": "Reviewer_QU2t", "summary": "This paper introduces DPaI (Differentiable Pruning at Initialization), a novel method for pruning neural networks at initialization using the Node-Path Balancing (NPB) principle. Unlike traditional methods, DPaI applies a differentiable approach to optimize pruning masks, enabling efficient gradient-based optimization of sparse networks without needing training data. By balancing the number of effective nodes and paths, DPaI maintains information flow in sparse networks, achieving superior accuracy at high sparsity levels. Experimental results show that DPaI outperforms state-of-the-art methods across various network architectures and datasets, including CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1K. DPaI also achieves lower pruning time and reduced FLOPs, demonstrating its efficiency and practical applicability.", "questions": "I currently hold a positive view of this work, but I’m somewhat lacking the sense of recent progress about pruning at initialization. I believe the most critical aspect is comparing it with other works in the same track, so I may further adjust my review after considering feedback from other reviewers.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Differentiable Optimization: DPaI introduces a differentiable approach, making pruning more efficient and compatible with standard neural network training, unlike discrete, layer-wise pruning methods.\n2. High Performance at Extreme Sparsity: By balancing effective nodes and paths, DPaI maintains high accuracy even at sparsity levels of 96-99%, outperforming traditional methods.\n3. Architecture-Agnostic: DPaI works effectively across different architectures, such as ResNet, VGG, and EfficientNet, making it broadly applicable.\n4. Reduced Pruning Time and FLOPs: DPaI achieves low pruning times and significantly reduces FLOPs, enhancing computational efficiency compared to other methods.", "weaknesses": "1. Hyperparameter Sensitivity: I assume DPaI’s performance will depend on hyperparameters which may require tuning for different datasets and sparsity levels, as this usually happens to pruning at initialization methods.\n2. Implementation Complexity: DPaI involves complex gradient calculations and differentiable mask updates, which may be harder to apply to different task or models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JjIIPuV61U", "reviewer": "Reviewer_CGAa", "summary": "This paper extends Node-path Balancing Principle (NBP) into differentiable formulation by the learnable masks. To solve the gradient flow problem of top-k operation, authors introduce the Straight-Through Estimator (STE) technique during mask updating, while using Erdo ̋s-Rényi Kernel (ERK) technique to determine the layer-wise sparsity ratios.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The differentiable NPB formulation achieves competitive performance compared to previous Pruning at Initialization (PaI) methods.\n2. The proposed method can be easily integrated into the previous gradient-based methods.", "weaknesses": "1. This paper employs methods that are well-established in prior works, such as the top-k operation with Straight Through Estimator (STE) [1, 2, 3], and the Erdős-Rényi Kernel (ERK) technique[4]. Although the differentiable masks have not been previously applied in the PaI domain, many methods[1, 2, 3, 5, 6] in pruning during/post training methods have used the differentiable masks for model compression. I suggest the authors provide a detailed discussion regarding the difference.\n\n2. This paper uses the tanh activation function for the mask in the top-k with STE method, particularly when sigmoid and its variants are more commonly employed in previous works[1, 2, 3]. Therefore, what is the rationality of the tanh function. \n\n3. The models and datasets employed in the paper are predominantly small-scale validations. How does the method proposed in this paper perform on larger-scale models and datasets, such as experiments using Resnet-50 on ImageNet-1k?\n\n4. In Fig.3, why does the proposed pruning method in the paper take more time in the Resnet20 on CIFAR-10 experiments compared to other methods, while it takes less time in the Resnet18 on Tiny-ImageNet and VGG19 on CIFAR-100 experiments? \n\n5. I suggest including a brief introduction or citation for the abbreviations of previous methods used in this paper, as relying solely on abbreviations from related work can reduce the readability of the paper for readers who are not fully familiar with PaI-related research.\n\n[1] Disentangled Differentiable Network Pruning.\n[2] Growing Efficient Deep Networks by Structured Continuous Sparsification.\n[3] Learning sparse neural networks through l_0 regularization.\n[4] The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training.\n[5] Filter Pruning for Efficient CNNs via Knowledge-driven Differential Filter Sampler.\n[6] Convolutional Networks with Adaptive Inference Graphs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nXmP0l5pVW", "reviewer": "Reviewer_gcQx", "summary": "(+) The authors propose a novel pruning method referred to as Differentiable Pruning at Initialization (DPaI), which involves differentiable optimization of the pruning mask. DPaI adopts a dynamic and adaptable pruning process, allowing easier optimization processes and better solutions. \n\n(+) DPaI’s differentiable formulation enables easy use of the gradient-based methods for Pruning at Initialization (PaI). \n\n(+) The empirical results demonstrate that DPaI significantly outperforms current state-of-the-art PaI methods on various architectures, including Convolutional Neural  Networks (CNN) and Vision-Transformers (ViT).", "questions": "Please see the above weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(+) The author analyzes the interpretations of the proposed method (DPaI) and proves its behavior using convergence theory.\n\nPlease see the strength of the paper in summarization further.", "weaknesses": "(-) It isn't easy to follow how the proposed method (DPaI) finds the best nodes and paths using only an importance score. \n\nThrough extensive ablation studies on the update rule (line 8) stated in Algorithm 1, the authors should show the effectiveness: the importance of nodes and paths seems to depend only on score s_{i,j}. The empirical observations (small neural networks) with the ablation studies help us understand the contributions of the proposed method, DPaI, and the convergence theory. \n\n(-) What about pre-trained model-based DPaI performances of ViT?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oPFGpH15fA", "reviewer": "Reviewer_v68w", "summary": "Update:\nThanks for the authors' feedback. My concerns have been fully addressed.\n\n----- original review -----\nThe authors propose a novel differentiable algorithm for deep network pruning at initialization. The key idea of the algorithm is to optimize the number of effective paths and nodes while pruning the network parameters. The non-differentiable issue can be addressed by Straight-Through Estimator. The authors demonstrate the superiority of the proposed method on several popular benchmark datasets.", "questions": "* Why choosing logarithm in NPB principle? Is logarithm loss an optimal design in this problem?\n* Is the method applicable to LLM? How it performs on popular LLMs such as Llama-3.2 families?\n* Why using STE while there are several possible methods to relax the differentiable issue? For example the popular [0,1] + L1 norm relaxation seems a good choice too.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed algorithm presents a novel approach to Node-Path Balancing in network pruning. It distinguishes itself by incorporating this methodology into a differentiable framework. Additionally, the article provides extensive numerical results for various vision tasks and includes detailed ablation studies, offering readers valuable insights into the performance of the proposed algorithm under different conditions. Furthermore, the authors provide some convergence analysis, which can be useful for those interested in understanding the underlying principles of the method.", "weaknesses": "* The experiments are mostly for vision models. It would be nice to see some LLM experiments. For exmaple, how does the method perform on Llama-3.2-90B?\n*  It is unclear why choosing logarithm in maximizing NPB principle. For example, why not using square root or other functions? Please suggest the theoretical support of this choice, and conduct numerical ablation studies to verify the optimality of this design. A possible approach is to make this loss function learnable, then find an optimized loss function on a set of small datasets, finally generalize to larger datasets.\n\n* Using STE to convert non-differentiable optimization to differentiable one has been proposed in many previous works. Please justify the novelty of this approach when appling STE to this work. Any special adaptation to make STE more efficient / robust in this problem?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)