Title: Towards Data-Agnostic Pruning At Initialization: What Makes a Good Sparse Mask?
Venue: NeurIPS 2023
Accepted: yes
Keywords: Pruning Neural Network

Sparsity

Neural Architecture Search
Abstract: Pruning at initialization (PaI) aims to remove weights of neural networks before training in pursuit of training efficiency besides the inference. While off-the-shelf PaI methods manage to find trainable subnetworks that  outperform random pruning, their performance in terms of both accuracy and computational reduction is far from satisfactory compared to post-training pruning and the understanding of PaI is missing.  For instance, recent studies show that existing PaI methods only able to find good layerwise sparsities not weights, as the discovered subnetworks are surprisingly resilient against layerwise random mask shuffling and weight re-initialization.
In this paper, we study PaI from a brand-new perspective -- the topology of subnetworks. In particular, we propose a principled framework for analyzing the performance of Pruning and Initialization (PaI) methods with two quantities, namely, the number of effective paths and effective nodes. These quantities allow for a more comprehensive understanding of PaI methods, giving us an accurate assessment of different subnetworks at initialization. We systematically analyze the behavior of various PaI methods through our framework and observe a guiding principle for constructing effective subnetworks: *at a specific sparsity, the top-performing subnetwork always presents a good balance between the number of effective nodes and the number of effective paths.*
Inspired by this observation, we present a novel data-agnostic pruning method by solving a multi-objective optimization problem. By conducting extensive experiments across different architectures and datasets, our results demonstrate that our approach outperforms state-of-the-art PaI methods while it is able to discover subnetworks that have much lower inference FLOPs (up to 3.4$\times$). Code will be fully released.

Towards Data-Agnostic Pruning At Initialization: What Makes a Good Sparse Mask?

Abstract

Pruning at initialization (PaI) aims to remove weights of neural networks before training in pursuit of training efficiency besides the inference. While off-the-shelf PaI methods manage to find trainable subnetworks that outperform random pruning, their performance in terms of both accuracy and computational reduction is far from satisfactory compared to post-training pruning and the understanding of PaI is missing. For instance, recent studies show that existing PaI methods only able to find good layerwise sparsities not weights, as the discovered subnetworks are surprisingly resilient against layerwise random mask shuffling and weight re-initialization. In this paper, we study PaI from a brand-new perspective - the topology of subnetworks. In particular, we propose a principled framework for analyzing the performance of PaI methods with two quantities, namely, the number of effective paths and effective nodes. These quantities allow for a more comprehensive understanding of PaI methods, giving us an accurate assessment of different subnetworks at initialization. We systematically analyze the behavior of various PaI methods through our framework and observe a guiding principle for constructing effective subnetworks: at a specific sparsity, the top-performing subnetwork always presents a good balance between the number of effective nodes and the number of effective paths. Inspired by this observation, we present a novel data-agnostic pruning method by solving a multi-objective optimization problem. By conducting extensive experiments across different architectures and datasets, our results demonstrate that our approach outperforms state-of-the-art PaI methods while it is able to discover subnetworks that have much lower inference FLOPs (up to 3.4 × ). Code is available at: https://github.com/pvh1602/NPB .

1 Introduction

Deep neural networks have achieved state-of-the-art performance in a wide range of machine learning applications [3, 13, 39, 38]. However, the huge computational resource requirements limit their applications, especially in edge computing and other future smart cyber-physical systems [23, 48, 37, 47, 2]. To overcome this issue, a number of approaches have been proposed to reduce the size of deep neural networks without compromising performance, among which pruning has received voluminous attention [10, 24, 8]. Traditional pruning approaches mainly focus on accelerating inference, which usually require a pre-trained dense network [33, 20, 31].

As large language models (LLMs) [3, 43, 44] continue to gain popularity, endeavors start to explore the possibility to prune models before training while matching the dense performance. Lottery Ticket

Hypothesis (LTH) [15, 4, 5] provides empirical evidence for this research goal by discovering sparse subnetworks that can be trained from scratch to the comparable performance of the dense network. However, LTH typically involves the costly iterative pruning and re-training process, whose overall cost is much more than training a dense network.

This issue raises an intriguing research question: How to identify sparse, trainable subnetworks at initialization without pre-training? Specifically, a successful pruning before training method can significantly reduce both the cost of memory and runtime, without sacrificing performance much [46]. This would make neural networks applicable even in scenarios with scarce computing resources [1, 47]. As such, many methods for PaI have been proposed [26, 42, 9, 45, 1, 29]. While these methods are based on a number of intuitions (e.g., leveraging the gradient information [26, 45]), they typically measure the importance of network parameters. Their performance in terms of both accuracy and computational reduction is far from satisfactory compared to post-training pruning and the understanding of PaI is missing. More recently, Frankle et al. [17], Su et al. [40] observe a rather surprising phenomenon: for PaI methods, layerwise shuffling connections of pruned networks does not reduce the network's performance, which suggests that layerwise sparsity ratios might be more important than weight-level importance scores. This indicates that in searching for good subnetworks at initialization, the topology of subnetworks, particularly, the number of input-output paths and active nodes, plays a vital role and should be investigated more extensively.

In this paper, we first present a counter-argument against the previous findings: while Frankle et al. [17] show that PaI methods are insensitive to random shuffling, we find this is not true in the extreme sparsity regime ( > 99%), as the number of effective paths heavily suffers from random shuffling. In layerwise shuffling experiments (see Section 3.3), shuffling connections results in more effective nodes but substantially fewer input-output paths. In normal sparsity levels, shuffling weights in regular sparsities can maintain and even increase effective parameters [17] and the number of activated nodes [36]. This increases representation capacity and performance of the subnetwork. However, at extremely sparse levels, shuffling still preserves roughly the same number of effective nodes, but the performance of shuffled subnetworks drops significantly compared to their unshuffled counterparts. This is because random weight shuffling damages effective paths and hampers information flow within the subnetwork. These findings suggest that separately considering effective paths or nodes is insufficient to fully capture the behavior of subnetworks generated by PaI methods.

To underscore the critical importance of nodes and paths in designing network architecture, we conduct an analysis of their quantities in the context of Neural Architecture Search (NAS). For instance, Figure 1 illustrates the relationship between candidate's performance, nodes and paths in NAS-Bench-Macro [41] which has 6 , 561 models (more in Section 3.4). To ensure fairness in terms of model size, we compare networks with similar number of parameters (i.e., in a range of 300k parameters in Figure 1). Remarkably, architectures exhibiting a higher number of input-output paths and active neurons concurrently yield superior performance. This highlights the crucial role of simultaneously considering both node and path in the successful design of networks at initialization.

9.25

9.30

9.35

9.40

9.45

9.50

9.55

The number of effective nodes (ln scale)

35

40

45

50

55

The number of effective paths (log scale)

Architectures with parameters 2150k-2450k

82

84

86

88

90

92

Test Accuracy

To improve upon the current understanding of PaI methods, we introduce a novel framework from the perspective of subnetwork topology to provide a more accurate and comprehensive explanation of the performance of different approaches . In particular, we propose the joint usage of both the number of input-output paths (a.k.a. effective paths) and activated (i.e., effective) nodes to interpret behaviors of different PaI methods in a more comprehensive way (see Section 3.3 for more details). Inspired by the guiding principle observed through our framework, we further introduce NPB, a novel pruning at initialization by solving a node-path balancing objective. We show that NPB outperforms state-of-the-art PaI methods in almost all settings, while achiveing much lower inference FLOPs . In summary, our main contributions are:

We propose a unified effect node-path perspective to understand the behavior of PaI, particularly considering metrics of effective nodes and paths as proxies for the performance of PaI methods (Section 3). We revisit the layerwise shuffling sanity check on PaI methods and provide unified explanations for their behaviors based on these metrics in a wide range of sparsities (Section 3.3).

We discover a new relation between the proposed metrics and the performance of subnetworks, termed the Node-Path Balancing Principle, that suggests a non-trivial balance between nodes and paths is necessary for optimal performance of PaI (Section 4).

We present a novel data-agnostic pruner, NPB, by solving a multi-objective optimization problem to give a proof-of-concept for our principle. With extensive experiments, we show that NPB outperforms the best state-of-the-art PaI methods, PHEW, on 11 out of 12 settings while ours produces more efficient subnetworks with much fewer inference FLOPs (up to 3.4 × ) and faster in pruning time than PHEW (up to 14.5 × ) (Section 5).

2 Related Work

Neural Network Pruning. Neural network pruning methods [25, 21, 20, 12, 32] traditionally focus on pruning trained models based on pre-defined criteria, and then resulting subnetworks will be fine-tuned to converge. Recently, [15, 16, 4, 5] empirically show the existence of randomly initialized subnetworks (lottery tickets) which when trained from scratch or in early training iterations, that can achieve competitive performance with their dense counterparts. Unfortunately, finding lottery tickets is computationally expensive due to the train and prune cycle [16, 15]. Gradual pruning methods [49, 18] interleave the pruning and training, which are usually cheaper than pruning after training, but the network still needs to be trained to choose the ideal sparse subnetwork. Other methods [6, 7] apply one-shot pruning during training to further reduce the computational cost. Dynamic sparse training [30, 14, 28, 27], on the other hand, start with a (random) sparse network and update network connectivity during training. While pruning before training methods [26, 45, 36, 42, 1] determine subnetworks by the network initialization, gradient information, and network topology. However, experimental results done by Frankle et al. [17], Su et al. [40] show that current criteria of PaI methods may not be sufficient to obtain a subnetwork with good performance.

Pruning and Network Shape. Since PaI methods do not utilize training data [42, 36] or use only negligible portions of data [26, 45] to obtain gradient information without training, the configuration of nodes and connections is an essential source of information for optimizing the performance of pruned networks. It turns out that some PaI methods implicitly optimize certain aspects of network shape. In particular, SynFlow [42] preserves the number of input-output paths as synaptic strength, but often creates more isolated neurons in pruned networks. The works of Patil and Dovrolis [36] and Gebhart et al. [19] aim to preserve proxies in terms of path kernels which are also directly related to the shape of subnetworks. Furthermore, while PHEW [36] additionally implements random walks to increase the number of effective nodes, it unintentionally decreases the number of input-output paths. Our new point of view on node-path balancing would be helpful to systematically optimize network configuration for better performance. Other works also consider the number of effective nodes and effective paths to capture the capacity of pruned subnetworks [45, 34] where these numbers are considered separately.

3 Methodology

3.1 Pruning at Initialization Methods

Given a L layer neural network, we denote w = ( w 1 , . . . , wL ) as the set of vectorized weights. Pruning generates binary mask vectors m ℓ ∈ { 0 , 1 } d ℓ ( d ℓ is the number of connections in layer ℓ ) yielding sparse neural networks with sparse weights m ℓ ⊙ w ℓ - the elementwise product of masks and weights. Sparsity is defined as the fraction of weights being removed: s = 1 -P m ℓ P d ∈ [0 , 1].

ℓ

A pruning method usually consists of two operations: score and remove , where score takes as input weights of the network and outputs an important score for each weight: z ℓ = score ( w ℓ ) ∈ R ℓ ; then remove takes as input the scores z = ( z 1 , . . . , zL ) and the sparsity s , and outputs the masks m ℓ with

overall sparsity s . Pruning can be done in one-shot or iteratively. For one-shot pruning, we only generate the scores once, then prune the network up to sparsity s . For iterative pruning, we repeat the processes of the score, then prune from sparsity s ( t -1) / T to s t / T repeatedly T times.

Random. This method assigns each connection with a random score from a uniform distribution U (0 , 1). Random pruning empirically prunes each layer to target sparsity s [29].

SNIP. SNIP was introduced by Lee et al. [26] with the pruning objective of reducing connection sensitivity to the training loss. One passes a mini-batch of data through the network and computes the score z for weight w of SNIP as z = | w ⊙ ∇ w L| .

Iterative SNIP. This is an iterative variant of SNIP [9] with the same important score. But, iterative SNIP gradually prunes the remaining weights with lowest scores from sparsity s t -1 T to sparsity s t T iteratively T times for t = 1 , 2 , . . . , T .

SynFlow. SynFlow [42] is an iterative and data-agnostic PaI method. The pruning objective of SynFlow is to make the network remains connected until the most extreme possible sparsity. The weight scores are computed as follows. One first replaces all weights in the network by their absolute values. Then, an 1 input tensor is passed through the network, and one computes the sum of the logits as R = 1 ⊤ ( Q L ℓ = 1 | w ℓ | ) 1 . Finally, the score of weight w is computed as z = | w ⊙∇ w R | . SynFlow prunes the network iteratively T times.

PHEW. PHEW [36] is also an iterative and data-independent PaI method. It selects a set of inputoutput paths to be preserved. These paths are chosen through random walks, biased towards higherweight magnitudes. The selection starts with a unit that is selected through round robin procedure. This process continues until the subnetwork achieves the predefined sparsity.

ERK. Erd˝ os-Rényi (ER) first used by Mocanu et al. [30] to sparsify Multilayer Perceptron (MLP) networks using a random topology that allocates higher sparsity to larger layers. Evci et al. [14] extends ER to a convolutional version called Erd˝ os-Rényi-Kernel (ERK) which scales the sparsity of the convolutional layer in proportion to the number of neurons/channels in a layer.

3.2 Metric Definition

3

2

4

1

7

6

8

5

0

0

0

0

0

0

0

0

1

0

0

1

1

0

5

4

8

2

1

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

c

b

d

a

0

0

0

0

In a sparse network, it is intuitively clear that one should arrange the connections into a configuration neither too thin nor too spread-out to have good information propagation during training. For a better measurement, we propose using two metrics to evaluate the quality of subnetworks. Please refer to Appendix B for detailed discussions and Python code for calculating the metrics.

Effective Path. We define a path to be effective if it connects an input node to an output node without interruption (see Figure 2). Metrics based on paths are mentioned in [42, 19] as l 1 and l 2 path norms, respectively. In this paper, we only take into account the number of paths.

Effective Node/Channel. A node/channel is effective if at least one effective path goes through it (demonstrated as the right part in Figure 2). This concept is also considered in [36, 17]. For convolutional layers, we consider a kernel as a connection, and a channel as a node, and then convert the convolutional layer into a fully connected layer (see the left part in Figure 2).

3.3 Layerwise Shuffling Phenomenon

In this section, we investigate the intriguing phenomenon that layer-wise reshuffling the subnetwork found by PaI methods still produces competitive accuracy [17, 40]. Based on metrics, we provide a

SNIP

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

0.2

0.4

0.6

0.8

Effective path ratio

Unmodified

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

0.2

0.4

0.6

0.8

1.0

Effective node ratio

Unmodified

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

20

40

60

80

Top-1 Accuracy (%)

Unmodified

SynFlow

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

0.2

0.4

0.6

0.8

Effective path ratio

Unmodified

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

0.0

0.2

0.4

0.6

0.8

1.0

Effective node ratio

Unmodified

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

20

40

60

80

Top-1 Accuracy (%)

Unmodified

PHEW

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

Sparsity (%)

0.2

0.4

0.6

0.8

Effective path ratio

Unmodified

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

Sparsity (%)

0.0

0.2

0.4

0.6

0.8

1.0

Effective node ratio

Unmodified

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

Sparsity (%)

20

40

60

80

Top-1 Accuracy (%)

Unmodified

new way to understand why reshuffling subnetworks work and when they fail. We first use three PaI methods, i.e., SNIP [26], SynFlow [42], and PHEW [36], to find the subnetworks. Then, we randomly shuffle the pruning mask ml . All the subnetworks (both unmodified and shuffled) are trained in the same setting. Finally, we compute the ratio of the number of active paths and nodes after and before pruning and visualize the average scores in Figure 3.

In SNIP and SynFlow, the number of effective nodes drops significantly as sparsity increases (blue lines in second columns in Figure 3). After reshuffling, the connections are distributed uniformly in each layer leading to wider subnetworks while the number of effective paths decreases. In contrast, PHEW focuses on increasing the number of effective nodes by gradually adding new paths such that the network is as wide as possible. Consequently, reshuffling hurts the concrete network configuration and then reduces both the number of effective nodes and effective paths as sparsity is higher.

At sparsity levels below 99%, layerwise shuffled subnetworks demonstrate competitive performance or even better than unmodified counterparts, especially with SynFlow and SNIP. In addition to effective connection preservation as discussed in prior work [17], the representation capacity of shuffled sparse networks is enhanced with SynFlow and SNIP, attributed to the increase in the number of effective nodes [35] while maintaining considerable input-output paths. In more details, Figure 3 shows that these two metrics for SynFlow and SNIP shuffled versions closely resemble to unmodified subnetworks of PHEW with corresponding sparsities.

However, when the network becomes more sparse, the number of input-output paths decreases substantially (along with the reduction of effective parameters see Appendix D). Even though layerwisely shuffled networks become wider, the limited number of effective paths damages the information flows in the subnetworks . These explain why the accuracy of shuffled subnetworks is reduced significantly compared to the unmodified ones in intensive sparsities.

These observations indicate that increasing the number of active paths or nodes alone might not be sufficient in the design of PaI methods. We hypothesize that to have good subnetworks, the number of effective paths and effective nodes should be concurrently considered. If we balance these two metrics well, the performance after the training of subnetworks will be enhanced.

3.4 NAS Observations

Our research delves into the fundamental aspects of network architecture design by investigating the interplay between nodes and paths. Going beyond the realm of pruning literature, we conduct an analysis of node and path metrics in the NAS benchmark. It is noteworthy that networks in NAS are dense networks. In particular, we focus on NAS-Bench-Macro [41], a macro benchmark comprising 8 searching layers. Each layer offers three candidate blocks, resulting in a staggering 6,561 distinct networks with parameter counts ranging from 300k to 3M. To ensure a fair comparison

among candidates, we specifically consider networks with similar parameter counts within the 300k range. We compute metrics across four different parameter ranges and visualize them in Figure 4.

In the context of NAS, it's important to consider that other aspects beyond node-path balance can contribute to final classification accuracy. For instance, in the case of NAS-Bench-Macro, networks with similar node-path balance but varying classification accuracies can be affected by other architectural configurations, such as differences in pooling layers, kernel sizes, and expansion ratios in the MobileNet-v2 block. These architectural variances often result in different numbers of parameters, influencing the overall network performance. However, in the sparse neural network context, pruning methods focus on maintaining the same network structure while pruning connections within the network based on specific sparsities. Consequently, our experiments did not explore other architectural elements beyond node and path balance. In general,

8.6

8.8

9.0

20

25

30

35

40

#Effective paths (log scale)

#parameters 650k-950k

65

70

75

80

85

90

Test Accuracy

8.8

8.9

9.0

9.1

20

25

30

35

40

45

#parameters 950k-1250k

70

75

80

85

90

Test Accuracy

8.8

8.9

9.0

9.1

9.2

9.3

#Effective nodes (ln scale)

25

30

35

40

45

50

#Effective paths (log scale)

#parameters 1250k-1550k

80

82

84

86

88

90

92

Test Accuracy

9.3

9.4

9.5

#Effective nodes (ln scale)

35

40

45

50

55

#parameters 2150k-2450k

82

84

86

88

90

92

Test Accuracy

we can see that networks with larger numbers of nodes and paths tend to exhibit higher performance. Our findings unveil a strong correlation between the two metrics (node, path) and the final performance of network candidates, both playing pivotal roles in achieving superior performance. This highlights the critical significance of both nodes and paths in the initial stages of designing subnetworks that yield exceptional results.

4 Node-Path Balancing Principle

4.1 Node-Path Balancing Principle

From observations in Section 3.3 and NAS observations in Section 3.4, both effective paths and nodes have shown their critical roles in the performance of subnetworks. We now formally state the Node-Path Balancing Principle (NPB): The combination of both the numbers of effective nodes and effective paths is a proxy for the potential performance of subnetworks under pruning. A pruning at initialization method which produces pruned subnetworks with too many effective paths (resp. effective nodes) will have less than necessary the number of effective nodes (resp. effective paths) and consequentially has suboptimal performance. It is necessary to balance the number of effective nodes and effective paths for better performance of pruned subnetworks.

4.2 Proposed Method: NPB

Building upon the aforementioned principle, we elegantly transform the pruning problem into a multi-objective optimization problem. Specifically, our objective is to maximize both the number of effective nodes and effective paths concurrently given an architecture and desired sparsity ratio. We formulate this intriguing problem as follows:

Given an architecture A with parameter W ∈ R N where N is the total number of parameters and sparsity ratio s . Denote f p as the total number of input-output paths, f n as the number of activated nodes, and consider the mask for parameter M = { 0 , 1 } N as variable to solve.

where α is a coefficient. Solving node-path balancing objective globally over the whole neural networks is a non-trivial problem. We sidestep this challenging issue by solving a sequence of easy problems to obtain good approximated solutions. In particular, we propose an approximation for solving this problem by doing layer by layer through convex optimization. The approximation problem is solved efficiently via the available convex optimization library.

To be simple, we consider a linear layer as an example, in which, v denotes the node, l is the layer index, m ( l ) ∈ R h ( l ) × h ( l + 1) is the mask for layer l in which h ( l ) the number of neurons in layer l . We denote P ( v ( l ) i ) as the number of path go to node vi in layer l (e.g., at input layer P ( v (0) i ) = 1). We formulate the pruning problem in layer l as an optimization problem in which we maximize the number of paths from input to layer l + 1 f ( l + 1) p and the total activated nodes f ( l + 1) n in two layers l and l + 1. For brevity, we denote them as f p and f n . We set m ( l ) as:

The number of paths to v ( l + 1) j in layer l + 1 is sum of all paths to nodes in layer l connecting to v ( l + 1) j

Then the total number of effective paths to layer l + 1 is:

A node vi in layer l is activated if and only if there are paths pass through it and edges connect from it to nodes in layer l + 1, which is formulated as below:

And a node vj in layer l + 1 is effective when there exist attached nodes in layer l that connect to it,

The layer-wise objective for the node becomes:

In convolution layers with a kernel of height h and width w , we let the variable mij have the value from 0 to hw representing this kernel. After solving m , in each kernel, we assign | mij | parameters to entries whose initialized weights have the highest magnitude.

Along with two objectives, we also consider a regularization term which aims to encourage activating as many kernels as possible in each layer. Besides, since we optimize the node and path per layer, the solution will not be the global one. We can view the regularizer as an adjustment term, which moves the local solution around to figure out the better ones.

From Equations 1, 2, and 3 the final objective becomes:

where α is a coefficient to control the balance, β is a hyperparameter, and the constraint is the number of unpruned parameters that satisfies the sparsity. To ensure the same scale between elements in the main objective function, we normalize the number of paths, nodes, and the regularizer in each layer with their maximum possible values, respectively. Note that, optimizing nodes is much easier than paths so we select small values of alpha which can be prior knowledge. In particular, in Section 5, we fix α = 0 . 01 and β = 1 to all settings. We optimize 1 the network structure sequentially layer by layer from input to output in which the layer-wise sparsity level is found by ERK method [14, 29]. We describe our method in Algorithm 1 and the pseudo code for optimizer in Appendix C.

Algorithm 1 Node-Path Balancing Pruner

1: Inputs: Final sparsity s , weights w 0 = 1 , balancing coefficient α , and hyperparameter β

2: Obtain layer-wise sparsity s ( l ) by using ERK method

3: Define unit input x = 1 ⇒ P (0) = 1

4: for l = 0 , . . . , L do

5: m ( l ) ← optimize ( m ( l ) , s ( l ) , P ( l ) ) # solve Eq.4

6: Set w ( l ) 0 = m ( l )

7: Extract layer l + 1 output when put x to the network with w 0 to obtain P ( l + 1)

8: end for

9: Return: Mask M

5 Evaluation

5.1 Experimental Settings

We conduct experiments on three standard datasets: CIFAR-10, CIFAR-100, and Tiny-Imagenet. Following [42], we use ResNet-20 for CIFAR-10, VGG-19 for CIFAR-100, and a variant ResNet-18 with 18 layers for Tiny-Imagenet. We treat weights of all convolutional and linear layers as prunable, but we do not prune biases and weights in batch normalization layers. We run five seeds with experiments on CIFAR-10, CIFAR-100, and three seeds on Tiny-Imagenet. The average results are used for visualizations. Since the number of active paths is orders of magnitude larger than that of active nodes, and both numbers are of exponential scales, we take logarithm base 10 of the number of active paths, and logarithm base e (natural logarithm ) of the number of active nodes to visualize the results. To ensure the fair comparison between NPB and other baselines, we fix α = 0 . 01 and β = 1 in NPB for all settings. More details on our experimental setting are in Appendix A.

ResNet20 on CIFAR-10

6.20

6.25

6.30

6.35

6.40

6.45

6.50

6.55

6.60

Number of effective nodes (ln scale)

38

39

40

41

42

43

44

45

46

Number of effective paths (log scale)

88.03

88.64 87.88

88.17

90.38

90.69

Sparsity 68.38%

5.8

5.9

6.0

6.1

6.2

6.3

6.4

6.5

6.6

Number of effective nodes (ln scale)

28

30

32

34

36

38

40

42

84.65

84.02

84.94

84.22

87.41

87.61

Sparsity 90.00%

5.2

5.4

5.6

5.8

6.0

6.2

6.4

6.6

Number of effective nodes (ln scale)

18

20

22

24

26

28

30

32

34

36

67.56

76.72

78.22

77.05

81.05

80.55

Sparsity 96.84%

4.8

5.0

5.2

5.4

5.6

5.8

6.0

6.2

6.4

6.6

Number of effective nodes (ln scale)

10

15

20

25

30

35.99

62.43

66.05

64.95

70.44

70.7

Sparsity 99.00%

VGG19 on CIFAR-100

8.40

8.45

8.50

8.55

8.60

8.65

8.70

Number of effective nodes (ln scale)

46

47

48

49

50

51

Number of effective paths (log scale)

70.92

71.31

71.63

72.84

73.18

74.05

Sparsity 68.38%

7.9

8.0

8.1

8.2

8.3

8.4

8.5

8.6

8.7

Number of effective nodes (ln scale)

38

40

42

44

46

48

68.14

70.65

69.18

70.86

70.7

71.76

Sparsity 90.00%

7.4

7.6

7.8

8.0

8.2

8.4

8.6

8.8

Number of effective nodes (ln scale)

28

30

32

34

36

38

40

42

44

61.55

67.82

66.98

67.72

68.18

68.87

Sparsity 96.84%

7.00

7.25

7.50

7.75

8.00

8.25

8.50

8.75

Number of effective nodes (ln scale)

20.0

22.5

25.0

27.5

30.0

32.5

35.0

37.5

40.0

55.74

61.77

62.11

63.13

64.43

64.82

Sparsity 99.00%

ResNet18 on Tiny-Imagenet

8.0

8.1

8.2

8.3

8.4

Number of effective nodes (ln scale)

48

49

50

51

52

53

54

55

56

Number of effective paths (log scale)

56.78

56.99

56.71

56.73

58.09

58.39

Sparsity 68.38%

Random

SNIP

SynFlow

Iter-SNIP

PHEW

NPB

7.6

7.8

8.0

8.2

8.4

Number of effective nodes (ln scale)

38

40

42

44

46

48

50

52

54

55.32

53.43

54.68

53.6

55.93

56.82

Sparsity 90.00%

7.0

7.2

7.4

7.6

7.8

8.0

8.2

8.4

Number of effective nodes (ln scale)

30.0

32.5

35.0

37.5

40.0

42.5

45.0

47.5

50.0

42.77

48.77

49.03

48.55

50.81

51.37

Sparsity 96.84%

6.50

6.75

7.00

7.25

7.50

7.75

8.00

8.25

8.50

Number of effective nodes (ln scale)

20

25

30

35

40

45

33.02

36.02

39.79

36.42

40.54

41.05

Sparsity 99.00%

5.2 Comparison with PaI Methods

Figure 5 shows the experimental results for different PaI methods: Random, SNIP, Iter-SNIP, SynFlow, PHEW, and our NPB, with different sparsities on three settings: VGG-19 on CIFAR-100, ResNet20 on CIFAR-10, and ResNet-18 on Tiny-Imagenet. Overall, our method NPB shows significant improvements on standard PaI methods, both data-dependent like Iter-SNIP, SNIP (improve up to

14%) and data-independent like SynFlow (increase up to 7%) in all three settings. Notably, NPB achieves better results than state-of-the-art PaI method PHEW in almost all settings ( 11 out of 12 settings). Importantly, we set the same alpha and beta hyperparameters for NPB across all three settings, thereby eliminating the need for extensive hyperparameter search . This demonstrates the robustness and generalizability of our method, making it easily applicable to various settings with consistently high performance. Ablation studies on alpha and beta can be found in Appendix G.

We discuss the experimental results from the point of view of the Node-Path Balancing principle and network shape optimization. In Figure 5, SynFlow and Iter-SNIP produce sparse networks with large number of input-output paths, which results from iterative pruning procedures. These methods gradually remove nodes of low connection degree, which makes the subnetworks become narrower while maintaining the high number of paths. As a consequence of a very high number of input-output paths, the width of resulting subnetworks is significantly reduced, limiting the representation capacity of the subnetworks [35, 36], leading to suboptimality.

Random pruning is a PaI method which produces subnetworks with the very high number of active nodes. On average, it distributes parameters uniformly to all layers and kernels, creating subnetworks of large width yet a low number of active paths. At sparsities below 90%, with an adequate number of input-output paths, subnetworks of Random pruning still perform well and are even competitive with more sophisticated PaI methods. This phenomenon is exhibited clearly in experiments on the more complex dataset Tiny-Imagenet. However, when sparsity increases over 90%, Random pruning creates a huge number of ineffective paths and parameters. Consequently, the performance of subnetworks drops significantly. We again observe the necessity to balance the optimization of the number of nodes and paths.

In Figure 5, our method NPB and PHEW produce sparse networks which, in visualization, lie in specific areas which have higher paths than Random and greater nodes than SynFlow, or Iter-SNIP. We call these are balancing regions . Subnetworks in these regions have broader widths and considerably many paths, which provides good representation capacity and preserves the information flow of the network. These two factors together contribute to better training of pruned subnetworks leading to superior performance. This illustrates the effectiveness of our principle.

One drawback of one-shot pruning methods like SNIP is creating a large number of ineffective parameters, which decreases the capacity and performance of subnetworks. Although our method is also one-shot, it efficiently distributes the parameters by direct optimization. Consequently, NPB enjoys a better node-path balancing level and better performance (up to 14%) compared to SNIP.

Results of all settings in Figure 5 provide evidence to support our Node-Path Balancing principle and the existence of a specific balancing region between nodes and paths at given sparsity levels. Particularly, we posit that when the number of active neurons and input-output paths of a sparse network fall within this balancing range, it will probably have a good performance after training.

5.3 Pruning Time and FLOPs Reduction

Since NPB optimizes layer by layer, the complexity of NPB depends on network architecture (i.e., the number of layers and the size of each layer). With large layers, we sidestep the time-consuming issues by making a further step where we divide the layer into chunks with the same sparsity constraint. In particular, we split nodes in the next layer into equal parts while the input node is fixed. Then, optimizing chunks in a layer can be solved in parallel. Thank to the available convex optimization libraries we can find subnetworks more efficiently and quickly. We have computed the pruning time of our proposed method and compared it with other PaI methods in Table 1. Our pruning time is not

significantly slow compared to those iterative approaches (e.g., Iter-SNIP, SynFlow) while it is much faster (up to 14 . 5 × ) than PHEW in lower sparsity levels.

Besides pruning time, we find that the FLOPs reduction of subnetwork after pruning is more important in the context of pruning before training. We have measured FLOPs of subnetworks produced by different methods in Table 1. The result indicates that our NPB can produce subnetworks with lower FLOPs than other baselines while NPB outperforms PaI methods. More details are in Appendix E.

6 Conclusion

In this paper, we present a novel framework for studying PaI methods that systematically employs the configuration of pruned subnetworks based on two different metrics: the number of effective paths and the number of effective nodes. Through our framework, we discover a new relationship between these metrics, called the Node-Path Balancing Principle, which provides guidance for understanding and optimizing PaI methods. Our framework offers unified explanations for the intriguing layerwise connection reshuffling phenomenon [40, 17] of subnetworks produced by PaI methods in normal pruning sparsity regime, as well as the failure of this phenomenon in extreme sparsity levels. Furthermore, we propose a novel pruning method based on optimization approach, namely NPB, that demonstrates the potential of balancing the numbers of effective paths and nodes to improve the performance of PaI methods. Extensive experiments conducted on various model architectures and datasets show that NPB outperforms the best baseline, PHEW, on almost settings while ours produces more efficient subnetworks with much fewer inference FLOPs and faster in pruning time then PHEW. Our new perspective on the configuration of subnetworks, in terms of effective nodes and effective paths, provides new insights into the working mechanism of PaI methods and opens new research directions on neural network pruning methods, as well as designs of sparse neural network.

References

M. Alizadeh, S. A. Tailor, L. M. Zintgraf, J. van Amersfoort, S. Farquhar, N. D. Lane, and Y. Gal. Prospect pruning: Finding trainable weights at initialization using meta-gradients. In International Conference on Learning Representations , 2022. URL https://openreview. net/forum?id=AIgn9uwfcD1 .

P. Bithika, B. Arindam, K. Sudeshna, M. Pabitra, and B. Biswajit. A study on the ramanujan graph property of winning lottery tickets. In International Conference on Machine Learning , volume 162, pages 17186-17201, 2022.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

T. Chen, J. Frankle, S. Chang, S. Liu, Y. Zhang, Z. Wang, and M. Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems , 33:15834-15846, 2020.

T. Chen, J. Frankle, S. Chang, S. Liu, Y. Zhang, M. Carbin, and Z. Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16306-16316, 2021.

T. Chen, B. Ji, T. DING, B. Fang, G. Wang, Z. Zhu, L. Liang, Y. Shi, S. Yi, and X. Tu. Only train once: A one-shot neural network training and pruning framework. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id=p5rMPjrcCZq .

T. Chen, L. Liang, T. DING, Z. Zhu, and I. Zharkov. OTOv2: Automatic, generic, userfriendly. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=7ynoX1ojPMt .

Y. Cheng, D. Wang, P. Zhou, and T. Zhang. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Processing Magazine , 35(1):126-136, 2018.

P. de Jorge, A. Sanyal, H. Behl, P. Torr, G. Rogez, and P. K. Dokania. Progressive skeletonization: Trimming more fat from a network at initialization. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=9GsFOUyUPi .

L. Deng, G. Li, S. Han, L. Shi, and Y. Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE , 108(4):485-532, 2020.

S. Diamond and S. Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research , 17(83):1-5, 2016.

X. Dong, S. Chen, and S. Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in Neural Information Processing Systems , 30, 2017.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=YicbFdNTTy .

U. Evci, T. Gale, J. Menick, P. S. Castro, and E. Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning , pages 2943-2952. PMLR, 2020.

J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 , 2018.

J. Frankle, G. K. Dziugaite, D. Roy, and M. Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning , pages 3259-3269. PMLR, 2020.

J. Frankle, G. K. Dziugaite, D. Roy, and M. Carbin. Pruning neural networks at initialization: Why are we missing the mark? In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=Ig-VyQc-MLK .

T. Gale, E. Elsen, and S. Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574 , 2019.

T. Gebhart, U. Saxena, and P. Schrater. A unified paths perspective for pruning at initialization. ArXiv , abs/2101.10552, 2021.

S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems , 28, 2015.

B. Hassibi, D. G. Stork, and G. J. Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks , pages 293-299. IEEE, 1993.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016.

G. Hinton, O. Vinyals, J. Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2(7), 2015.

T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden, and A. Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res. , 22(1), jan 2021.

Y. LeCun, J. Denker, and S. Solla. Optimal brain damage. Advances in neural information processing systems , 2, 1989.

N. Lee, T. Ajanthan, and P. Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations , 2019. URL https: //openreview.net/forum?id=B1VZqjAcYX .

S. Liu, T. Chen, X. Chen, Z. Atashgahi, L. Yin, H. Kou, L. Shen, M. Pechenizkiy, Z. Wang, and D. C. Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems , 34:9908-9922, 2021.

S. Liu, L. Yin, D. C. Mocanu, and M. Pechenizkiy. Do we actually need dense overparameterization? in-time over-parameterization in sparse training. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 6989-7000. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/liu21y.html .

S. Liu, T. Chen, X. Chen, L. Shen, D. C. Mocanu, Z. Wang, and M. Pechenizkiy. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. In International Conference on Learning Representations , 2022. URL https: //openreview.net/forum?id=VBZJ_3tz-t .

D. C. Mocanu, E. Mocanu, P. Stone, P. H. Nguyen, M. Gibescu, and A. Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications , 9(1):1-12, 2018.

P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440 , 2016.

P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=SJGCiw5gl .

M. C. Mozer and P. Smolensky. Using relevance to reduce network size automatically. Connection Science , 1(1):3-16, 1989.

S. M. Naji, A. Abtahi, and F. Marvasti. Efficient sparse artificial neural networks. ArXiv , abs/2103.07674, 2021.

Q. Nguyen, M. C. Mukkamala, and M. Hein. Neural networks should be wide enough to learn disconnected decision regions. In International conference on machine learning , pages 3740-3749. PMLR, 2018.

S. M. Patil and C. Dovrolis. Phew: Constructing sparse networks that learn fast and generalize well without training data. In International Conference on Machine Learning , pages 8432-8442. PMLR, 2021.

I. Price and J. Tanner. Dense for the price of sparse: Improved performance of sparsely initialized networks via a subspace offset. In International Conference on Machine Learning , pages 8620-8629. PMLR, 2021.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning , pages 8821-8831. PMLR, 2021.

J. Su, Y . Chen, T. Cai, T. Wu, R. Gao, L. Wang, and J. D. Lee. Sanity-checking pruning methods: Random tickets can win the jackpot. Advances in Neural Information Processing Systems , 33: 20390-20401, 2020.

X. Su, T. Huang, Y. Li, S. You, F. Wang, C. Qian, C. Zhang, and C. Xu. Prioritized architecture sampling with monto-carlo tree search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10968-10977, 2021.

H. Tanaka, D. Kunin, D. L. Yamins, and S. Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems , 33:6377-6389, 2020.

R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085 , 2022.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

C. Wang, G. Zhang, and R. Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations , 2020. URL https: //openreview.net/forum?id=SkgsACVKPH .

H. Wang, C. Qin, Y. Bai, Y. Zhang, and Y. Fu. Recent advances on neural network pruning at initialization. In L. D. Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022 , pages 56385645. ijcai.org, 2022. doi: 10.24963/ijcai.2022/786. URL https://doi.org/10.24963/ ijcai.2022/786 .

G. Yuan, X. Ma, W. Niu, Z. Li, Z. Kong, N. Liu, Y. Gong, Z. Zhan, C. He, Q. Jin, et al. Mest: Accurate and fast memory-economic sparse training framework on the edge. Advances in Neural Information Processing Systems , 34:20838-20850, 2021.

R. Zhao, Y. Hu, J. Dotzel, C. De Sa, and Z. Zhang. Improving neural network quantization without retraining using outlier channel splitting. In International conference on machine learning , pages 7543-7552. PMLR, 2019.

M. Zhu and S. Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878 , 2017.

A Experiment details

In this section, along with pruning at initialization (PaI) methods in the main text, we provide experimental results with GraSP [45]. In particular, GraSP is another gradient-based pruning PaI that aims to preserve the gradient flow of sparse networks obtained by pruning. The score z of weight w in GraSP is computed as z = -w ⊙ ( H ∇ w L ), where H is the Hessian of the training loss after passing a mini-batch of data through the network.

We describe our experiment settings on architectures and datasets. We use Pytorch 2 library and conduct experiments on a single GTX 3090Ti or A100 (depend on their available). We adapt from Tanaka et al. [42] source code 3 for SNIP, GraSP, SynFlow, and Random, and change the prune epochs to 100 for Iterative SNIP instead of 1 in SNIP. And we use the official code 4 of Patil and Dovrolis [36] for PHEW. For NPB method in the main experiments, we set α = 0 . 01 and β = 1 for all settings.

Datasets. Our main experiments are conducted with CIFAR-10, CIFAR-100, and Tiny-Imagenet datasets, where:

CIFAR-10 is augmented by normalizing per-channel, randomly flipping horizontally.

CIFAR-100 is augmented by normalizing per-channel, randomly flipping horizontally.

Tiny-ImageNet is augmented by normalizing per channel, cropping to 64x64, and randomly flipping horizontally.

Architectures. We use three different networks:

VGG-19 is a CIFAR-100 network used in SynFlow [42]. We choose a batch-normalization version.

ResNet-20 is a 20-layer CIFAR-10 version of ResNet created by [22]. This version has added batch normalization layers before each activation function.

ResNet-18 is a ImageNet version with 18 layers adapted from SynFlow [42]. The first convolution has kernel size 3x3 (instead of 7x7) and max-pooling layer that follows has been removed.

We treat all of the weights from convolutional and linear layers of these networks are prunable parameters, but we do not prune the biases nor the weights in the batch normalization layers. The weights in convolutional and linear layers are initialized with Kaiming normal, while biases are initialized to be zero. We run five seeds with CIFAR experiments and three seeds with experiments on Tiny-Imagenet.

Training details With Iterative pruning methods SNIP and SynFlow, we use 100 pruning epochs. With methods using training data like SNIP, GraSP, and Iterative-SNIP, we randomly select 10 samples for each class, particularly, 100 data points for CIFAR-10, 1000 data points for CIFAR-100, 2000 data samples for Tiny-ImageNet. Other hyperparameters are chosen as follow:

B Effective metrics calculation

Effective path. To exactly compute the number of effective paths, we remove the batch normalization layers, we initialize all the remaining parameters to 1. Then, we put the input vector one to the network, and the number of effective paths is the sum of logits on the output layer R = 1 ⊤ ( Q L = 1 | w ℓ | ) 1 .

ℓ

More precisely, we face problems with pooling layers in convolutional neural networks. With max pooling layer, we simply do not modify the output of this layer. At that time, the result is the maximum number of paths in subnetworks. With average pooling layer, since all inputs of this layer contribute to the output, we change the average operator to the sum operator to exactly compute the number of effective paths. We all use ReLU activation functions in computing this metric since this function does not affect the results of calculations.

Effective parameter. We follow [17] when identifying which is effective parameter. Similar to computing effective paths, we make further steps. After having the sum of logits, we compute the gradients of this sum with respect to weights ∇ w R . Then, if an unpruned weight has a non-zero gradient, it is effective and vice versa. Effective parameters are dense edges that connect two effective nodes as visualized in Figure 2.

Effective node/channel. With fully connected layers, if all connections to one node or out of one node are pruned, this node is pruned node. If there exist connections to a node but all of these connections are ineffective, then this node becomes ineffective In convolutional layers, instead of nodes, we have channels. We consider a kernel as a connection, a channel as a node, and then convert the convolutional layer into a fully connected layer. The connection is pruned if and only if all parameters in the corresponding kernel are removed. Finally, identifying the effective nodes/channels is similar to the way in fully connected layers.

C Layer-wise Mask Optimization

We provide the pseudo code solving the mask optimization problem based on the cvxpy library [11]. We use the default solver provided in this tool.

D Layerwise Shuffling Experiments

With each setting, at each sparsity ratio, we seek subnetworks with 5 different seeds, and with each seed, we randomly shuffle the subnetwork two times. In addition to effective path ratios and effective node ratios, we compute the number of effective parameters after pruning and the actual remaining ones, then calculate the ratio between these two values.

Similar to [17] results, the performance and the number of effective parameters of high-density subnetworks after permuting the connections are similar to or even higher (in SNIP) than the unmodified ones. However, when the sparsity level becomes more intensive, the configuration of subnetworks is more concrete. Randomly rearranging connections within layers destroys this strict structure by detaching important edges, which drastically reduces the number of effective paths. The shuffled subnetworks lack input-output paths to transfer information during training, leading to a drop in performance compared with unmodified ones.

SNIP

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

0.2

0.4

0.6

0.8

Effective path ratio

Unmodified

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

0.2

0.4

0.6

0.8

1.0

Effective node ratio

Unmodified

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

0.4

0.6

0.8

1.0

Effective parameter ratio

Unmodified

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

20

40

60

80

Top-1 Accuracy (%)

Unmodified

SynFlow

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

0.2

0.4

0.6

0.8

Effective path ratio

Unmodified

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

0.0

0.2

0.4

0.6

0.8

1.0

Effective node ratio

Unmodified

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

0.2

0.4

0.6

0.8

1.0

Effective parameter ratio

Unmodified

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

20

40

60

80

Top-1 Accuracy (%)

Unmodified

PHEW

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

Sparsity (%)

0.2

0.4

0.6

0.8

Effective path ratio

Unmodified

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

Sparsity (%)

0.0

0.2

0.4

0.6

0.8

1.0

Effective node ratio

Unmodified

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

Sparsity (%)

0.0025

0.0050

0.0075

0.0100

0.0125

Effective parameter ratio

Unmodified

68.38

90.00

96.84

99.00

99.44

99.68

99.82

99.90

Sparsity (%)

20

40

60

80

Top-1 Accuracy (%)

Unmodified

E Pruning Time and FLOPs Reductions

Since NPB optimizes layer by layer, the complexity of NPB depends on network architecture (i.e., the number of layers and the size of each layer). With large layers, we sidestep the time-consuming issues by making a further step where we divide the nodes in the next layer into chunks with the same sparsity constraint, and optimizing chunks in a layer can be solved in parallel. Particularly, we consider layer l with mask m ( l ) ∈ R h ( l ) × h ( l + 1) in which h ( l ) and h ( l + 1) are the number of nodes in layer l and l + 1. We divide h ( l + 1) nodes into K equal chunks. Instead of directly solving for m ( l ) , we solve K problems [ m ( l ) 1 , m ( l ) 2 , ..., m ( l ) K ] where m ( l ) k ∈ R h ( l ) × h ( l + 1) k . Thank for available convex optimization libraries we can find subnetworks more efficiently and quickly. We have computed the pruning time of our proposed method and compared it with other PaI methods in Tables 3, 4, and 5.

Besides pruning time, we find that the FLOPs reduction of subnetwork after pruning is more important in the context of pruning before training. We have measured FLOPs of subnetworks produced by different methods in Tables 3, 4, and 5. The result indicates that our NPB can produce subnetworks with lower FLOPs than other baselines while NPB outperforms PaI methods.

F Additional Results with PaI comparison

43.77

68.38

90.00

96.84

99.00

Sparsity

60

65

70

75

80

85

90

Accuracy (%)

ResNet20 on CIFAR-10

PHEW

Random

SNIP

Iter-SNIP

SynFlow

Magnitude

GraSP

NPB

LTH

Dense

43.77

68.38

90.00

96.84

99.00

Sparsity

35

40

45

50

55

60

Accuracy (%)

ResNet18 on TinyImageNet

PHEW

Random

SNIP

Iter-SNIP

SynFlow

GraSP

NPB

LTH

G Additional Results with Ablation Studies

In this section, we conduct experiments with different balancing coefficients (alpha) and regularization coefficients (beta) in setting ResNet18 on Tiny-Imagenet and visualize results in Figure 8. We run NPB with alpha in { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 0 . 75 , 0 . 9 } and beta in { 0 . 1 , 0 . 5 , 0 . 75 , 1 . 0 , 1 . 5 , 2 . 0 } . In the first row, we visualize the performance of subnetworks generated by different alpha beta hyperparameters. In addition, to illustrate the correlation between subnetworks of NPB and PaI methods in terms of the number effective nodes and paths, we visualize alpha and beta subnetwork variants of NPB and PaI methods in the second row of Figure 8. Although NPB's variants have difference performance, they still outperform baselines. When visualize NPB variants with different PaI methods (SynFlow, Random, and PHEW) together, we can see that NPB variants concentrate around a specific region near PHEW which we hypothesize as a balancing area. Besides, networks in this region show a better results compared with SynFlow or Random, and have competitive performance with each others. This strengthens our Node-Path Balancing Principle.

0.01

0.05

0.1

0.5

0.75

0.9

Alpha

0.1

0.5

0.75

1.0

1.5

2.0

Beta

58.29

57.66

58.08

58.35

58.26

58.38

58.86

57.99

57.84

58.25

58.43

58.54

58.31

58.14

58.29

57.98

58.61

58.34

58.69

58.54

58.89

58.25

58.69

58.70

57.92

58.29

58.84

58.02

58.04

57.95

58.83

58.42

58.04

58.03

58.13

58.21

Sparsity 68.38%

57.8

58.0

58.2

58.4

58.6

58.8

0.01

0.05

0.1

0.5

0.75

0.9

Alpha

0.1

0.5

0.75

1.0

1.5

2.0

Beta

56.22

55.34

55.69

56.03

56.61

57.04

56.91

56.71

57.07

57.39

56.59

56.84

57.64

56.87

57.29

56.14

57.21

57.07

56.66

56.95

57.44

56.78

57.42

56.52

56.75

57.19

57.09

56.46

57.24

57.15

57.06

56.76

56.76

56.68

57.24

57.43

Sparsity 90.00%

55.5

56.0

56.5

57.0

57.5

0.01

0.05

0.1

0.5

0.75

0.9

Alpha

0.1

0.5

0.75

1.0

1.5

2.0

Beta

50.44

50.16

49.09

50.84

51.92

51.55

50.60

50.52

50.55

52.34

52.17

52.07

51.45

51.26

51.19

51.32

52.36

52.14

51.12

50.94

51.74

51.76

51.87

52.48

51.24

51.93

51.47

51.50

52.17

52.02

51.08

51.50

51.62

51.77

52.08

51.81

Sparsity 96.84%

49.5

50.0

50.5

51.0

51.5

52.0

8.22

8.24

8.26

8.28

8.30

8.32

Number of effective nodes (ln scale)

50

51

52

53

54

55

Number of effective paths (log scale)

Sparsity 68.38%

SynFlow

PHEW

Random

NPB's variants

7.9

8.0

8.1

8.2

8.3

Number of effective nodes (ln scale)

40

42

44

46

48

50

Sparsity 90.00%

SynFlow

PHEW

Random

NPB's variants

7.4

7.6

7.8

8.0

8.2

Number of effective nodes (ln scale)

32

34

36

38

40

42

44

46

Sparsity 96.84%

SynFlow

PHEW

Random

NPB's variants

NPB depends on the layer-wise sparsity ratios, disparate ratios guide the pruner to a diverse set of solutions. Then, it is essential to investigate the effect of sparsity initializers on NBP's performance. We follow Liu et al. [29] to use ERK [14], Uniform, Magnitude ratio, SNIP ratio found by applying SNIP [26], and GraSP ratio by using GraSP [45] as five methods to produces layer-wise sparsity levels. Figure 9 illustrates results with these sparsity initializers for ResNet18 on Tiny-Imagenet with different sparsity levels. With the same value of α and β , we observe that different layer-wise sparsifiers produce different network structures and performances as well. This difference raises a question for further work how we can specify ratios that help obtaining promising solutions.

8.0

8.2

8.4

8.6

Number of effective nodes (ln scale)

50

51

52

53

54

Number of effective paths (log scale)

58.69

58.01

58.25

57.25

57.4

Sparsity 68.38%

ERK

GraSP ratio

SNIP ratio

Uniform

Magnitude ratio

8.125 8.150 8.175 8.200 8.225 8.250 8.275 8.300 8.325

Number of effective nodes (ln scale)

42

43

44

45

46

47

56.66

56.87

57.07

56.6

54.53

Sparsity 90.00%

ERK

GraSP ratio

SNIP ratio

Uniform

Magnitude ratio

7.8

7.9

8.0

8.1

8.2

8.3

Number of effective nodes (ln scale)

35

36

37

38

39

40

41

42

43

51.12

50.31

50.09

49.26

Sparsity 96.84%

ERK

GraSP ratio

SNIP ratio

Uniform

H NAS-Bench-Macro Experiments

NAS-Bench-Macro [41] is a macro benchmark which is used in Neural Architecture Search. It includes 8 searching layers, each having three candidate blocks which construct 6 , 561 different networks with the parameter varying from 300k to 3M. The network structure is presented in Table 6. The structure is inspired from MobileNetV2 search space. In particular, candidate blocks are (i) Identity connection, (ii) MB3_K3 which is MobileNetV2 block with kernel size 3 and expansion ratio 3, and (iii) MB6_K5 which is MobileNetV2 block with kernel size 5 and expansion ratio 6. To make a fair comparison between candidates, we only compare networks with a similar number of parameters (i.e., in a range of 300k parameters). Note that, each candidate here is a dense network. We compute metrics in four different parameter ranges and visualize them in Figure 10.

8.0

8.1

8.2

8.3

8.4

8.5

8.6

15.0

17.5

20.0

22.5

25.0

27.5

30.0

32.5

35.0

The number of parameters from 350k-650k

50

60

70

80

90

Test Accuracy

8.5

8.6

8.7

8.8

8.9

9.0

20

25

30

35

40

The number of parameters from 650k-950k

65

70

75

80

85

90

Test Accuracy

8.8

8.9

9.0

9.1

20

25

30

35

40

45

The number of effective paths (log scale)

The number of parameters from 950k-1250k

70

75

80

85

90

Test Accuracy

8.8

8.9

9.0

9.1

9.2

9.3

25

30

35

40

45

50

The number of parameters from 1250k-1550k

80

82

84

86

88

90

92

Test Accuracy

9.00

9.05

9.10

9.15

9.20

9.25

9.30

9.35

9.40

30

35

40

45

50

55

The number of parameters from 1550k-1850k

70

75

80

85

90

Test Accuracy

9.20

9.25

9.30

9.35

9.40

9.45

9.50

The number of effective nodes (ln scale)

30

35

40

45

50

55

The number of effective paths (log scale)

The number of parameters from 1850k-2150k

70

75

80

85

90

Test Accuracy

9.25

9.30

9.35

9.40

9.45

9.50

9.55

35

40

45

50

55

The number of parameters from 2150k-2450k

82

84

86

88

90

92

Test Accuracy

9.45

9.50

9.55

9.60

9.65

37.5

40.0

42.5

45.0

47.5

50.0

52.5

55.0

57.5

The number of parameters from 2450k-2750k

85

86

87

88

89

90

91

92

93

Test Accuracy

9.60

9.62

9.64

9.66

9.68

9.70

9.72

44

46

48

50

52

54

56

58

The number of parameters from 2750k-3050k

87

88

89

90

91

92

93

Test Accuracy

The number of effective nodes (ln scale)

The number of effective nodes (ln scale)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors perform a large-scale empirical analysis of models from the NAS-Bench-Macro benchmark which motivates to the development of (1) the node-path balancing principle and (2) Node-Path Balancing Pruner (NPB) -- a data-agnostic pruning-at-initialization (PAI) scheme. At a high level, the node-path balancing principle suggests that networks that have a good balance between the number of nodes and edges on a continuous/unbroken path from the neural network input to the neural network have a higher capacity for performance (i.e. classification accuracy). Their pruning algorithm, NPB, then solves a constrained optimization problem to strike this node-path balance for a given sparsity level. The driving factor for this research is to design advanced PAI algorithms for producing efficient models with better performance (i.e., classification accuracy) than existing PAI schemes.\n\n**Main contributions:**\n\n* Node-path balancing principle: Empirical analysis of NAS-Bench-Macro benchmark yields new perspective and principle for guiding design of pruning-at-initialization (PAI) schemes.\n* NPB: A novel data-agnostic PAI scheme designed using the node-path balancing principle which outperforms existing PAI methods at lower FLOPs during inference.\n\n**Post-rebuttal revision:**\n\nFollowing the rebuttal, I feel that the authors have clarified my questions and concerns. Accordingly, I am increasing my rating to weak accept with the expectation that they will accommodate my requested minor revisions.\n\nSTRENGTHS:\n* Large scale analysis of NAS-Bench-Macro benchmark models suggesting connection between model performance and balance of effective nodes and paths.\n* **NPB pruning-at-initialization algorithm**: Produces results comparable to state-of-the-art PAI scheme PHEW with fewer FLOPs during inference.\n\nWEAKNESSES:\n* **Looseness/inexactness of the node-path balancing principle**: I found the statement/rigor of this guiding principle (which was used to inform the design of NPB) to be very inexact. The motivating plots in Figure 4 suggest that there is nuance between this node-path balance and the resulting classification accuracy, but this was not further explored. Specifically, some networks with similar node-path balance in Figure 4 appear to have large variations in classification accuracy.\n* **Lack of theory for node-path balancing principle**: Developing theory to study the principle would help design/inform a more exact guiding principle for PAI schemes.\n* **Figure clarity**: It was not always evident what the takeaway is from certain figures and I feel that the presentation of certain figures could be improved. I found myself staring at some figures for a while trying to decipher what was important/meaningful as there is a lot of data. This isn’t to say there is not meaningful information in the figures but I feel like more careful curation/design of the figures could be enacted. For suggestions/specific figures, see comments under “Questions”.\n\nQUESTIONS:\n1. How did you compute/measure the FLOPs presented in Table 1? I may have missed something, but I did not see an explanation in the main body or in the appendix. \n2. Did you observe training/inference speedup due to reduced FLOPs? If so, it would be worth mentioning as NPB tends to require fewer FLOPs than other methods (at least at sparsity >= 90%).\n3. In some instances, models appear to have very different balance of nodes and paths but comparable accuracy (e.g., ResNet20 on CIFAR-10 sparsity 99%, VGG19 on CIFAR-100 sparsity 90%) or similar balance of nodes and paths but very different accuracy (VGG19 on CIFAR-100 sparsity 68.38%, ResNet18 on Tiny-Imagenet sparsity 90%). How would your principle in its current form explain this?\n\nComments about figures:\n\n* **Figure 3**: I had to stare at this for a while to figure out what the takeaway was. It seems that more effective nodes can help improve accuracy, but it is not enough on its own. I’m not sure if there is a different way to present the data to make it more readily apparent (and I’m not counting it against you) but I just wanted to mention it.\n* **Figure 4**: I think the scale of the accuracy colormap should be the same across all 4 plots (i.e. darkest blue is min accuracy across all 4 plots ~65% and yellow is max across all 4 plots ~93%). Then accuracy diversity across the four groups would be more apparent (i.e. more params -> higher accuracy) and you could just have a single colorbar.\n* **Figure 5**: For each dataset (i.e. row) I think the x- and y-axis range should be the same across the four plots. This revision should better highlight trends/changes in node-path balance with increasing sparsity.\n\nLIMITATIONS:\nRepeating a weakness here as I believe it is a limitation that is not acknowledged/discussed:\n\n* **Looseness/inexactness of the node-path balancing principle**: I found the statement/rigor of this guiding principle (which was used to inform the design of NPB) to be very inexact. The motivating plots in Figure 4 suggest that there is nuance between this node-path balance and the resulting classification accuracy, but this was not further explored. Specifically, some networks with similar node-path balance in Figure 4 appear to have large variations in classification accuracy.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines Pruning at Initialization (PaI) methods using two novel metrics: the number of effective paths and the number of effective nodes. The authors find that layer reshuffling negatively impacts the performance of sparse neural networks obtained through PaI methods in the extreme sparsity regime. Based on the previous finding, authors present a novel data-agnostic PaI method, Node-Path Balancing Principle (NPB), which achieves the SOTA performance by effectively balancing the two proposed metrics.\n\nSTRENGTHS:\n- This paper provides a clear rationale for introducing the two proposed metrics; the authors empirically demonstrate the importance of the metrics via two experiments. First, experiments conducted on the NAS benchmark reveal a strong correlation between the two metrics and the final performance of sparse networks. Secondly, through layer shuffling experiments, the authors illustrate that simply increasing the number of effective nodes in the extreme sparsity regime is insufficient to prevent performance degradation due to the sharp decline in the number of effective paths.\n- The proposed method demonstrates superior performance compared to the baselines, and the paper covers a fair amount of relevant previous studies.\n- The paper is effectively structured and exhibits clear and concise writing\n\nWEAKNESSES:\n- While many previous PaI methods prioritize the weight magnitude as the importance metric, the main motivation behind NPB focuses on the topology of the sparse network. Thus, the ablation study with respect to different weight initialization would strengthen the paper. Further, is NPB robust to *weight reinitialization*?\n- While the layer shuffling experiments are well-justified, it is confined to a single configuration (CIFAR-10, ResNet20). I wonder if similar observations can be made under different experimental settings. Also, does layer-wise shuffled NPB exhibit similar behavior as to layer-wise shuffled PHEW?\n- There are two points in need of clarification. First, in lines 332-333, the term \"chunks\" is unclear, and it is not apparent how the parallel computation can be effectively achieved. Second, in the case of convolutional layers mentioned in lines 261-263, it is unclear whether equations 3-5 are still applicable when the mask vector is not boolean.\n- Although the authors claim that NPB generally outperforms PHEW, the standard deviation with respect to random seeds should be included in the paper for fair comparison. For instance, the performance gain depicted in Fig. 7 seems marginal. Additionally, in lines 694-695, it is stated that NPB consistently outperforms the baselines regardless of the hyperparameter configuration. However, it should be noted that certain choices may lead to worse performance compared to the baselines (refer to Table 1).\n\nI am willing to raise my score if above concerns are well addressed during rebuttal.\n\nQUESTIONS:\n- In equation 3, what is the purpose of “min” operation? Or is it just a typo of $\\min(m_{ij}^l - 1, 0)$?\n- What kind of optimizer is implemented in the code as to approximating the original integer programming formulation?\n\nLIMITATIONS:\nThe paper presents a future research direction in lines 706-707, which is included in the Appendix. However, it is equally important for the authors to acknowledge and discuss the limitations of their proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper posits that the performance of (neural network) pruning at Initialization methods depends on a balance between effective nodes and paths. With this framework, authors explain why randomly shuffled subnetworks are sometimes more effective than subnetworks found by pruning at initialization methods. Finally, using the Node-Path balancing principle, the authors propose a pruning at initialization scheme by solving a linear program that optimizes pruning mask to balance effective path and nods, which outperforms other pruning at initialization methods.  \n\n\nEmpirical experiments are done on image datasets (CIFAR and TinyImagenet) with VGG and Resnet-20 models. Further empirical evidence for the node-path balancing principle is demonstrated through NAS benchmarks.\n\nSTRENGTHS:\n- The paper is well-written, and generally content is well-organized. \n- The proposed node-path balancing principle is used to explain an existing phenomenon (random shuffling) as well as present a new pruning at initialization scheme, which is quite interesting. \n- The convex program-based pruning method (NPB) outperforms other pruning at initialization methods and requires fewer FLOPs. PHEW is close in performance to NPB. However, the node-path balancing framework can also explain that, is interesting. \n\nOverall, the paper does a good job of introducing the node-path balancing principle and provides several empirical evidence in its support.\n\nWEAKNESSES:\n- The NAS experiments are interesting, but it wasn't clear to me to connect them with pruning at initialization or data-agnostic pruning, as NAS experiments are data-dependent. \n- The term balance may be vaguely used. \n   - The proposed principle states that at a particular sparsity, the best-performing subnetwork has to strike a balance between the effective nodes and paths. However, it does not predict the right balance.\n    - It would be interesting to see if this balance depends on sparsity, model, and dataset or model and datasets only. Could the best-performing subnetwork network be maximizing average effective paths per node?\n- The experiments are limited to image datasets and two neural network architectures. However, I appreciate the authors reporting results over three random seeds though.\n\nQUESTIONS:\n- Please see the weaknesses section. \n- Shouldn't Eq 4/5 consider the mask M as binary? As such, M is a real value; how is the solution converted to a pruning mask?\n- Line 262: Should M vary from 0 to hw instead of kw for convolution masks?\n\nLIMITATIONS:\nNone, that are not discussed or addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGiven the numerous pruning-at-initialization (PaI) methods, the performance of them are still far from satisfactory compared to the post-training pruning methods. In this work, the authors provide a novel perspective to understand the relationship between the performance and the architecture of the subnetworks generated by PaI methods. In particular, this work studies the topology of the subnetwork and observed a positive correlation between the PaI performance and the balance between the effective nodes and effective paths. Further, the authors propose a new PaI method, named Node-Path Balancing Pruner (NPB) to explicitly balance effective nodes and paths. The empirical results demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. This paper studies the PaI methods from a novel perspective of model topology. This is a very interesting but also challenging perspective. \n2. The figures and graphs in this paper are very delicately plotted and are of high quality. They help the readers better understand the methods proposed. \n3. This work made enough literature review, which has covered the most important PaI literature.\n4. This work is written in a very coherent manner, which shows clear the motivation, the method, the rationale behind the method, as well as the logic of the empirical studies. \n5. The empirical results of the proposed method are indeed very impressive.\n\nWEAKNESSES:\nI listed several weaknesses below from different perspectives. I will consider to raise my score if they are properly addressed.\n* [Method] Based on my understanding of the NAS observation, the reason for balancing the effective nodes and the effective paths is to maximize the usage of the limited parameter quota. However, if this is true, wouldn't directly cutting down the width of the network a perfect option for NPB? Correct me if I missed anything.\n* [Method] Mathematically, the formulation of the optimization objective in Eq. (0) (between Line 243 and Line 244) and Eq. (4~5) are not standard. The to-be-optimized variables should be clearly stated beneath the maximization symbol.\n* [Method] It is not very clear to me, how the optimization is carried out through the objective Eq. (4~5). Are there back-propagations involved? It seems not.\n* [Experiments] For Fig. 5, the information conveyed is not as clear as I expected. Probably this is because the performance of each method in each setting is annotated with numbers. I would suggest the authors to plot a normal figure showing the performance change (y-axis) vs. the sparsity level (x-axis) as many pruning paper does (there are too many and thus I spare the references here). It can help the readers compare the final accuracy of different methods if they do not care the intermediate results too much (e.g. the effective paths/nodes).\n* [Experiments] In the experiments, the $\\alpha$ is set to $0.01$. However, the word \"balanced\" claimed in the abstract seems a bit deceptive. To me, the value of $\\alpha$ is very crucial in the proposed method, but the authors fail to demonstrate its role through some ablation studies. I would suggest the authors either explain their choice of $\\alpha$ and/or conduct some ablation studies on $\\alpha$.\n* [Experiments] In Fig. 5, will it be better to add a reference line such as $y = \\alpha x$ to help judge the balance of the nodes/paths? Correct me if I missed anything.\n* [Experiments] This is not a must but I think the baselines compared in this paper are a bit old. Some methods like ProsPr (already cited in the paper) are recommended to be compared. The lack of this result will not change my evaluation on this paper.\n* [Minor] Rephrase: \"An effective node/channel is one that at least one effective path goes through it\" ==> \"A node/channel is effective if at least one effective path goes through it\".\n\nQUESTIONS:\n1. It would help the readers to understand the details of the motivation studies if the authors could explain how the \"effective sparse ratio\" is calculated. Is it done by traversing? Or is there any effective algorithm to calculate that automatically?\n2. I am very surprised by the results of the motivation studies. Thus, I am curious if the same rules can be applied to post-training pruning?\n3. In the algorithm of NPB, the layer-wise sparsity is obtained through ERK. Is there a particular reason for this choice? Will other methods also be applicable?\n\nLIMITATIONS:\nI do not have additional comments on the limitation of this work. Please refer to the \"Weaknesses\" and \"Questions\" sections.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a Pruning at Initialization (PaI) method that considers the balance between the number of effective nodes and effective paths. This design principle is based on the observations on the NAS benchmark as well as layer-wise reshuffling. The pruning problem is nicely formulated as a multi-objective optimization problem. The experiment results show that the proposed method NPB outperforms the state-of-the-art method PHEW in some configurations.\n\nSTRENGTHS:\n* A new Pruning at Initialization (PaI) method is developed considering both the number of effective nodes and the number of effective paths.\n* A lot of analyses and experiments have been conducted to show the superiority and motivation of the proposed method. \n* The pruning problem is formally transformed into a multi-objective optimization problem.\n* In some settings, the proposed method NPB is shown to outperform the state-of-the-art method PHEW.\n\nWEAKNESSES:\nThe motivation and technical details are not very clear in several places, as follows.\n\n**Overall:**\n* The sweet spot of the proposed framework seems to be the extreme sparsity regime (> 99%).  I am really wondering if we need to prune a network to the extreme. My impression is that the sweet spot is a corner case.\n\n**Section 3.3:**\n* Line 187: Why does reshuffling make subnetworks wider?\n* Line 205: How is the hypothesis derived from reshuffling?  PaI actually does *not* involve reshuffling.  \n\n**Section 4.1:**\n* Figure 4 (and Figure 1): It seems that, in typical architectures, the number of effective nodes and the number of effective paths are strongly correlated.  Then, why do we need to balance these two metrics?\n* Line 223: Along the same lines, the strong correlation between the two metrics indicates that one of them is redundant. Thus, the overall claim cannot be supported by the observation.\n* Figure 4 (and Figure 1): What's the meaning of each value in the x- or y-axis?  For example, what do you mean by $9.0$ in the x-axis?\n\n**Section 4.2:**\n* Due to the strong correlation between the number of effective nodes and the number of effective paths, it may be unnatural to produce subnetworks with too many effective paths (or nodes) and too few effective nodes (or paths).\n\n**Section 4.3:**\n* Even though it is reasonable to balance these two metrics, their ranges significantly differ. Actually, they have completely different scales (natures). This *incompatibility* between the two metrics would explain a very small value of $\\alpha$=0.01. Syntactically, Equation (4) does not achieve the balance, but mainly considers only $f_p$. Also, $\\alpha=0.01$ is indeed arbitrary and does not provide any insight on the optimal balance. \n* Appendix G (Ablation Study): Figure 8 shows that the accuracy is not that sensitive to $\\alpha$. I am still very confused why the balance between the two metrics is indeed important. Figure 8 directly shows that the optimal balance may not be important.\n\n**Section 5.2:**\n* It is not clear how the sparsity levels were chosen for the datasets in Figure 5. That is, how are the 12 settings chosen? I believe that a wider range of sparsity levels should be analyzed.\n* In contrast to Figure 4 (and Figure 1), it is not clear why most of the circles are not placed on the diagonal. \n* Line 316: Many cases of NPB do not lie in the so-called balancing regions. It would be better to mark the balancing regions in the figures.\n\nOverall, I believe the proposed framework is interesting and has some potential. However, the main contribution is neither well motivated nor rigorously presented. It seems that my concerns may not be sufficiently addressed by the rebuttal process. Therefore, I would like to give my rating as a reject.\n\n--\n\n**After Rebuttal**\n\nSome of the concerns and questions are resolved by the authors' rebuttal. However, the observational motivation and the need for careful optimization are not convincingly presented. Thus, I am a little reluctant to give a high score and would like to increase my rating to 5.\n\nQUESTIONS:\nSee the weak points mentioned above.\n\nLIMITATIONS:\nThe authors did not discuss the limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-18"}