Structured Pruning of CNNs at Initialization
Authors: Yaohui Cai
Weizhe Hua
Hongzheng Chen
Fengyu Li
G. Edward Suh
Christopher De Sa
Zhiru Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Pruning
Pruning-at-Initialization
Structured Pruning
Efficient Deep Learning
Efficient Model

Abstract: Pruning-at-initialization (PAI) proposes to prune the individual weights of the CNN before training, thus avoiding expensive fine-tuning or retraining of the pruned model. While PAI shows promising results in reducing model size, the pruned model still requires unstructured sparse matrix computation, making it difficult to achieve wall-clock speedups. In this work, we show theoretically and empirically that the accuracy of CNN models pruned by PAI methods only depends on the fraction of remaining parameters in each layer (i.e., layer-wise density), regardless of the granularity of pruning. We formulate the PAI problem as a convex optimization of our newly proposed expectation-based proxy for model accuracy, which leads to finding the optimal layer-wise density of that specific model. Based on our formulation, we further propose a structured and hardware-friendly PAI method, named PreCrop, to prune or reconfigure CNNs in the channel dimension. Our empirical results show that PreCrop achieves a higher accuracy than existing PAI methods on several modern CNN architectures, including ResNet, MobileNetV2, and EfficientNet for both CIFAR-10 and ImageNet. PreCrop achieves an accuracy improvement of up to $2.7\%$ over the state-of-the-art PAI algorithm when pruning MobileNetV2 on ImageNet. PreCrop also improves the accuracy of EfficientNetB0 by $0.3\%$ on ImageNet with only $80\%$ of the parameters and the same FLOPs.


Under review as a conference paper at ICLR 2023
STRUCTURED PRUNING OF CNNS AT INITIALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Pruning-ar atr--a--inn at-inntialiizaioon (PAI) methods can prune the individual weights of convo- lutional neural network (CNN) before training, thus avoiding expensive fine-tuning or retraining of the pruned model. While PAI shows promising results reducing model size, the pruned model still requires unstructured sparse matrix computation, making it difficult to achieve a real speedup. In this work, we show both theoreti- cally and empirically that the accuracy of CNN models pruned by PAI method depends on the layer-wise density (i.e., the fraction of the remaining parameters in each layer), irrespective of the granularity of pruning. We formulate PAI as a convex optimization problem based on an expectation-basel proxy for model accuracy, which can produce the optimal allocation of the layer-wise densities with respect to the proxy model. Using our formulation, we further propose structured and hardware-friendly PAI method, named PreCrop, to prune or reconfigure CNNs in the channel dimension. Our empirical results show that PreCrop achieves higher accuracy than existing PAI methods on several popular CNN architectures, including ResNet, MobileNetV2, and EfficientNet, both CIFAR-I0 10 and Ima- geNet. Notably, PreCrop achieves an accuracy improvement of up to 2. 2.7% over a state-of-the-art PAI algorithm when pruning MobileNerV2 ImageNet. PreCrop also improves the accuracy of Efficien(NetBO by 0.3% on ImageNet with only 80% of the parameters and the same FLOPs.
INTRODUCTION
Convolutional neural networks (CNNs) have achieved state-of-the-art accuracy in wide range of machine learning (ML) applications. However, the massive computational and memory requirements of CNNs remain major barrier more widespread deployment on -smmtte edge and mobile devices. This challenge has motivated a large and active body of research on CNN compression, which attempts to simplify the original model without significantly compromising the accuracy.
Weight pruning 15| 7 17 4 8 has been extensively explored to reduce the computational and memory demands of CNNs. Existing methods create sparse CNN model by iteratively removing ineffective weights/activations and training the resulting sparse model. Such an iterative pruning approach usually enjoys the least accuracy degradation but at the cost of more computationally expensive training procedure. Moreover, training- based pruning methods introduce additional hyper- parameters, such as the learning rate for fine-tuning and the number of epochs before rewinding (20). which make the pruning process even more complicated and less reproducible.
To minimize the cost of pruning, a new line of research proposes pruning-at-inen -iiiiiiaaaaiiioooonn (PAT) [16 27 224,, which identifies and removes unimportant weights in CNN before training. Similar to training-
pruning, PAI assigns an importance score to each individual weight and retains only subset of them by maximizing the sum of the importance scores of all remaining weights. The compressed model is then trained using the same hyperparameters (e.g., learning rate and the number of epochs) the baseline model. Thus, the pruning and training of CNNs are cleanly decoupled, greatly reducing the complexity of obtaining a pruned model. Currently, SynFlow (24) is considered the ffte--ff-the-ar PAI technique it eliminates the need for data during pruning required in prior arts [16] 27 and achieves a higher accuracy with the same compression ratio.
However, existing PAI methods mostly focus fine-graing iee-grained weight pruning. which removes individ- ual weights from the CNN model without preserving any structures. As a result, both inference and training of the pruned model require sparse matrix computation, which is challenging accelerate

---

Under review as a conference paper at ICLR 2023
commercially-availabe ML hardware that is optimized for dense computation (e.g., GPUs and TPUs [14).. According to recent study [6]. even with the NVIDIA cuSPARSE library, one can only achieve meaningful speedup for sparse matrix multiplications
GPUs when the sparsity is over 98%. In practice, it is difficult to compress modern CNNs by more than 50x without a drastic degradation in accuracy (2) Therefore, structural pruning patterns (e.g., pruning weights for the entire output channel) are preferred to enable practical memory and computational saving by avoiding irregular sparse storage and computation.
In this work, we propose novel structured PAI techniques and demonstrate that they can achieve the same level of accuracy as the unstructured methods. We first introduce synaptic expectation (SynExp). a new proxy metric for accuracy, which is defined to be the expected sum of the importance scores of all the individual weights in the network. SynExp is invariant to weight shuffling and reinitialization, thus addressing some of the deficiencies of the fine-grained PAI approaches found in recent studies [22 [5]
We also show that SynExp does not vary as long the layer- density remains the same, irrespective of the granularity of pruning. Based on this key observation, we formulate an optimization problem that maximizes SynExp to determine the layer-wise pruning ratios, subject to model size and/or FLOPs constraints. We then propose PreCrop, structured PAI that prunes CNN models at the channel level in a way to achieve the target layer-wise density determined by the SynExp optimization. PreCrop can effectively reduce the model size and computational cost without loss of accuracy compared to existing fine-grained PAI methods. Besides channel-level pruning, we further propose PreConfig, which can reconfigure the width dimension of CNN achieve better excuracy-complee
trade-off' with almost zero computational cost. Our empirical results show that the model after PreConfig can achieve higher accuracy with fewer parameters and FLOPs than the baseline for a variety of modern CNN architectures.
We summarize our contributions as follows:
We propose to use the SynExp proxy for accuracy and formulate PAI an optimization problem that maximizes SynExp under model size and/or FL.OPs constraints. We also show that the accuracy of the CNN model pruned by solving the constrained optimization is independent of the pruning granularity.
We introduce PreCrop, a channel-level structured pruning technique that builds on the proposed SynExp optimization. Our experiments show that CNN models pruned by PreCrop achieve a similar or better accuracy compared to the state-of-the-ar unstructured PAI approaches. Compared to SynFlow, PreCrop achieves 2.7% and 0.9% higher accuracy on MobileNetV2 and EfficientNet on ImageNet with fewer parameters and FLOPs.
We show that PreConfig can be used to optimize the width of each layer in the network with almost zero computational cost (e.g.. within one second on CPU). Notably, PreConfig can effectively optimize the structure of EfficientNet and MobileNerV2, increasing the accuracy by 0.3% on ImageNet while using 20% fewer parameters and the same FLOPs.
RELATED WORK
Model Compression in General can reduce the computational cost of large networks to ease their deployment in resource-constrained devices. Besides pruning, quantization 3 30 13 neural
architecture search (NAS) B1 23 and distillation [2 28
are also commonly used to improve the
efficiency of the model.
Training-Ban -Baee Pruning uses various heuristic criteria to prune unimportant weights. They typically employ an iterative training stage, which may increase the overall training cost by several folds.
retne- process where the pruning stage is intertwined with the
Existing training- brasedg-bbsedd pruning methods can be either unstructured (7]15) structured [ depending on the granularity and regularity of the pruning scheme. Training- basedd unstructured pruning usually achieves better accuracy given the same model size budget, while structured pruning can achieve more practical speedup and compression without special support from custom hardware.
(Unstructured) Pruning-at- Pruning-a-Iniitializaiioo Initialization (PAD) [6 27 24 methods provide a promising approach to mitigating the high cost of training- based pruning. They can identify and prune unimportant weights right after initialization and before the training starts. Related to these efforts, authors of [
2

---

Under review as a conference paper at ICLR 2023
and [22] independently find that for the existing PAI methods, randomly shuffling the weights within a layer or reinitializing the weights does not cause any accuracy degradation.
large space of candidate trade-off. The typical bases of the NAS search space include the width, depth, resolution, and choice of building blocks. However, existing approaches can only search in a small subset of the possible configurations due to the cost. For example, the search space of the channel width usually only contains a limited of integer values. The cost for NAS is
Neural Architecture Search (NAS)
3 I 26 automatically explores
models to achieve a better accuracy-efficiene
also orders of magnitude higher than training single model. Some NAS algorithms [1 29 use cheap proxy instead of training the whole network, but an expensive reinforcement learning or evolutionary algorithm [9] is still used to predict a good network.
PRUNING-AT-NITIALEZZTTO VIA SYNEXP OPTIMIZATION
In this section, we first review the preliminaries and deficiencies of existing PAI methods. To overcome the limitations, we introduce proxy for the accuracy of the PAI compressed model. We then propose a new formulation of PAI that maximizes the proxy metric using convex optimization.
3.1 PAI BACKGROUND
Preliminaries. PAI aims to prune neural network after initialization but before training to avoid the time-consuming training paaiinng-pninggrre pruning-retraining process. Prior to training. PAT typically uses the magnitude of gradients (with respect to weights) to estimate the importance of individual weights. This requires both forward and backward propagation passes. PAI prunes the weights (W) with smaller importance scores by setting the corresponding entries in the binary weight mask (M) to zero. More concretely, to remove weights W, M is applied to W in an element-wise manner as W O M, where o denotes the Hadamard product.
Popular PAI approaches, such as SNIP 16]. GraSP 27 and SynFlow [24), employ different methods to estimate the importance of individual weights. Single-shot PAI algorithms, such SNIP and GraSP, prune the model to the desired sparsity in single pass. Alternatively, SynFlow, which represents the state-of-the-arr PAI algorithm, repeats the process of pruning a small fraction of weights and re-evaluating the importance scores until the desired pruning rate is reached. Through the iterative process, the importance of each weight can be estimated more accurately.
Specifically, the importance score for a fully connected network used in SynFlow is defined as:
N
[-1 W
S(W)))
T
II k=1+1
W O M+
My
II
M* 1
where N is the number of layers, W and M are the weight and weight mask of the I-th layer, S(W))) is the SynFlow score for a single weight W sg. I. is element-wise absolute operation, and is an all-one vector. Here no training data or labels are required to compute the importance score, thus making SynFlow a data-agnostic algorithm.
Deficiencies. Similar to training- basedd fine-grained pruning 7115. existing PAI methods also use the sum of importance scores the remaining weights as proxy for model accuracy. Specifically, PAI obtains binary weight mask (i.e., the pruning decisions) by maximizing the following objective:
N maximize Σ
N
M
over M subject to
Σ
|
Bpurams
where St is the importance score matrix for the (-th layer, || llo is the number of nonzero entries in a matrix, and Bparans is the target size of the compressed model.
Given the setup of this optimization, is natural that subset of the individual weights will be deemed more important than others. Moreover, existing methods for computing the importance scores all depend on the values of the weights, thus any updates to the weights (such as reinitialization) will easily result in change the accuracy metric (i.e., the of the individual importance scores).
3

---

Under review as a conference paper at ICLR 2023
However, recent studies in 22 show that randomly shuffling the weight mask M' or reinitializing the weights W' does not affect the final accuracy of models compressed by any the existing PAI methods. In addition, they show that the different pruned models have similar accuracy as long as they have the same layer-wise density. This finding suggests that the aforementioned metric is not a good proxy for indicating the accuracy of the pruned model.
3.2 SYNEXP INVARIANCE THEOREM
In this section, we propose proxy metric called SynExp to address the deficiencies of the existing PAI approaches. We argue that a good accuracy proxy should enable PAI to achieve the following:
1. The pruning decision (i.e., weight mask M) can be made before the model is initialized.
2. Maximization of the proxy should output layer-wise density as the result, as opposed to pruning decisions for individual weights.
For the ease of later discussion, we formalize the weight matrix W and weight mask matrix M as two random variables, given fixed density pt for each layer, for random pruning before initialization. If W contains oy parameters, A = {M, M E {0, v1 S i S on, E, Mi = Pi on is the set of all possible M with the same shape as the W that satisfies the layer-wise density (pi) constraint. Then, the random weight mask M for layer / is sampled uniformly from A'. Also, each individual weight W! in layer independently sampled from a given distribution D'.
The observations in Section 3.1 indicate that different values of these two random variables M and W result in similar final accuracy of the pruned model. However, different values do change the proxy value for the model accuracy in existing PAI methods. For example, the SynFlow score in Equation
2 may change under different instantiations of M and W. Therefore, we propose new proxy that is invariant to the instantiation of M and W for the model accuracy the context of PAI the expectation the sum of the importance scores of all unpruned (i.e., remaining) weights. The proposed proxy be formulated as follows:
maximize E. [S] M.W
E. M.W.
Σ St.
M
over pl subject to Σ ai pt S Bparams
where [S] stands for the expectation of the importance score over W and M. In this new formulation, pt is optimized to maximize the proposed proxy for model accuracy. Since the expectation is computed over the W and M, the instantiations of these two random variables do not affect the expectation.
To evaluate the expectation before weight initialization, we adopt the importance metric proposed by SynFlow, i.e., plugging S in Equation I into Equation As result, analytically without forward or backward propagations. This new expectation- based proxy is referred as SynExp, i.c., synaptic expectation.
compute the expectation
We show SynExp is invariant to the granularity of pruning PAI in the SynExp Invariance Theorem, which is stated follows.
Theorem 1. SynExp Invariance Theorem. Given specific CNN architecture, the SynExp (Bia,,:: (SSF)) of any randomly compressed model with the same layer-wise density pr is a constant, independent of the pruning granularity. The constant SynExp equals:
N [Ss]] = NCN+1 (pr Ba-D[[>|)) M.W
E
where N is the number of layers in the network, [<>]] is the expectation of magnitude of distribution D. Ct is the input channel size of layer and is also the output channel size of / 1, and 1 Millo is the layer-wise density. ao
In Equation N and C are all hyperparameters of the CNN architecture and can be considered is also constant under a particular distribution D'. The layer-wise density pr is constants. E E
4

---

Under review as a conference paper at ICLR 2023
the only variable in the equation. Thus, SynExp satisfies both of the aforementioned properties: 1) pruning is done prior to the weight initialization; 2) the layer-wise density can be directly optimized. Furthermore, Theorem shows that the granularity of pruning has no impact on the proposed SynExp metric. In other words, the CNN model compressed using either unstructured or structured pruning method is expected have similar accuracy.
The detailed proof of SynExp Invariance The- orem can be found in Appendix A We also
90
empirically verify it by randomly pruning each
Accuracy (%)
layer of CNN at three different granular levels
80
but with the same layer-wiss wise density. Specifi- cally, we perform random pruning (1) weight-
70
(2) filter-, and (3) channel-level to achieve the desired layer-wise pruning ratios obtained from solving Equation 3 For weight and filter pruning, randomly pruning each layer to match the layer-wise density pl occasionally detaches some weights from the network, especially when the density is low. The detached weights do not contribute to the prediction but are counted as remaining parameters. Thus, we remove the de- tached weights for a fair comparison following
SynFlow
Weight
60
Filter
Channel
(PreCrop)
50
100
50
20
10
5
3
2
1
Params remained (%)
Figure 1: Comparison of the performance using different pruning granularities on ResNet20 using CIFAR-10.
the same approach described in [25]. For channel pruning, it is trivial achieve the target layer-wiss wiir-wiie density while satisfying the constraint that the number of output channels of the previous layer must equal the number of input channels of the next layer. Therefore, we employ PreCrop proposed in Section 4.2. As shown in Figure [ random pruning with different granularity can obtain a similar accuracy compared to SynFlow, as long as the layer-wise density remains the same. consistent with SynExp Invariance Theorem and also demonstrate the
The empirical results efficacy of the proposed SynExp metric. We include additional empirical results using different CNN architectures and other importance scores (e.g. SNIP, GraSP) in Appendix
3.3 OPTIMIZING SYNEXP
As discussed in Section 3.2 the layer-wise density matters for our proposed SynExp approach. Here, we show how obtain the layer-wise wise density in Equation [3 that maximizes SynExp under model size and/or FL.OPs constraints.
3.3. OPTIMIZING SYNEXP WITH PARAMETER COUNT CONSTRAINT
Given that the goal of PAI is to reduce the size of the model, we need to add a constraint on the total number of parameters Bparams (i.e., parameter count constraint), where Bparams is typically greater than zero and less than the number of parameters in the original network. Since lise--wiee density pt the only variable Equation we can simplify the equation by removing other constant terms, as follows:
maximize
Σ
log
over p
subject Σ an . Pt
B Bparams:
(5)
0< Pt L
where is the number of parameters in layer L.
Equation 5 is a convex optimization problem that can be solved analytically We compare the layer-wise density derived from solving Equation 5 with the density obtained using SynFlow. As shown in Figure 2 the layer-wise densities obtained by both approaches are nearly identical, where new formulation eliminates the need for the iterative re-evaluation of the SynFlow scores well as the pruning process in SynFlow. It is also worth noting that the proposed method can find the optimal layer-wise density even before the network is initialized.
5

---

Under review as a conference paper at ICLR 2023
SynFlow
Ours
Layerwise Density
5
10
15
0
10
20
30
Layers
Layers
(a) ResNet-20
(b) MobileNetV2
Comparison of the layer-wise densities obtained by SynExp optimization with parameter count constraint and SynFlow. Higher transparency means that the problem is constrained by smaller parameter count.
Figure 2:
3.3.2 OPTIMIZING SYNEXP WITH PARAMETER COUNT AND FLOPS CONSTRAINTS
formulate PAI convex optimization problem with constraint on the model size. However, the number of parameters does not necessarily reflect the performance (e.g., throughput) of the CNN model. many cases, CNN models compute-bound on commodity hardware 14 Therefore, we also introduce FLOPs constraint in our formulation.
As discussed in Section 3.3.1
With the existing PAI algorithms, is not straightforward directly constrain the optimization using a bound on the FLOP count. The savings in FLOPs are instead the byproduct of the weight pruning as specified in Equation 2 In contrast, is much easier account for FLOPs in our formulation. which aims to determine the density of each layer opposed to the inclusion of individual weights from different layers. Thus for each layer, we can easily derive the required FLOP count based on the density (pr). After incorporating the constraint on FLOPs (BFLoPs), the convex optimization problem becomes:
maximize Σ
log
over subject to
01 Pt I B paramss
Σ
3: P S BFOOR
(6)
t=
t=1
0 Pt S SI
where Br the number of FLOPs the thh layer.
Since the additional FLOOS constraint is linear, the optimization problem Equation 6 remains convex and has analytical solutior By solving SynExp optimization with a fixed Bpurans but different BELORs obtain the layer-wist density for various mod- els that have the same number parameters but dif- ferent FLOPS. Then, perform random weight pruning on the CNN model to achieve the desired layer-wise density. We compare the proposed Syn- Exp optimization (denoted as Ours) with other pop- ular PAI methods. As depicted in Figure 3l given a fixed model size (1.5 10º in the figure), our method can be used to generate Pareto Frontier that spans the spectrum of FLOPs, while other methods can only have fixed FLOPs. Our method dominates all other methods in terms of both accuracy and FLOPs reduction.
85
84
Top-1 Accuracy
83
Ours
82
SynFlow
SNIP
GraSP
2
3
4
5
6
FLOPs
1e6
Figure 3: Comparison our method with other PAI methods - we repeat the experiment using ResNet-20 on CIFAR-10 five times and re- port the and variance (error bar) of the racy. All models in figures have 1.5 10* parameters.
We include analytical solutions for Equation and Equation 6 in Appendix for completeness.

---

Under review as a conference paper at ICLR 2023
VPIC
C
Drop channels
Pad zeros
tth CONV layer
Figure 4: Illustration of PreCrop for layers with residual connections Ct and C.+1 represent the number of input channels of layer I and + respectively, pi represents the density of layer
4 STRUCTURED
The SynExp Invariance Theorem shows that the pruning granularity of PAI methods should not affect the accuracy of the pruned model. Channel pruning, which prunes the weights of the CNN at the output channel granularity, is considered the most coarse-grained and hardware-friendll pruning technique, Therefore, applying the proposed PAI method for channel pruning can avoid both complicated retraining/re-tus ttnnigg procedures and irregular computations. In this section, propose structured PAI method for channel pruning, named PreCrop. to prune CNNs in the channel dimension. In addition, propose variety of PreCrop with relaxed density constraints to reconfigure the width of each layer in the CNN model, which is called PreConfig.
4.1 PRECROP
Applying the proposed PAI method to channel pruning requires a two-step procedure. First, the layer-wise density is obtained by solving the optimization problem shown in Equation [ Second, we need to decide how many output channels of each layer should be pruned to satisfy the layer-wise density. However, it is not straightforward to compress each layer to match given layer-wise density due the additional constraint that the number of output channels of the current layer must match the number of input channels of the next layer.
We introduce PreCrop, which compresses each layer to meet the desired layer-wise density. Let C and C++1 be the number of input channels of layer / and / 1, respectively. C++11 is also the number of output channels of layer l. For layers with no residual connections, the number of output channels of layer is reduced to [vpi- Ci+1]. The number of input channels of layer I needs to match the number of output channels of layer I, which is also reduced to y C.+1 Therefore, the actual density of layer after PreCrop is /PI-1 Pi instead of pt. We empirically find that /PI-1 Pi is close enough to pt because the neighboring layers have similar layer-wise densities. Alternatively, one can obtain the exact layer-wise density p by only reducing the number of input output channels of layer. However, this approach leads to significant drop in accuracy, because the number of the input and output channels can change dramatically (e.g., PrCi & C1+1 G; > 1)) This causes the shape of the feature map to change dramatically in adjacent layers, resulting in information loss.
For layers with residual connections, Figure 4 depicts approach to circumvent the constraint on the number of channels of adjacent layers. We can reduce the number input and output channels of layer 1 from C and C1+1 CPICi and /piCC+1, respectivll... In this way, the density of each layer can match the given layer-wise density obtained from the proposed PAI method. Since the output of layer needs to be added element-wisely with the original input to layer 1, the output of layer / is padded with zero-valued channels to match the shape of the original input. In our implementation, we simply add the output of layer to the first VprC141 channels of the original input layer thus requiring no extra memory or computation for zero padding. PreCrop eliminates the requirement for sparse computation in existing PAI methods and thus can be used to accelerate both training and inference of the pruned models.
4.2 PRECONFIG: PRECROP WITH RELAXED DENSITY CONSTRAINT
PreCrop uses the layer-wise density obtained from solving the convex optimization problem, which is always less than 1 following the common setting for pruning (i.e., Pl 1). However, this constraint on layer-wise density is not necessary for our method since we can increase the number of channels (i.e.. expand the width of the layer) before initialization. By solving the problem in Equation 6 without the constraint p1 < 1, we can expand the layers with a density greater than (p) > 1) and prune the layers with density less than I (pi 1). We call this variant of PreCrop PreConfig
7

---

Under review as a conference paper at ICLR 2023
SynFlow
PreCrop-FLOPS
PreCrop-Paramt
90
90
ResNet20
70
70
50
50
30
30
100
50
20
10
5
100
50
20
10
5
WideResNet20
Accuracy (%)
90
90
70
70
100
50
20
10
5
3
2
1
100
50
20
10
5
3
2
1
MobileNetV2
90
90
80
80
100
50
20
10
100
50
20
10
3
Params remained (%)
FLOPs remained (%)
(a) PreCrop-Params vs. SynFlow.
(b) PreCrop-FLOPs vs. SynFlow.
Figure Comparison of PreCrop-Params and PreCrop-FLOPs with SynFlow we repeat the experiment using ResNer20 (top), WideResNet20 (middle), and MobileNetV2 (bottom) on CIFAR-10 10 three times and report the mean and variance (error bar) of the accuracy.
If we set Bparams and BELOPs to be the same as the original network, we essentially reconfigure the width of each layer of a given network architecture under certain constraints on model size and FLOPs.
The width of each layer in CNN is usually designed manually, which often relies extensive experience and intuition. Using PreConfig, we can automatically determine the width of each layer in the network achieve better cost-accuracy trade-off. PreConfig can also be used (a part of) an ultra-fast NAS. Compared to conventional NAS, which typically searches on the width, depth, resolution, and choice building blocks, PreConfig only changes the width. Nonetheless, PreConfig only requires minimum amount of time and computation compared to NAS methods; it only needs to solve a relatively small convex optimization problem, which can finish within second on a CPU.
EVALUATION
In this section, empirically evaluate PreCrop and PreConfig. We first demonstrate the effectiveness of PreCrop by comparing it with SynFlow. We then use PreConfig to tune the width of each layer and compare the accuracy of the model after PreConfig with the original model. We perform experiments using various modern CNN models, including ResNet (10), MobileNetV2 [21] and EfficientNet (23), on both CIFAR-10 and ImageNet. We set all hyperparameters used to train the models pruned by different PAI algorithms to be the same. See Appendix B for detailed experimental settings.
5.1 EVALUATION OF PRECROP
For CIFAR-I0, we compare the accuracy of SynFlow (red line) and two variants of PreCrop: PreCrop- Params (blue line) and PreCrop-FLOPs (green line). PreCrop-Paramy Paramm adds the parameter count imposes the FLOPPs constraint into the convex optimization constraint whereas PreCrop-FLOOh PreCrop-Params achieves similar even better accuracy as problem. As shown in Figure 5a
SynFlow under wide range of different model size constraints, thus validating that PreCrop- Paramms can be as effective the fine-grained PAI method. Considering the benefits of structured pruning, PreCrop-Parama should be favored over existing PAI methods. Figure further shows that PreCrop- FLOPs consistently outperforms SynFlow by a large margin, especially when the reduction in FLOPs large. The experimental results show that PreCrop-FLoo -LOPP should adopted when the performance of the model is limited by the computational cost.
8

---

Under review as a conference paper at ICLR 2023
Table I: Comparison PreCrop with SynFlow on ImageNet - The dagger()) implies that the
numbers are theoretical without considering the overhead of sparse matrices in storing and computing.
NETWORK
METHODS
FLOPs (G)
PARAMS (M)
ACCURACY (%)
BASELINE
3.64
21.80
73.5
RESNET34
SYNFLOW
2.781
10.91
72.1 (-1.4)
PRECROP
2.73
11.09
71.5 (-2.0)
BASELINE
0.33
3.51
69.6
SYNFLOW
0.26!
2.441
67.6 (-2.0)
MOBILENETV2
PRECROP
0.26
2.33
68.8 (-1.8)
SYNFLOW
0.21
1.91
64.5 (-5.1)
PRECROP
0.21
1.85
67.2 (-2.4)
BASELINE
0.40
5.29
73.0
EFFICIENTNETBO
SYNFLOW 0.30
3.721
71.8 (-1.2)
PRECROP
0.30
3.67
72.7 (-0.3)
Table I summarizes the comparison between PreCrop and SynFlow ImageNet. For ResNet-34, PreCrop achieves 0.6% lower accuracy compared to SynFlow with a similar model size and FLOPs. For both MobileNetV2 and EfficientNetBO, PreCrop achieves 1.2% and 0.9% accuracy improvements compared to SynFlow with strictly fewer FLOPs and parameters, respectively. The experimental results on ImageNet further support SynExp Invariance Theorem that coarse-grained structured pruning (e.g., PreCrop) can perform as well as unstructured pruning. In conclusion, PreCrop achieves favorable accuracy and model size/FLOPs tradeoff compared the state-of-the-art PAI algorithm.
5.2 EVALUATION PRECONFIG
As discussed in Section 4.2
PreConfig can be viewed as an ultra-fast NAS technique, which adjusts the width of each layer in the model even before the weights are initialized.
Table 2: PreConfig on ImageNet.
NETWORK
METHODS
FL.OPs (G)
PARAMS (M)
ACCURACY (%)
BASELINE
3.64
21.80
RESNET34
PRECONFIG
3.64
16.52 (75.8%)
73.3 (-0.2)
BASELINE
0.33
3.51
69.6
MOBILENETV2
PRECOnfIg
0.32
(97.0 %
2.83 (80.6%)
69.9 (+0.3)
BASELINE 0.40
5.29
73.0
EFFICIENTNETBO
PRECONHI 0.40
4.29 (81.1 %
73.3 (+0.3)
Table compares the accuracy of the reconfigured model with the original model under similar model size and FLOPs constraints. For ResNet34, with similar accuracy, we reduce the parameter count achieve 0.3% higher accuracy than the baseline with 20% fewer
by 25%. For MobileNetV2,, parameters and 3% fewer FLOPS. For the EfficientNet, we can also achieve 0.3% higher accuracy than the baseline with only 80% of the parameters and the same FLOPs. Note that EfficientNet is identified by a NAS method. As PreConfig only changes the number of channels of the model before initialization, we believe it also applies other compression techniques.
CONCLUSION
In this work, we theoretically and empirically show that the accuracy of the CNN models pruned using PAI methods depends on the layer-wise density. We formulate PAI as a simple convex SynExp further propose PreCrop and PreConfig to prune
optimization. Based on SynExp optimization, and reconfigure CNNs in the channel dimension. Our experimental results demonstrate that PreCrop outperform existing fine-grained PAI methods various networks and datasets.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
The proof of SynExp Invariance Theorem is stated in the appendix with explanations. We provide the source code for the key experiments in the paper. We thoroughly checked the implementation and also verified empirically that the results in this paper are reproducible. The source code will be made available through GitHub.
REFERENCES
[1] Mohamed S Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas D Lane. Zero-cost proxies for lightweight nas. arXiv preprint wrXiv:2101.08134 2021.
[2] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? arXiv preprint arXiv:2003.03033 2020.
[3] Zhen Dong. Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace-weighted quantization of neural networks. arXiv preprint arXiv: 1991.038522 2019.
[4] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03633 2018.
[5] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neu- ral networks at initialization: Why are we missing the mark? arXiv preprint arXiv:2009.08577 2020.
[6] Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '20, IEEE Press, 2020. ISBN 7781728199986
[7] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
[8] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[9] Pawan Harish and Petter J Narayanan. Accelerating large graph algorithms on the gpu using cuda. In International conference on high-performance computing, pp. 197-208. Springer, 2007.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
[11] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. Proceedings of the IEEE international conference on computer vision, pp. 1389- 1397, 2017.
[12] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint
2015.
[13] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient anteger-aithhmetic-on/ inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713, 2018.
[14] Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami. Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy,
10

---

Under review as a conference paper at ICLR 2023
James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. In-datacenter performance analysis of tensor processing unit. SIGARCH Comput. Archit. News, 45(2): 12, jun 2017. ISSN 0163-5964. doi: 10.1145/ 1140659.30802465 URL URL
11e...141111115111 3080246
[15] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
[16] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:/810.023440 2018.
[17] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXi:: 2018.
[18] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-5066, 2017.
[19] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference artificial intelligence. pp. 4780-4789, 2019.
[20] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint arXiv:2003.02338 02388, 2020.
[21] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
[22] Jingtong Su, Yihang Chen, Tianle Cai, Tianhao Wu, Ruiqi Gao. Liwei Wang, and Jason D Lee. Sanity-checking pruning methods: Random tickets can win the jackpot. arXiv preprint arXiv:2009.1 / 1094, 2020.
[23] Mingxing Tan and Quoc Lc. Efficientnet: Rethinking model scaling for convolutional neural networks. International Conference on Machine Learning, pp. 6105-6114, 14. PMLR, 2019.
[24] Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. arXiv preprint arXiv:2006.00 054677 2020.
[25] Artem Vysogorets and Julia Kempe. Connectivity matters: Neural network pruning through the lens of effective sparsity. arXiv preprint 1rXiv:2107.02306, 2021.
[26] Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. Fhnetv2: Differentiable neural architecture search for spatial and channel dimensions. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pp. 12965-12974, 2020.
[27] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.073376, 2020.
[28] Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming distill: Pree-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8715-8724, 2020.
11

---

Under review as a conference paper at ICLR 2023
[29] Dongzhan Zhou, Xinchi Zhou, Wenwei Zhang. Chen Change Loy, Shuai Yi, Xuesen Zhang, and Wanli Ouyang. Econas: Finding proxies for economical neural architecture search. In Proceedings of the IEEE/CVF Conference computer vision and pattern recognition, pp. 11396-1 1404,, 2020.
[30] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160, 2016.
[31] Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 2016.
12

---

Under review as a conference paper at ICLR 2023
A PROOF OF SYNEXP INVARIANCE THEOREM
Theorem 1. Given specific CNN architecture, the SynExp Fias.w((SSP) of any randomly com- pressed model with the same layer-wise density Pt is a constant, independent of the pruning granular- ity. The constant SynExp equals:
[85 NCN+1 [m EnnD (1=|)) M,W
where N is the number of layers in the network, E the expectation of magnitude of distribution D. C is the input channel size of layer and is also the output channel size of l 1, and pt ai || lo the layer-wise density.
Proof. Assuming the network has N layers, weight matrix W E RCixC mask matrix M' E {0, 13 1 Ci and C1+1 the input and output channel size of layer 1. As the output channel size of any layer I equals to the input channel size of the next layer + 1, we have C++1 = C1+1, < N.
We first prove the Theorem on fully-connected network, and can extend it to CNNs easily. From Equation| | in a fully-connected network, the Synaptic Flow score for any parameter W with mask Mi in layer equals to:
N
(W())
T
II k=1+1
wh @
M²
W Mas)
W O OM*
I
(8)
We compute the SynExp of the layer (BiM, (Srr) then the SynExp of the network is simply the sum of SynExp of all layers:
E B (SsF)
Σ
E [MM. (Ssr))
We define the expectation value for input channel i, output channel j. and the whole layer in layer / as Fl Fy F(..)): and E [().))
Elo,,) = C1+1
1
Σ
W Manny)Maa
(10)
Bl(a) Ci Σ
(11)
E B(o.)) = Bl.,)) = C,0H+ Σ Di (1a
1
Σ
M (KJ))
pr
(12)
1.j
Here Eydy to denote EneD [|=]]-
As the weight in layer / is sampled from distribution D, and the mask matrices are also randomly sampled, have
E (x,+) (s,a) L = prEE
(13)
With Ey E. and E)..)) we can rewrite Equation 8 to:
N II
Cet1 ) CHe2 Eaas Cr C_
1-2 Crtttn)) (14)
E(SSF(W//.))) (1.))]]
k=1+2
k=1
13

---

Under review as a conference paper at ICLR 2023
Combining Equation 3 and 14 for each layer are independent:
because the instantiation of the weight matrices and mask matrices
C1 CL+
Cr
C1+1
Bj =E
SSF( S
(121)
Σ
E
(SF(W((.)))
N
C.
C++1
1-2
II 1=1-2
P& C Bypa
Σ
0411142BBoh B411 (1,))
p Bay Pi- Pt-1C1-1R(1. (+
II C
1= 1=1
N
1-2
II k=1-2
Ps. P Bypz)
CiC++1 (014:C142BBDH++) PeRyD' Pi- 12-1C
II
Ph C
Cx+1
II
(P:C
(15)
According to Equation 9
R WI,,W((SsF)
Σ
CN+1 (CE
1=1
(16)
N
= NCN41
I [PI E E=>D(<=]),
SynExp Invariance Theorem can also be extended to CNNs, as it is obvious that SynExp CNNs proportional to that of fully connected networks. Thus the difference of SynExp between CNNs and fully connected networks for each layer is only factor equal to C.41* the convolutional layer.
K² where K the kernel size of
B SOLUTION OF THE OPTIMIZATION PROBLEM
For the convex optimization problem in Equation 5 Equation 6 or PreConfig. we can simply use -ukreee (KKT) conditions to analytically solve it. We include the solutions as follows completeness. practice, we use convex solver to solve the problem to avoid the piecewise function.
B.1 SOLUTION FOR EQUATION
Pt min /
1)
N
(17)
where satisfies:
min(or, 11) B Bparams
B.2 SOLUTION FOR EQUATION
1
Pt min(
+ pior
u231
where μt, u1 satisfy:
Oy min(
1) B Bparams
(18)
+ Dotln
8 min
+ p2 1) = Baoss
14

---

Under review as a conference paper at ICLR 2023
94
Accuracy (%)
92
90
SynFlow
88
Weight
Filter
86
Channel (PreCrop)
100
50
20
10
5
3
2
1
Params remained (%)
Figure 6: Comparison of the performance using different pruning granularities on VGG16 using CIFAR-10.
90
90
Accuracy (%)
Accuracy (%)
80
80
70
Weight (SNIP)
Weight (GraSP) Channel (PreCrop)
70
Channel (PreCrop)
60
100
50
20
10
5
2
100
50
20
10
5
3
2
Params remained (%)
Params remained (%)
Figure 7: Comparison the performance using different pruning granularities on ResNer20 using CIFAR-10. SNIP (left) and GraSP (right) importance scores are used.
B.3 SOLUTION FOR PRECONFIG
1
+ b0ln
U228t
1
where pr, p2 satisfy:
Bparams
H2Bt + a1
(19)
BBops
illl + P2Bi
In practice, avoid solving the μ. convex optimization solver, which can obtain the solution with CPU within a second for such a small scale convex optimization.
C MORE EMPIRICAL RESULTS ON SYNEXP INVARIANCE THEOREM
We show more empirical results that validates SynExp Invariance Theorem. We first show the comparison of the performance using different pruning granulariteis on VGG16 using CIFAR-10. All the settings in this experiment is the same in Figure [ except this experiment is done VGG16.
Then we also verify that SynExp Invariance Theorem not only holds for SynFlow, but also holds for other PAI algorithms. In this experiment, first other PAI (i.e., SNIP and GraSP) obtain the layerwise density p:- Then we use random pruning to match pt in the channel level. The results are shown in Figure 7
As shown all the above experiments, as long as the layerwise density is the same, the pruning granularties do not affect the model accuracy.
15

---

Under review as a conference paper at ICLR 2023
Baseline
Channel Width
1000
PreConfig
500
10
Layers
Figure 8: Comparison of the channel width of EfficientNetBO before and after PreConfig.
D CHANNEL WIDTH COMPARISON
We also include comparison the channel width between the baseline EfficientNetBO and PreConfig EfficientNetBO in Figure g
E EXPERIMENT DETAILS
E.
IMPLEMENTATION
We adapt model implementations of ResNet, ShuffleNet, and MobileNetv2 from imgelsmot 2 implementations of SynFlow, SNIP, and GraSP are based the codebase of SynFlow
The
E.2 HYPERPARAMETERS
Here we provide the hyperparameters used in training all models in Table 3 No AutoAugment, Label Smoothing, or stochastic depth is used during training. All the CIFAR- 10 models are trained with same hyperparameter setting.
Table 3: Hyperparameters used in training.
ImageNet
CIFAR-10
MobileNet
ResNet
EfficientNet
Optimizer
momentum
momentum
momentum
momentum
Training Epochs
160
180
90
150
Batch Size
128
256
512
256
Initial Learning Rate
0.1
0.025
0.2
0.035
Learning Rate Schedule
linear
drop each epoch
drop at 30, 60 epoch
drop each epoch
Drop Rate
N.A.
0.98
0.99
Weight Decay
10-4
x 10-5
10
4 x 10-5
16


Reviews:
[{"review_id": "kvG4zoeGn2", "reviewer": "Reviewer_kacd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied Pruning-at-initialization (PAI). The authors show that the accuracy of CNN models pruned by a PAI method depends on the layer-wise density. They further propose a structured PAI method, PreCrop, to prune CNNs in the channel dimension.", "summary_of_the_review": "Overall speaking, the paper provides some insights into PAI, especially the results of theorem 1. However, the low baseline accuracy of MobileNet-V2 and Efficientnet-B0 makes the empirical results less meaningful. In addition, soft channel pruning methods can achieve good performance giving a similar training budget to PAI.", "strength_and_weaknesses": "+PAI saves training costs compared to standard pruning methods.\n\n+They theoretically show that PAI only depends on the layer-wise density (in theorem 1). \n\n+Emperical results seem good across different pruning settings, such as weight level, filter level, and channel level.\n\n-The advantage of the proposed method is not always obvious compared to SynFlow.\n\n-The baseline accuracy of MobileNet-V2 and Efficientnet-B0 is much lower than the reported accuracy in their original papers. Authors should double-check their implementations. The results on top of lower baselines are less meaningful.\n\n-Soft channel pruning methods, such as SCP [1], use no or small fine-tuning epochs. It would be better if the authors could compare the proposed method with soft channel pruning methods since their overall budget is similar.\n\n[1] Kang, Minsoo, and Bohyung Han. \"Operation-aware soft channel pruning using differentiable masks.\" International Conference on Machine Learning. PMLR, 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "a-kEbfCvTPm", "reviewer": "Reviewer_seit", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed SynExp as a proxy for accuracy and format pruning at initialization as an optimization problem to maximize SynExp. Based on this, they proposed PreCrop for channel pruning and PreConfig for channel optimization, both at the initialization. Experimental results show the proposed method can further improve existing models like EfficientNet and MobileNetV2 in terms of accuracy vs. model size trade-off. ", "summary_of_the_review": "Overall, the proposed method aims to solve an important problem. However, it lacks a comprehensive comparison to existing PAI/zero-shot NAS work, which makes the contribution less convincing. The core novelty also highly depends on whether SynExp is actually better than existing ones. ", "strength_and_weaknesses": "Strength:\n1. Firstly, the paper is generally well-written and easy to follow. \n2. The proposed heuristic to estimate the network's performance outperforms existing ones on the ablation benchmark.\n3. Compared to the existing PAI method SynFlow, the proposed method has a better FLOPs/params vs. accuracy trade-off at an even coarser granularity. \n\nWeakness:\n1. Apart from PAI methods, the proposed method uses a very similar setting as zero-shot neural architecture search (NAS) (e.g., [a, b, c]) when only the channel dimension is searched (zero-shot NAS also needs to estimate the performance of a model architecture before training and compare different architectures to pick out a good one). The paper should reference such literature and compare the proposed method with zero-shot NAS, especially the performance estimation function.  We can keep the search space the same (i.e., only the channel dimension) and use the model performance estimator from zero-shot NAS literature for comparison to see if the proposed method is better. \n2. Although the proposed method aims for a training-free setting, it would be better to also include a comparison with post-training pruning results to see how large the gap is.  \n3. In the paper, it says, \"The SynExp Invariance Theorem shows that the pruning granularity of PAI methods should not affect the accuracy of the pruned model\". However, in real cases, the granularity could affect the performance of pruning by a lot (e.g., fine-grained pruning can achieve a larger learning rate). How do we explain the contradiction?\n\n[a] Zhang and Jia, GradSign: Model Performance Inference with Theoretical Insights\n[b] Lin et al., Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition\n[c] Mellor et al., Neural architecture search without training \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RGDmPZIm8s6", "reviewer": "Reviewer_Ss1f", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper firstly proposes a metric called SynExp that is an expectation of the SynFlow values over the weights and mask distributions. According to the authors, SynExp is a constant and independent of the pruning granularity so the pruning granularity may not have an influence on the trained models' accuracy. Later on, the authors propose two techniques, PreCrop and PreConfig, that prune the model structurally and adjust the models' parameters, respectively. PreCrop solves a convex optimization problem that results in an optimal structure, while PreCrop optimizes the models' structures in place. Experiments show the empirical performance of the proposed techniques. ", "summary_of_the_review": "I am open to change my score - but currently I feel there are some points make me confused in this paper, so I am giving a 5. ", "strength_and_weaknesses": "Strengths:\n- The presentation is generally clear and easy to follow. \n- The methods are conceptually simple yet seem to be effective. \n\nWeakness:\n- Some claims are not well supported. (See below)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5u3VQNUL4W", "reviewer": "Reviewer_CtCn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied the problem of structured Pruning-at-initialization (PAI) on CNNs. It first introduced synaptic expectation (SynExp) as a proxy metric for accuracy. Then, this paper formulated an optimization problem to maximize SynExp for determining the layer-wise pruning ratios, which are subject to model size/FLOPs constraints. The authors observed empirically and prove theoretically that the accuracy of CNN models pruned by PAI depends on the layer-wise density. Experiments are conducted on several CNN architectures like ResNet, MobileNetV2, and EfficientNet, on CIFAR-10 and ImageNet datasets to verify their statement and proposed method.", "summary_of_the_review": "Overall, this paper has merits in writing, formulation, and method originality. However, the experiments and comparisons in this paper are insufficient. The results also seem not competitive.\n\nMy final rating will be based on the authors’ responses to my concerns.\n", "strength_and_weaknesses": "### Strengths:\n\n   - This paper is generally well-written, and the proposed method is clearly introduced.\n\n   - This paper formulated structured PAI as an optimization problem via maximizing SynExp under model size/FLOPs constraints, which seems interesting.\n\n   - Multiple CNN architectures are used, including ResNet, MobileNetV2, and EfficientNet, on CIFAR-10 and ImageNet datasets.\n\n### Weaknesses:\n\n   - The comparisons in the experimental section are insufficient. This paper stated SynFlow [1] is the current state-of-the-art approach, and this paper only compared to the baseline model and SynFlow without any other PAI methods on each individual architecture. However, SynFlow was published in 2020 which was two years ago. There must be some new PAI papers in the same field. It’s better to compare with them in this paper. This is necessary to examine the advancement and advantages of the proposed method over other competitive approaches.\n\n[1] Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. arXiv preprint arXiv:2006.05467, 2020.\n\n   - In Figure 2, it seems the layer-wise densities of the proposed method are close to SynFlow and have no significant advantage over it. The results in Table 1 also reflect this.\n\t\n   - In the experimental part, since ResNet-50 is a more typical architecture than ResNet-34, it’s better for the authors also to provide results on ResNet-50, so that we can have an intuitive and straightforward comparison with other strong pruning methods.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject