The Crossword Puzzle: Simplifying Deep Neural Network Pruning with Fabulous Coordinates
Authors: Yicun Duan
Wangkai Jin
Xiangjun Peng
Venue: Submitted to ICLR 2023 (2023)
Keywords: Pruning

Abstract: Pruning is a promising technique to shrink the size of Deep Neural Network models with only negligible accuracy overheads. Recent efforts rely on experience-derived metric to guide pruning procedure, which heavily saddles with the effective generalization of pruning methods. We propose The Cross Puzzle, a new method to simplify this procedure by automatically deriving pruning metrics. The key insight behind our method is that: \textit{For Deep Neural Network Models, a Pruning-friendly Distribution of model's weights can be obtained, given a proper Coordinate}. We experimentally confirm the above insight, and denote the new Coordinate as the Fabulous Coordinates. Our quantitative evaluation results show that: the Crossword Puzzle can find a simple yet effective metric, which outperforms the state-of-the-art pruning methods by delivering no accuracy degradation on ResNet-56 (CIFAR-10)/-101 (ImageNet), while the pruning rate is raised to 70\%/50\% for the respective models.

Under review as a conference paper at ICLR 2023
THE CROSSWORD PUZZLE: SIMPLIFYING DEEP NEU- RAL NETWORK PRUNING WITH FABULOUS COORDI- NATES
Anonymous authors Paper under double-blind review
ABSTRACT
Pruning is a promising technique shrink the size of Deep Neural Network mod- els with only negligible accuracy overheads. Recent efforts rely on experience- derived metric to guide pruning procedure, which heavily saddles with the effec- tive generalization of pruning methods. We propose The Cross Puzzle, new method to simplify this procedure by automatically deriving pruning metries. The key insight behind our method is that: For Deep Neural Network Models, a frieiny-rriendly Distribution of model's s's weights can be obtained, given a proper Coordinate. We experimentally confirm the above insight, and denote the new Coordinate as the Fabulous Coordinates. Our quantitative evaluation results show that: the Crossword Puzzle find simple yet effective metric, which outper- forms the state-of-the off-the-an art pruning methods by delivering no accuracy degradation ResNet-56 (CIFAR-10)-100 (ImageNet), while the pruning rate is raised 70%/50% for the respective models.
INTRODUCTION
Pruning Deep Neural Network models is promising the reduce the size of these models while keep- ing the same level of the accuracy. Prior arts focus the designs of the pruning method, such as iterative pruning (Han et al. (2015a), one-shot pruning (Lee et al. (0188), pruning without training (Ramanujan et al. (020)), etc. However, prior works craft the pruning metrics via additional efforts, based on the testing experiences.
Our goal in this work is to design a method for automatically searching proper metric for model pruming. Based on the classic pipelines (e.g. Genetic Algorithm (Mitchell (1998) and Ant Colony Optimization (Dorigo & Di Caro (1999)), first systematically summarize such method requires three components: Basic building blocks of pruning criteria; Objective function to evaluate auto-generated pruning metrics; Heuristic searching process to guide the searching. Based on the above summary, prior works mainly focus on the first and third components (for instance, we can use Lnrmorm (Li al. (2016)) and geometric median (He et al. (2018b) as building blocks, and simulated annealing (Kirkpatrick et al. (1983)) as our searching guider). Therefore, it's still unclear that how objective functions should be measured for the quality of a certain pruning metric (namely the unfilled letters in our "crossword puzzle" denotation).
This motivates us to examine the essential condition(s) of a good-quality pruning criterion. Based on a simple magnitude- based pruning method (Han et al.) (2015)) and the follow-up weight distribution analysis (Liu et al. (0188), we formalize that one essential condition and describe it as follows:
Given coordinate V (the formal expression of a pruning criterion) and neural network model M, y highly likely to high-qualified! if the distribution D(M) got from V/ (.M) obeys the following requirements:
Centralized distribution: the statistics concentrated which is an important symbol of overparameterized neural networks.
center in the distribution,
We refer "a coordinate to he highly-qualifedd,, if we can use it to prune neural network model with (almost) no accuracy drop under relatively-high pruning rate.

---

Under review as a conference paper at ICLR 2023
Retraining recovers centralized distribution: through retraining, statisties regather at the original distribution center after the peak on that center is cutting off by pruning, which means that the objects located at the center are able to replace each other.
Central collapse: the end of the pruning, we can observe that retraining can't drive the statistics to fill the center void again (namely central collapse). This is a signal demonstrat- ing that there is nearly none redundancy in the model, which also alludes the success of our criterion selection.
We denote such coordinate v as the Fabulous Coordinates, and the distribution D generated by it as the Fabulous Distribution. Based on our formalization, we can convert the pruning criterion searching problem into finding the Fabulous Coordinates. By quantitatively depicting the Fabulous Distribution using Loose-KI-CCntten -l-CCerralization--egeee (LKIL-CD), we formulate the objective function and build the Crossword Puzzle, a pruning criteria searching framework.
We experimentally evaluate the effectiveness of the Crossword Puzzle. First, we use the Crossword Puzzle to find Fabulous Coordinate. Then, we leverage the found Fabulous Coordinate guide our pruning, and the results confirm the effectiveness of our method on CIFAR-10 10 and ImageNet. Our results show that we can prune VGG-16-bn (CIFAR-10/ResNett- to remain about 50/30/50% weights") without accuracy degradation, which beats the human-tumed pruning pipelines such as FPGM (He al. (2018b)) and RL-MMTS (Wang reduction on weights also brings about faster inference. On CIFAR-10, we can achieve maximal 3x accelcration of ResNet-50 with the same accuracy, compared to original model. The boost of inference speed and the reduction of memory footprint make the application of high-accuracy models on edge devices feasible.
(CIFAR- 10)- 101 (ImageNet)
(2022). This
2 RELATED WORKS
In this section, we brief related works to justify the novelty of our work. We first classify prior works based on the pruning criterion, which include magnitude-fmmpat--oss pruning. We then give an overview works that utilizes distribution analysis. Finally, we justify the novelty of our method.
2.1 MAGNITUDE-BASED PRUNING
We refer magnitude-basee pruning to the network-slimming approaches based on the importance of neural network's 's weights, which are measured by L, Ln/[L--orrm/absoluee value of network's (either locally or globally). Though the rationale behind them is intuitive, the methods can usually achieve outstanding pruning results with an casy-to-operate pipeline, which extensible to be applied on different types of neural network (like Multi-layer Perceptron Hertz et al. (1991). Convolution Neural Network (CNN) Han et al. (2015b) and Trans- former Mao et al. (021)). For CNN. Han et al. (2015a)'s 's Deep Compression intrigues lots of follow-up works on this direction (e.g. Li et al. (2016); Gordon et al. (2020); Elesedy et al. (2020); Tanaka et al. 22020) Liu et al. (2021)) ). More recently, the Lottery Ticket Hypothesis Frankle & Carbin (019) shares some similarities with this line of works: such method assumes that subnet, neural network, can be trained separately; and it can achieve a similar level of the testing accuracy (under the same amount of training iterations). A substantial amount of efforts focus on extending this hypothesis furthermore (e.g. Zhou et al. (2019): Ramanu-
from dense randomly-initialin
jan al. (2020); Malach et al. (2020); Pensia al. (2020); Orseau et al. (2020) Qian & Klabjan
(2021) Chijiwa et al. 20212) da Cunha et (2022)
2.2 IMPACT-BASED PRUNING
We refer impact-based pruning to methods for eliminating the weights while minimizing overheads on the model. The ancestors (LeCun et al. (989); Hassibi et al. (1993) of this series of works aim to find a criterion different from those described in magnitude- approaches with a possibly more reasonable theoretical explanation. OBD LeCun et al. (1989) and OBS Hassibi al. (1993)
Since the discrepancy between weight reduction and FL.OPs decrease is usually small, we only showcase FL.PPs decrease in our experimental results (Section 5)
2

---

Under review as a conference paper at ICLR 2023
successfully craft slimmer but fairly-simple model, using approximated Hessian matrix its in- verse. up works inherit their philosophy, and extensively apply to more complicated network architectures. Among them, MMlchanov et al. (2019) proposes to utilize the first-order derivative to query the importance of weights, and, if needed, coarse-estimated second-order derivative can assist the decision making. After that, [Singh & Alistarh (2020) leverages empirical fisher matrix to better approximate the Hessian matrix and achieve smaller accuracy degradation. Waan et al. (2020) views the problem from the perspective of gradient flow, rather than traditional weights-andd relation. Other works attempt to bypass the constrains of measuring weights' importance using first-/second- derivative. For example, these works score weights from the scaling factor of batch normalization layers (Liu et al. (2017) or channel's 's output (Huang & Wang (0077). and etc. A substantial amount of works utilize such a philosophy to improve the pruning performance (e.g. [Lebedev Lempitsky 2013); Molchanov et al.] (2016); Yu et al. (2017); Dong et al. (2017a); /Zeng & Urtasun (201 8);
Baykal et al. (2018): Lee et al. 20088; Wang et al. (2019): Liebenwein et al. (2019); Xing et al.
(020)).
2.3 DISTRIBUTION-AAL PRUNING
We define Distribution- Pruning as the pruning pipelines diminishing the redundancy within neural network according to the distribution of certain property (for instance, it can be the value/norm of weights, the geometric median of filters or the importance scores calculated from the second derivative of loss with respect to weights, etc) in a specific space (e.g., for scalar met- rics like importance score, we can project them into a ID number axis; for vector objects such as geometric median, spatial n-dimensional coordinate might be more suitable). To the best of our knowledge, there are few works aware of/dedicated to this line of research. We just describe some of them obeying our definition here. Xu et al. (2020) argues that the L1 -o-oo of filters in CONV module should obeys Gaussian Distribution, and they cut off the filters with norms distant from distribution's 's center. [Zheng et al.) (2019) and Labach & Valaee (2020) fast narrow the neural network search space according to their assumptions about networks' commection/arciieetuue dis- tribution. Yoshida al.) (2018) adds group LASSO into the training process to drive the network to follow their expected distribution. A statistically principled scoring strategy is proposed by Xing al. (2020) for accurately quantifying association between networks' connections and final output. Chang et al. (2022) studies the distribution of scaling factor of BN layers in Bayesian DNN for pruning without any other information
2.4 THE NOVELTY OF OUR METHOD
The novelty of our work is two-folded. First, our work focuses on the pruning metric search problem, while prior arts focus on building pruning frameworks and leave the pruning metric search as hands- tuned issues. Second, our work delivers a new formalization to guide the determination of the proper pruning metric. which introduces new observation point called Fabulous Coordinate. Third, our work builds a complete solution upon our formalization, which has been quantitatively demonstrated for the effectiveness of our method.
KEY PROPERTIES OF FABULOUS DISTRIBUTION
Enlightened from Liu al. 2018), we find that there is meaningful weight distribution (denoted as Fabulous Distribution) hidden a simple magnitude-basee based pruning process (Han et al.] (2015b)) In this section, we will introduce three key characteristies of the Fabulous Distribution, which are Centralized Distribution, Retraining Recovers Centralization and Central Collapse. We also report preliminary results of our Fabulous Coordinate, the Scaled Filter Norm (SFN), which fit the afore- mentioned characteristics. The details of how to find this coordinate are introduced in Section
$ 'Other works also involve distribution analysis into their pipeline, but the distribution itself is not the pro- tagonist. For instance, Ramanujan et al. (2020) tests the influence of different weight initialization schemes to their Strong Lottery Ticket Hypothesis verification; Qian & Klabjan (2021) presupposes some restrictions to parameter distribution for the convenience of their theory proof. Therefore, we consider these works don't t helong to distribution-hause pruning.
3

---

Under review as a conference paper at ICLR 2023
3.1
CENTRALIZED DISTRIBUTION
700
000
300
000
400
500
300
200
200
160
100
200
1400
Counting
4000
3300
2000
3000
2000
1540
1000
1000
540
200
-0.01
5:00
9.32
-0.55-0.01 0.00
B-02
-0.05
0.00
0:02
-0.01
6.01 9:22
13000
10000
400
300-
300
6000
4000
2000
2000
C40
-8 000
0.009
6.992
02:004
202
5.000
5.002
5.000
2100
0.010
0.,00
07700
8.000 0.002
0.004
Weights
Figure 1: Centralized weight distribution for CONV layers during the prune-retrain-prune process: We select three CONV from the shallow/middle/deep layers of ResNet-50 to observe the variance of weight distribution during the pruning. Pruning pipeline and experimental settings are the same what's described in Han et al. (2115bis work.
Observation: Figure reports the distribution of weights in the form Laplace Distribution C(u, b) for each CONV layers during the prune-retrain-prums process (except the tail end of pruning when central collapse happens. which we discuss in Section 3.3 We name this phenomenon the central- ization of statistics and view it as an important symbol of the overparameterized neural network. As discussed in [iu al. (2018) and Zhong al. (2020), this dense-in-center distribution indicates the hidden redundancy in neural networks and the objects (weights as in Liu et al. (2018), neuron as in Zhong et (2020) located the center might be able to represent each others.
Formal Description: For a formal description of this phenomenon, we need to first measure the de- gree of centralization quantitatively. Therefore, we introduce the metric, Prose-KL-Centraliin Degree (LKL-CD), named after Kullback-Leibler divergence, for an accurate measure of statisties' density at the center of distribution but not restricting the distribution to be Laplace- Distribution LKL-CD of distribution ((z) can be calculated from the below formula:
LKL - CD)(()) min (
(
p(z) log( ((x) dz) (a> 0, B 0)
(x)
where,
if >
Ka
r+3
if <
Then, the description of centralized distribution can be formally written as: We call distribution "ccntralized distribution", if it's generated from a coordinate v and has the following property:
LKL LKL-CD
We quantitatively measure the centralization-dest deeree of three representative coordinates (L, -norm
Li et al. 2016)), geometric median (GM) (He et al. 2018b)) and GraSP Wang al. (2020)) )) but
'GraSP is originally designed for the importance evaluation of each weight. We get the importance score of certain filter under GraSP by gathering the per- per-weight scores using J.y-oom.
4

---

Under review as a conference paper at ICLR 2023
find none of them receive satisfactory LKL-CD score. As shown in the Table L1 norm gains the highest score under our LKL-CD measurement. However, there is still gap between its score and the score of Fabulous Distribution. In Figure 2 the distribution calculated by measuring L1 -nor appears to have multiple centers which greatly differs from our expectation. Therefore, all these
three coordinates don't fit requirement and novel coordinate is required.
Arch/LKL-CD
Raw Weight
L--norm
GM
GraSP
Ours
Layerl
0.4
2.3
3.4
2.5
1.7
Layer2
0.7
1.7
4.0
1.6
1.0
Layer3
0.3
2.1
2.9
1.7
1.0
Layer4
0.5
2.4
2.8
2.8
0.9
Avg.
0.475
2.125
3,275
2.15
1.15
Table I: Centralization degree comparison: We measure the LKL-CD values of each layer in ResNet-50 model with pre-trained weights from torchvision (PyTorch).
conva (L1-norm)
layert. O.conv3 (L1-norm)
1ayer2.4.comcz (L1-norm)
12
Counting
0.000
conv1
layer1.0.conv3 (FSN)
Inyar34.conv2 (FSN)
50
15-
10
10
-0.05
0.00
0.05
0.10
0,00 0..11
0.00
0.01
0.02
0.05
-0,005 0.000
0.005
0.010
Weights
Figure 2: Distribution comparison: We compare the distributions measured by L1 norm (upper) and SFN (lower) on the same model used in Table 1
Our Coordinate: To address the aforementioned issue, new coordinate, Scaled Filter Norm (SFN), is found using Crossword Puzzle search engine, to get the centralized distribution. We express it in Equation
|| 1[[Cwl
SPN(FW)
where, Fw is the weight of certain filter in CONV layer with weight Cw; and a are the scaling factor and bias applied to that filter the batch normalization layer.
As shown in Table and Figure 2 it not only outperforms three other coordinates in LKL-CD comparison, but also are consistent through all CONV layers. After the coordinate transfer, previous multi-center distribution becomes single-center, which demonstrates the high central density expect. Therefore, we choose SFN as the importance scoring mechanism our pruning pipeline.
3.2 RETRAINING RECOVERS CENTRALIZED DISTRIBUTION
The empty space created by pruning (through cutting off the central parts in the distribution) can be re-filled by retraining, which is similar to regather data at the center. Figure B demonstrates this phenomenon in retraining step of pruned ResNet-50 Figure 3 the remaining weights distributed in the two sides migrate to the central gradually and forms new summit in the frequency histogram. Another interesting observation is that the new-
Observation:
CIFAR-10. From
5

---

Under review as a conference paper at ICLR 2023
created summit has lower height than that of the summit before pruming. This might be explained as the consequence of redundancy decrease caused by pruning replaceable weights.
3..
Distribution recovery in several prune-retrain-prune steps: This graph depicts the LKL,-CD variance during four prune-retrain- prune iterations; The per-iteration pruning rate is set to 0.2, and the retraining process goes through 40 epochs CIFAR-I0. "Base" refers to Han et al. 2015b) s method.
Figure 4:
Figure 3: Distribution recovery in one retrain- ing step: We prune pre-trained ResNet-50 and then retrain it on CIFAR-10; EpochO is the ini- tial state of the retraining: Epoch30 means that we' ve trained the pruned model for 30 epochs.
Our Coordinate: As can be seen in the Figure 4 prune-retrain retraii--rin- prune loop also experiences a similar centralization recovery process. However, compared Han et al. (2015b)'s s work, the degradation of the centralization is apparent for our pruning. One possible explanation is that the discrepancy between our and Han et al. (2015))'s observation coordinates prevent us to obtain an exact same distribution, and this gap is enlarged during the mune-retrain--oe prunne process.
3.3 CENTRAL COLLAPSE
Observation: At certain stage of prune-retrain-prumg loop, retraining is unable to fill the empty space in distribution, and the original center will be filled with void (we call this phenomenon "Central Collapse"), accompanying by dramatic accuracy decline. We also find that the central collapse exists the weight distribution the final pruning output if follow Deep Compression's pruning pipeline Han et al. (2015a) which is the same the experimental results described in Han jet al. (2015))ss s and Liu et al. (2018)'s work. One additional discovery that there strong association between central collapse and accuracy degradation. In Figure 5) central collapse and the turning point of accuracy accuracy-pruming- rate-lirteennne nearly occur at the same time. We believe this is not coincidence, since the central collapse possibly also indicates close-to-zero redundancy in the current layer. Therefore, further pruning on this "already succinct" layer will greatly hurt the accuracy.
Our Coordinate: We also encounter the central collapse and the experimental evidence are depicted in Figure 5 After pruning 50% weights out of the model, the central collapse becomes noticeable in some layers and our model accuracy goes down simultaneously. For a closer observation, we provide break-down FSN distribution of representative CONV layers in Figure 6 From Figure most layers suffer irreversible damages and fall into distribution with central collapse.
4 CROSSWORD PUZZLE SEARCH ENGINE
In this section, we introduce the pipeline of our Crossword Puzzle search engine, which takes pre-trained neural network and pruning metric building block as inputs and outputs Fabulous Co- ordinate.
Neighbor Coordinate Generator: First, we generate coordinate according to the input pruning metrics, by taking the linear combination of several intuitively selected criteria. For instance, this generator might produce coordinate, V (FFw) | ll + F[ l|2. where Fw is the weight of filters. In practice, we classify these building blocks according to bagnittue-iimpaat-/diieeee based pruning and begin searching within these clusters first, metrics derived from methods in

---

Under review as a conference paper at ICLR 2023
#Certcal Culiante CON Han
Man FORd
11.4
00.0
23 Rate
FEN
Figure 5: Top-1 Accuracy v.s. Pruning Rate: We measure the accuracy variance with re- gard to pruning rate on ResNet-50 (CIFAR-100 "Han" refers to Han et al. (2015b)ss work.
Figure 6: The FSN distribution of representa- tive CONV layers of Resnet-50 (Pruned and re- trained on CIFAR-10). Similar characteristics are observed other layers.
Retrospect
Feedbback
fall
Seirching Guider
IF fall
No Goordinate System Generntor
Retrinn
Pruns
Rotrainn
One possible Fabulous Coordinate System
LKL-CD
Basic Pruning Metrica (Bullding Blocks)
Repeat times
Repest doean't appear, quit
Figure 7: Crossword Puzzle Search Engine: Input: 1) building blocks of pruning metrics (e.g., the L -noom of filter's weights) and 2) the pre-trained neural network (not shown in this figure); Output: pruning metric on which Fabulous Distribution can be observed.
same cluster are expected to have strong association. Note that during searching, our crossword puzzle engine don't 't need strictly obey this intuitive rule since it can jump out of when triggered by the simulated annealing search guider.
We add a simulated annealing (Kirkpatrick et al. (1883) search guider to steer the neighbor coordinate generator. The possibility of accepting coordinate is expressed as -u/tt, where t is the temperature and v is the LKL-CD value. From the equation, we can see that, a coordinate with high LKL-CD value still can be accepted as the start point of the next neighbor search. In practice, this annealing approach enables the engine to expand search space and find hybrid solutions.
Searching Guider:
Centralization Degree Filter: We set filter to halt coordinates with high LKL-CD values from continued searching. According to the coordinate, the filter can calculate the average LKL-CD value of the whole model. If the LKL-CD value exceeds the threshold A (a preset constant), this coordinate and inform the searching guider, since the high LKL-CD value directly violates our definition for Fabulous Coordinate. Experimental results show that this filter can filter out more than 80% coordinates, speeding up the searching process greatly.
drop
The second stage is to conduct some prume-retrain-ee rerunii---u- trials for the verification of the retraining recovery ability of a certain coordinate. In practice, we use the following equation measure this ability:
Retraining Recovery Verification:
7

---

Under review as a conference paper at ICLR 2023
where, Recovery_Abllity-- = A ( + AAc.TOP11
is the average LKL-CD value of 10 prune-train-pri prrnnn iterations, each with 20 epochs' retraining; TOP_ is the top-1 accuracy improvement during this pro- cedure; A and B are two scaling constants.
Central Collapse Inspector: Finally, we test the existence of central collapse and output this coor- dinate as a possible Fabulous Coordinate if it passes all the tests. To track the occurrence central collapse, closely monitor the variance of LKL-CD during the mrune-retrain-rrun process. The central collapse happens when an increment of more than 1.0 can be observed on LKL-CD,,
EXPERIMENTAL RESULTS
We
evaluate pruning method with SFN coordinate for two popular convolutional neural networks VGG-16-bn (Simonyan & Zisserman (2014)) and ResNee-550/-66-111/111 (He et al. (2013) tested on CIFAR-I0 (Krizhevsky) (2009) and ImageNet (Deng al. (2009)) with details of experimental settings and results displayed below.
5.1 EXPERIMENT SETTINGS
In this section, we elaborate the pruning and retraining settings, For pruning, we utilize PyTorch (Paszke et al. (2019)) and one third-party library Torch Pruning (Fang (022)). At the beginming of each prune-retrain- prune iteration, we cut off 10% filters in each CONV layer, which results in about 30% weights being pruned off in the whole model, due to the connection between CONV layers (more details in Section A). For retraining, we inherit the same training settings and functions from two standard examples (Phan (2022): Py TyTocc (2022)), except that the training epoch is adjusted to 100/120 on
to achieve a fair competition under the same computation budget (as suggested by Liu et al. (2018). All experiments are conducted machine with NVIDIA RTX A5000 and Intel Xeon Gold 6330.
5.2 RESULTS ON CIFAR-10
Our auto-geut irto-generated pruning pipeline achieve comparable pruning results on CIFAR-10 10 with regard to other human-buil built methods (such as PFEC (Li et al. (2016)). MIL (Dong et al. (2017b))
CP (He et al. ()17)). SFP (He al. (2018))), FPGM (He et al. 2018b)), PFGDF (Xu et 2020
FPPMO (Li et al. (2021)) RL-MCTS (Wang& & Li )022
As shown in Table 2 method reduce the FLOPs of ResNet-56/110 by 73.0% with slightly accuracy improvement, while, on VGG-16-bn, we can't defeat the ttateoof-the-at human- created pruning frameworks, PFGDF and RL-MCTS. The success on ResNet might be due to the specific pruning strategy (in Section A) we apply to shortcut connections. This strategy takes the CONV layers associated by shortcut connections as a whole, and, therefore, reveals useful global information to our pruning, which assists us to prune more accurately. However, VGG-16 doesn't 't possess such structure, and the limited information impair the effect our pruning (by default, our pruning is performed solely on each CONV layer). This might demonstrates that our proposed pruning pipeline (in Section A can be further enhanced by considering the global environment. But the focus of this work is not any dedicated pipeline, we just use simple one.
5.3 RESULTS ON IMAGENET
ImageNet, our pruning framework can also compete with methods fine-tuned by human (for instance, ThiNet (Luo et al. (2017)). SFP (He al. (201 8a)). Rethinking (Ye et al. (2018)). FPGM (He et al. (2018b)), FPPMO eL al. 2021)). RL-MCTS (Wang & Li (2022)).
Table 3 demonstrates that method can outperform the state-of-the-ard methods with more ac- curacy being preserved, given similar FLOPs drop. We owe this positive outcome to the strength of Fabulous Distribution. When the data amount goes up, the Fabulous Distribution reveal the redundancy more accurately, since the hidden characteristics within statistics are more likely to be discovered when there are numerous samples.
8

---

Under review as a conference paper at ICLR 2023
Baseline Top-1
Top-1 Acc
Top-1 Ace
AFLOPs
Model
Method
after
(%)
(%)
Acc (%)
Pruning (%)
PFEC CP
93.04
93.06
+0.02
-27.6
92.80
91.80
-1.00
-50.0
SFP (wlo pre-trained)
93.59
92.26
-1.33
-52.6
FPGM (GM-only 40%)
93.59
93.49
-0.10
-52.6
ResNet-56
FPPMO (45%)
93.59
93.50
-0.09
-58.0
RL-MCTS
93.20
93.56
+0.36
-5,.0
93.9
+0.25
-28.6
Ours (SNF)
93.65
93.66
+0.01
-73.0
MIL (w/o pre-trained)
93.63
93.44
-0.19
-34.2
PFEC
93.53
93.30
-0.23
33.66
SFP (w/o pre-trained)
93.68
93.38
-0.30
-40.8
ResNet-110
FPGM (mix 40%)
93.68
93.85
+0.17
-52.3
FPPMO (45%)
93.68
93.76
+0.08
-5..7
93.97
+0.29
-48.7
Ours (SNF)
93.68
93.74
+0,06
-63.0
PFEC
93.58
93.31
-0.27
-34.2
FPGM
93.58
93.54
-0.04
-34.2
VGG-16-bn
PFGDF
93.25
93.48
+0.23
-70.27
RL-MCTS
93.51
93.90
+0.39
-45.5
Ours (SNF)
94.00
94.29
+0.29
-53.1
Table 2: Pruning results on CIFAR-10: "pre-trained" means whether use pre-trained model as the initialization or not. Other labels like "GM-only" keep the same meanings as those in original papers. Our competitors' data are collected from their released codes/model/best results on paper.
Baseline
Top-1/5
Top-1155 Acc
AFLOPs (%)
Model
Method
Top-1/5
Acc after
Acc (%)
Pruning (%)
(%)
ThiNet
72.88/91.14 1.14
72.04/90.67
-0.84/-0.47
-36.7
SFP
76.15/92.87
62.14/84.60
-4..01/-827
-41.8
FPGM (GM-only 30%)
76.15/92.87
75.59/92.63
-0.56/-0.24
-42.2
ResNet-50
FPPMO (40%)
76.15/92.87
74.91/92.39
-1.24/-0.48
-53.5
RL-MCTS
77.34/93.27 27
76.80/93.00
0.54/-0.27
-46.1
Ours (SNF)
76.13/92.86
75.65/92.64
-0.48/-0.22
-49.3
Rethinking
77.37/93.56
75.271-
-2.10/-
-47.0
FPGM (GM-only 30%)
77.37/93.56
77.32/93.56
-0.05/0.00
-42.2
ResNet-101
FPPMO (35%)
77.37/93.56
77.18/93.52
-0.05/0.00
-50.2
Ours (SNF)
77.37/93.55
77.54/93.68
+0.17/40.13
-4..5
Table 3: Pruning results on ImageNet: labels have the same meanings as those Table competitors' data collected from their released codes/model/best results paper.
Our
CONCLUSIONS
We propose The Cross Puzzle, a new method to simplify this procedure by automatically deriving pruning metrics. The key insight behind our method is that: For Deep Neural Network Models, a Pruning-friendly Distribution of model's weights can be obtained, given a proper Coordinate. We experimentally confirm the above insight, and denote the new Coordinate the Fabulous Co- ordinates. Our quantitative evaluation results show that: the Crossword Puzzle can find a simple yet effective metric, which outperforms the ofate-of-the-se pruning methods by delivering curacy degradation on ResNet-56 (CIFAR-10y-101 (ImageNet), while the pruning rate is raised to 70%/550 for the respective models.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Cenk Baykal, Lucas Liebenwein, Igor Gilitschenski, Dan Feldman, and Daniela Rus. Data- dependent coresets for compressing neural networks with applications to generalization bounds, 2018. URLBtttps:/arxiv org/ abs/ 12///88444 804. 05345
Wan-Ting Chang, Chih-Hung Kuo, and Li-Chun Fang- Variational channel distribution pruning and mixed-precision quantization for neural network model compression. In 2022 International Symposium on VLSI Design, Automation and Test (VLSI-DAT), pp. 1-3, 2022. doi: 10.1
ASSL-DATT44769.2222.77666
Daiki Chijiwa, Shin'y ya Yamaguchi, Yasutoshi Ida, Kenji Umakoshi, and T. Inoue. Pruning randomly initialized neural networks with iterative randomization. In NeurlPS, 2021a.
Daiki Chijiwa, Shin ya Yamaguchi, Yasutoshi Ida, Kenji Umakoshi, and Tomohiro Inoue. Prun- ing randomly initialized neural networks with iterative randomization, 2021b. URL https:
/ //aaxiivcrgg/aaaaeeee.eaa bbrrg//bb///0 /2106.1166... 09269
Arthur da Cunha, Emanuele Natale, and Laurent Viennot. Proving the Strong Lottery Ticket Hypothesis for Convolutional Neural Networks. In ICLR 2022 10th International Con- ference on Learning Representations, Virtual, France, April 2022. URL https://hal.
archives-ouverten (twp/ttees..t/aa-0055555550
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Xin Dong. Shangyu Chen, and Sinno Jialin Pan. Learning to prune deep neural networks via layer- wisc optimal brain surgeon, 2017a. URL
org/ abs 705. 07565
Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan. More less: more complicated network with less inference complexity, 2017b. URL
08651
Marco Dorigo and Gianni Di Caro. The Ant Colony Optimization Meta-Heuristic, pp- 11-32. McGraw-Hill Ltd., UK, GBR, 1999. ISBN 0077095065.
Bryn Elesedy, Varun Kanade, and Yee Teh. Lottery tickets linear models: An analysis of iterative magnitude pruning. 07 2020.
Gongfan Fang. Torch-Pruning
Torch- Preninru 2022.
URL bttp::///itthb..ccm/aaaaa
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neu- ral networks. In ICLR. OpenReview.net, 2019. URL http:://blpp wnw.trier.ddedeeee..
conf/icl r201 9. htm #Frank
Mitchell Gordon, Kevin Duh, and Nicholas Andrews. Compressing bert: Studying the effects of weight pruning transfer learning. 02 2020.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. 2015a. URL /ttp::/aaxxxiii c abs/1510. 00149 cite 11.111110.0049mmmmm Published conference paper at ICLR 2016 (oral).
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume NIPS' 15, pp. 1135-1143, Cambridge, MA. USA, 2015b. MIT Press.
B. Hassibi, D.G. Stork, and G.J. Wolff. Optimal brain surgeon and general network pruning. In IEEE International Conference on Neural Networks, pp. 293-299 vol.1, 1993. doi: 10.1109/
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog nition, 2015. /tL.pttpps///xx..ooooo/o/// wttpv/o.rrrrb///112/2//// 03385
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for acceler- ating deep convolutional neural networks, 2018a. 06866
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration, 2018b. URL
abs/
1811100255
Yihui He, Xiangyu Zhang, and Jian Sun. Chanmel pruning for accelerating very deep neural net- works, 2017. /ttttp::/arii..og/////aa/a
John Hertz, Anders Krogh, and Richard G. Palmer. Introduction to the Theory of Neural Computa- fion. Addison-Wesle, Longman Publishing Co., Inc., USA, 1991. ISBN 0201503956.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks, 2016. URL httpssts:://axxx
06993
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks, 2017. http:/ttss///rxxvvoo/r///a/////
S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi. Optimization by simulated annealing. SCIENCE, 220(4598):671-6800 1983.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Alex Labach and Shahrokh Valace. A framework for neural network pruning using gibbs distri- butions. In GLOBECOM 2020 2020 IEEE Global Communications Conference. IEEE, dec 2020. doi: 10. 11.11099loheeom/2222000000222029
URL htttt::////d org/ 0ttp//1.1119991 1109%
(rglobbecm42200 20200002.22202222222 9322333
Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wise brain damage, 2015. URL
https: http:/wwww.rr///////eeee///
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touret- zky (ed.). Advances Neural Information Processing Systems, volume 2. Morgan-
Kaufmann, 1989. URL
www.eeeripp. wwtppccppppppppeeeeee (1989////11/ pdd
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Snip: Single-shot network pruning based on connection sensitivity. 2018. URL /atttps:/aaxxiv.rtt org/ abs/1810. 02340
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and H.P. Graf. Pruning filters for efficient convnets. 08 2016.
Qinghua Li, Cuiping Li, and Hong Chen. Filter pruning via probabilistic model-based optimiza- tion for accelerating deep convolutional neural networks. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, WSDM 221, pp. 653-661, New York, NY, USA. 2021. Association for Computing Machinery. ISBN 9781450382977. doi:
00.1145/3433963.34417283 URL httpt:ttt////ctt. /ttp:::///1...////10000 1145/ 3355/333755 343793333... 3441766
Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter prun- ing for efficient neural networks, 2019.
a/arxi v. org abs/ 911 07412
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. arXiv preprint 10X.:2106611404 2021.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming, 2017. URL hetps:
//arxiv..or//ab//1700 065111100666653
11

---

Under review as a conference paper at ICLR 2023
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning, 2018. URL/attps://aaxiii
http/abs/111111 05270
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural org/ /as///b////// 1707 .06442
network compression, 2017.
Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 6682-6691. PMLR, 13-18 Jul 2020. URL htttpa://ppooto /tttpp://proceedinggs. mlr. wwwps//119/maaaeeissreaa
Jaachen Mao, Huanrui Yang. Ang Li, Hai Li, and Yiran Chen. Tprune: Efficient transformer pruning for mobile devices. ACM Trans. Cyber-Phys. Syst., 5(3), apr 2021. ISSN 2378-962X. doi: 10. 10.1145/3446640, Utthttss: http://c.....o..///////o
Melanie Mitchell. An Introduction to Genetic Algorithms. MIT Press, Cambridge, MA, USA, 1998. ISBN 0262631857.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference, 2016. URL
1611.06440
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. pp. 11256-11264, 06 2019. doi: 10.110//CVP.2209900152222
Laurent Orseau, Mareus Hutter, and Omar Rivasplata. Logarithmic pruning is all you need. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 2925-2934. Curran Asso- ciates, Inc., 2020. URL Ittpss://proceedingg. wwwpe:edddiin.....neeeeeeeeeeeee newrips.r.pp....... wtwp://s.cc/pppe//2//EELL
Papeeee pdf
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL
9015 5- pytorch ar wwa.rimmeen imperatieeeeeeeeee style high perg-peeforrance dewpeaaaeeeeeeeee-eeeeeeeeeeeeee learning library..
pdf
Ankit Pensia, Shashank Rajput, Alliot Nagle, Harit Vishwakarma, and Dimitris Papailiopoulos. Op- timal lottery tickets via subsetsum: Logarithmic over- parameterization is sufficient. In Proceed- ings of the 34th International Conference on Neural Information Processing Systems, NIPS'20 20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Huy Phan. CIFARh-CIFAR-10, 7 2022.
URL https://ggthabb.cmm/ayyyppp
PyTorch_CCoe CIFARI07
PyTorch. PyTorch: MODELS AND PRE-TRAINED WEIGHTS.
https://pytorch..zz
wiision/staale/modeell html Accessed: 2022-09-21.
PyTorch. PyTncch--xaamlle-ImgeeNNttt 7 2022.
URI. httpp:///github..cmm/pttttt
examp cetamplea/tree wwteee//eeeeeeeeeieeeeeeeeeeeeee
Xin- Yao Qian and Diego Klabjan. A probabilistic approach to neural network pruning. In ICML, 2021.
V. Ramanujan, M. Wortsman, A. Kembhavi, A. Farhadi, and M. Rastegari. What's hidden in a randomly weighted neural network? In 2020 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 11890-1 1899, Los Alamitos, CA, USA, jun 2020. IEEE Computer Society. doi: 10.1100CCVP44260.222000099999999
URL https:://ooi.
01191
12

---

Under review as a conference paper at ICLR 2023
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2014. URL
1
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural 14340
network compression, 2020. /tL/tttpss//axxxi.orrrrr///////
Hidenori Tanaka, Daniel Kunin, Daniel Yamins, and Surya Ganguli. Pruning neural networks with-
out any data by iteratively conserving synaptic flow. 06 2020.
Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. Eigendamage: Structured prun- ing in the cromecker-factored cigenbasis, 2019. URL https: /
05934
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by
preserving gradient flow. 2020. doi: 1048550/ARXI..0002077766 URL https://arxit
07376
Zi Wang and Chengcheng Li. Channel pruning via lookahead search guided reinforcement learning- In 2022 IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pp. 3513- 3524, 2022. doi: 10.1 10///AAC51148822222.0000
Xin Xing, Long Sha, Pengyu Hong, Zuofeng Shang, and Jun S. Liu. Probabilistic connection im- portance inference and lossless compression of deep neural networks. In ICLR, 2020.
Jianrong Xu, Boyu Diao, Bifeng Cui, Kang Yang, Chao Li, and Yongjun Xu. Přgdf: Pruning filter via gaussian distribution feature for deep neural networks acceleration. 2020. doi: 10.48550/
ARXIV.2006.1299 129633. URL https: arxiv. orggabs// httg://bb2200...... 12963
Jianbo Ye, Xin Lu, Zhe Lin, and James Z. Wang. Rethinking the smallernoom-les--ifommaat assumption in channel pruning of convolution layers, 2018. URL https://arxiv.ort
abs/1802. 00124
Tsukasa Yoshida, Takafumi Moriya, Kazuho Watanabe, Yusuke Shinohara, Yoshikazu Yamaguchi, and Yushi Aono. Automatic dnn node pruning using mixture distribution-baseed group regulariza- tion. In INTERSPEECH, 2018.
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I. Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S. Davis. Nisp: Pruning networks using neuron importance score propagation, 2017. URL htt.t:tpp//// /tttps///arivee arxivv arggabbs//11..... 05908
Wenyuan Zeng and Raquel Urtasun. Mlprune: Multi-layer pruning for automated neural network compression. 2018
Xiawu Zheng, Rongrong Ji, Lang Tang, Yan Wan, Baochang Zhang, Yongjian Wu, Yunsheng Wu, and Ling Shao. Dynamic distribution pruning for efficient network architecture search, 2019.
URL https::/acxiv..nr
abs/1905.13543
Guoqiang Zhong, Wenxue Liu, Hui Yao, Tao Li, Jinxuan Sun, and Xiang Liu. Merging similar neurons for deep networks compression. Cognitive Computation, 12:577-588, 2020.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bun -ucce E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32, Cur- Associates, Inc., 2019. URL http::tttss.. htt.t:::ppproceednngee wwt.gssnuuripsccom ret.rips..cc/pppee/2019999 655
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX: THE PROPOSED PRUNING PIPELINE
Like many previous practices (Han et al. 2015a b Li et al. 2016): Frankle & Carbin (2019);
Chijiwa et al. (202Ib), we also prune the neural network iteratively (as shown in Algorithm However, the pruning step is slightly different from that of magnitude- based pruning frame- works, since we prune redundant parts according distribution analysis while handling shortcut connections specially.
Algorithm Iterative Pruning
Input: pre-trained model with weights W; pruning rate PR; prune times PT; retrain epochs Ep Output: pruned model weights W
1: function ITERATIVE PRUNING(W, PR, PT, .T.Epp Ep)
2: for i < to PT do
3:
D < Snalyse-Distrifff
4:
M < Prune Cenner(D,PP)
>> M is the mask > prune model according mask b retrain model for Ep epochs
5: W W о M
6:
Train(W. Ep)
7: end for
8: return W
9: end function
Pruning According to Distribution: We clip off the central parts in the distribution, which can be interpreted as the generalization of traditional magnitude- prggituue-bbsed pruning. Some classic magnitude- based pruning practices (such as Han et al. al.) (2115); Liu et al. (2017)) hold the belief, smaller-value less-important. However, we owe the success of these works to the coincidence that small values happen to lie in the center of distribution. For instance, in Figure the center of distribution overlaps the original point in the horizontal axis. The pruning of small values also takes the effeet of cutting down the peak of distribution. Based these observations, we claim that what's matter most is the center of distribution rather than the small values. Some experiments are conducted to verify this conjecture. As can be seen in Figure 8, pruning random/smallest/large http://malless/lggs/ttwsssssisie distribution doesn't t result a better outcome than pruning the center parts. This might point to the fact that the center of distribution has special meaning.
values in the
serpout
BOTTI CLEEEECKK
Miveltted www.raaaaiaaeeeea....
1op-
Dkunrlaation
Generrate Pruninn Mask
Figure 8: The results of pruning hawammalllee/neeeeet//ossseeeeee values in the distribution: Data are collected from a ResNet-50 model pruned using Algo- rithm I retrained on CIFAR-10.
Figure Prune CONV related to shortcut con- nections in ResNet-50: Last CONV layer of the bottleneck module and the downsample CONV layer associated together through the short- cut connections. When pruning one them, the others need to be adjusted accordingly.
How to Handle Shorteut Connection: Our pruning framework prune the CONV layers around shortcut connections, an essential component in many advanced CNN models like ResNet He et al. 2015) and DenseNet Huang et al. (0166), by taking the computation association into consideration.
14

---

Under review as a conference paper at ICLR 2023
In Figure 9 we illustrate how prune the CONV layers in ResNet. First, analyse the dependency of CONV layers to pick out the ones which should be pruned together. Then, by treating these layers whole, we can get the distribution and corresponding pruning mask. Finally, the mask is applied to these layers and the pruning can proceed to other layers unrelated to shortcut connections.
15


Reviews:
[{"review_id": "y_-LGzhbOfw", "reviewer": "Reviewer_bFZJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a Crossword Puzzle engine for searching the optimal pruning metrics automatically. It first gives three properties of a good metric/coordinate: centralized distribution,  retraining recovers centralized distribution and central collapse. Then, it introduces a metrics called LKL-CD  for measuring the centralization degree. The paper finally presents their coordinate obtained from the engine and empirically evaluates the effectiveness of the coordinate system for pruning.", "summary_of_the_review": "The paper proposes novel perspective of pruning. Intensive experiment is done to search better coordinate systems to guide the pruning procedure. However, some key points need further and clearer explanation.", "strength_and_weaknesses": "Strengths\n\n•\tThe paper gives a comprehensive introduction to previous work on pruning metrics categorized in 3 different groups, magnitude, impact, and distribution-based methods. These build the metrics blocks for search.\n•\tThe paper introduces the Fabulous Coordinate with three properties, a new perspective of understanding the model redundancies.\n\nWeaknesses\n\n•\tThere is little explanation on why the proposed LKL-CD is better than other metrics. The paper attempts to explain it through Table 1, the four layers in ResNet-50. It might not be statistically enough to claim the advantage of it using one type of network. More validation should be done either empirically or theoretically. It would be better to see that comparisons between them are included in the experiment section.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "e9yXrq60bCy", "reviewer": "Reviewer_bVnh", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes the Cross Puzzle to find something called the Fabulous Coordinate for pruning neural networks. ", "summary_of_the_review": "Overall, this paper id not ready for publication.\nThe key idea is not clearly communicated, the terms are not well defined, and there are not experiments that support the key idea. \n", "strength_and_weaknesses": "The paper is very difficult to read. Though experiments are performed, the experiments are not up to the quality expected. For example, there is no direct comparison with The Lottery Ticket Hypothesis paper (https://arxiv.org/pdf/1803.03635.pdf), which can reduce network size by over 90%. \n\nTerms are not clearly defined. I cannot find clear definitions of the Cross Puzzle, the Fabulous Coordinate or the Fabulous Distribution. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Mki5xzfO0p", "reviewer": "Reviewer_TNSn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Existing pruning approaches rely on intuition- or experience-based measures to determine the constraint based on which to prune neural networks. Instead, the current work proposes a framework, Crossword Puzzle, to guide the search of a pruning criterion given basic building blocks and evaluate the quality of the pruning criterion when applied on networks. Specifically, the proposed framework is based on the Fabulous coordinates, which indicate that when the distribution from a provided model match the constraints set for these coordinates, they are highly qualified to provide good pruning outcomes.", "summary_of_the_review": "While the core ideas are relatively straightforward, the evaluation and explanation of certain aspects of the work are obscure. Addressing the weaknesses mentioned above will bring more clarity to the results and content of the proposed work.", "strength_and_weaknesses": "Strengths\n- The proposed work offers an interesting idea on measuring the quality of a proposed pruning metric.\n- Combined with an almost meta-learning-like framework on identifying a high quality pruning metric, automation of this domain is certainly important and brings pruning a step closer to being part of the development cycle of DNNs.\n\nWeaknesses\n- Could the authors provide more quantitative bounds when mentioning \"relatively-high pruning rate\" in Pg. 1, footnote.\n- When highlighting the improvement in acceleration, could the authors clarify whether the \"3x\" improvement is in theoretical FLOPs or real-time inference latency?\n- I would encourage the authors to look closely into the literature discussing distribution-based pruning. Apart from the papers mentioned in the current manuscript, there are broad swathes of work on probabilistic pruning that can and should be referenced.\n- Could the authors provide an intuition for the formulation of the LKL-CD, including how they solve for the final values?\n- Across multiple instances there are mix ups between the terminology of SFN and FSN (E.g., Fig. 2, Fig. 3 and others). Please correct them.\n- The LKL-CD of Raw Weights in Table 1 provide a much better approximation than the chosen SFN formulation, assuming lower values are better. Could the authors justify the selection of SFN?\n- Could the authors clarify the meaning of the notation $C_W$?\n- I encourage the authors to increase the font of legends and Axes ticks for readability.\n- Quantifiably, could the authors provide metrics used to measure the collapse as well as highlight how exactly they were applied to measure the values provided in Fig. 5?\n- In addition, could the authors provide more comparisons on the recovery process of the central hole across the various conv layers?\n- From Tables 2 and 3,\n     - Across VGG16 and Table 3, SFN provides better accuracy at a lower % of FLOPs removed when compared to previous approaches.\n     - Could the authors clarify their choice in prior art and comparisons? Since there are a number of previous works with higher performances that haven't been included.\n     - In addition, could the authors include the decrease in number of parameters as well, since it provides a slightly different perspective on the degree to which a network can be pruned.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Da78YfRQB-Z", "reviewer": "Reviewer_uiqw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method called  \"The Cross(word) Puzzle\" that helps in getting better pruning metrics to further improve pruned neural networks in accuracy for the same compute footprint. This is informed by what the authors call Fabulous Coordinates (the choice of pruning criteria) and Fabulous distribution (the target distribution of weights after pruning). \n\nThe paper starts of by motivating what the useful properties of a good pruning function should look like and leverages it to build a search pipeline for semi-automating it. They evaluate the method with a couple of experiments on ImageNet and CIFAR-10\n\n\nThe brevity of the review is due to the fact that the paper is extremely incoherent and it took me over 5 hrs to digest the information. The paper is extremely poorly written, with no solid definitions, inconsistent naming (SFN vs FSN vs SNF -- what is this?), poor naming (Fabulous?), lack of details of the methods, no signs of reproducibility, and inconsistent experimentation and design motivations. \n\n\n", "summary_of_the_review": "Poor presentation and quality. While novelty exists in the motivation, it is executed in an unscientific fashion. It seems like a paper thrown all together at the last moment and I do not recommend publishing of the paper even after significant registrations.\n\n--------------------\nPost Rebuttal: Given the need for a significant revision, I recommend rejecting the paper from ICLR 2023 but encourage the authors to make the material more accessible to readers in the next iteration.", "strength_and_weaknesses": "Strength:\n1) The problem paper tries to solve is real and has real-world utility.\n\nWeakness:\n1) The abstract is extremely uninformative, so is the introduction.\n2) There are terms used without any explanation or pointers to -- what is a coordinate? is something it too me a long to process\n3) What is fabulous about the distribution?\n4) The motivation for the design principles come from unstructured pruning and the paper experiments on structured pruning, I am not sure how that works. I know the connections between both styles exist, but connecting Han et al 2015 to the current paper does not gel well. \n5) I have no idea how the candidates for coordinates are even generated for search, the entire pipeline is so under-explained at required places and over-exposited at places with little impact on the entire method. \n6) Table 1 is extremely hard to comprehend. \n7) definitions of equations follow the same suit.\n8) Figure 5 shows your method is more inaccurate than Han et al., I do not understand. \n9) The experiments are performed on limited evaluations and at 50% structured sparsity which is not even considered to be sparse because of the known redundancies. The usual comparisons happen at over 70% for ImageNet. \n10) FLOPS is not directly proportional to sparsity due to non-uniformity across layers, so that metric needs to be fixed. \n11) While the figure are slightly accessible, they are difficult to read due to poor placement and design. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0r_S4F-16Eg", "reviewer": "Reviewer_jFro", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a so-called Crossword Puzzle method to find the optimal criteria for network pruning. The key idea is to find a so-called Fabulous Coordinate which satisfies three key properties for pruning. The authors validated their method on ImageNet and show that they can compress ResNet family by 50% without accuracy degradation.", "summary_of_the_review": "Strong reject.", "strength_and_weaknesses": "[Weakness]\n\nThe paper is poorly written. It looks like the paper is generated by some Google translation engine with lots of meaningless words. I tried my best but honestly speaking, I cannot understnad what the authors are writing about. In Section 3, the authors throwed up lots of formulars and figures without sufficient explanation. I have no clew what these numbers or equations mean.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject