Towards Performance-maximizing Network Pruning via Global Channel Attention
Authors: Yingchun Wang
Linchuan Xu
Song Guo
Jingcai Guo
Weizhan Zhang
Jie Zhang
shuai chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Channle Pruning
Global Attention
Deep Neural Networks
Model Compression

Abstract: Network pruning has attracted increasing attention recently for its capability of transferring large-scale neural networks (e.g., CNNs) into resource-constrained devices. Such a transfer is typically achieved by removing redundant network parameters while retaining its generalization performance in a static or dynamic pruning manner. Concretely, static pruning usually maintains a larger and fit-to-all (samples) compressed network by removing the same channels for all samples, while dynamic pruning can adaptively remove (more) different channels for different samples and obtain state-of-the-art performance along with a higher compression ratio. However, since the system has to preserve the complete network information for sample-specific pruning, dynamic pruning methods are usually not memory-efficient. In this paper, our interest is to explore a static alternative, dubbed GlobalPru, to conventional static pruning methods that can take into account both compression ratio and model performance maximization. Specifically, a novel channel attention-based learn-to-rank algorithm is proposed to learn the global channel attention of the network for various samples, wherein, each sample-specific channel saliency is forced to reach an agreement on the global ranking. Hence, all samples can empirically share the same pruning priority of channels to achieve channel pruning with minimal performance loss. Extensive experiments demonstrate that the proposed GlobalPru can achieve better performance than state-of-the-art static and dynamic pruning methods by significant margins.

Under review as a conference paper at ICLR 2023
TOWARDS PERFORMANCE-MAXXIMIIOO NETWORK
PRUNING VIA GLOBAL CHANNEL ATTENTION
Anonymous authors Paper under double-blind review
ABSTRACT
Network pruning has attracted increasing attention recently for its capability of significantly reducing the computational complexity of large-scale neural networks while retaining the high performance of referenced deep models. Compared static pruning removing the same network redundancy for all samples, dynamic pruning could determine and eliminate model redundancy adaptively and obtain different sub-networks for each input that achieve state-of-the-art performance with higher compression ratio. However, since the system has to preserve the complete network information for running-time pruning, dynamic pruning methods are usually not memory memory-efficient efficient. In this paper, our interest to explore a static alternative, dubbed GlobalPru, to conventional static pruning methods that can take into account both compression ratio and model performance maximization. Specifically, we propose novel channel attention-based learn-torrank algorithm to learn the optimal consistent (global) channel attention prior among all sample-specific (local) channel saliencies, based on which Bayesian-based regularization forces each sample- specific channel saliency to reach an agreement on the global channel ranking simultaneously with model training. Hence, all samples can empirically share the same pruning priority of channels to achieve channel pruning with minimal performance loss. Extensive experiments demonstrate that the proposed GlobalPru can achieve better performance than state-of-the-art static and dynamic pruning methods by significant margins.
INTRODUCTION
Convolutional neural networks (CNNs) have achieved great success in many visual recognition tasks including image classification He al. (2016), object detection Ren al. (2015), image segmentation Dai et al. (2016), etc. The success of CNNs is inseparable from excessive number of parameters that are well organized to perform sophisticated computations, which conflicts with the increasing demand for deploying these resource-consuming applications resource-limite devices.
Network pruning has been proposed to effectively reduce the deep model's resource cost without significant accuracy drop. Unstructured pruning methods Han et al. (2015b a) usually reach higher compression rate, while relying on dedicated sondwares/librriies to achieve the actual effect. In contrast, structured pruning methods Li et al. (2016): He et al. (0077b) preserve the original Considering the greater reduction in terms
convolutional structure and are more hardware-friendd of fooii operations (FLOPs) and hardware commonality, this research focuses on channel pruning. Existing methods perform channel pruning either statically or dynamically. Static pruning methods remove the same channels for all images Molchanov al. (2019): Tang et al. (2020) while dynamic pruning removes different channels for different images Rao et al. (2018): Tang al.) (2021b). However, both existing static methods and dynamic methods have some limitations. Particularly, given the fact that the channel redundancy is highly sample-dependent static pruning methods may remove some channels that are not redundant for certain images. Consequently, static methods refrain from larger pruning rate to avoid significant accuracy drop. To tackle the issue of image-specific redundant channels, dynamic pruning methods remove image-speciff channels. In this way, dynamic methods achieve the ttate-of-the-ar pruning ratio without significantly sacrificing performance. Despite the significant advantage, dynamic pruning usually requires preserving the full original model during inference, which restricts its practical deployment resource-limite devices.

---

Under review as a conference paper at ICLR 2023
In this paper, we propose new paradigm of static pruning method named GlobalPru. Although born from static, GlobalPru tackles the issue of image-speciftt redundant channels by making all images share the same ranking of channels with respect to redundancy. In other words, GlobalPru forces all images to agree on the same ranking of channel saliency (referred to as global channel ranking) to reduce the image-specific channel redundaney to the greatest extent possible. By removing channels with the lowest global rankings, GlobalPru can avoid the problem of existing static pruning methods, which remove more important channels while retaining less important channels for specific images with a high probability. More specifically, we first propose a novel global channel attention mechanism. Channel attention local in the sense that the ranking of channels with respect their importance is image-specific. Different from existing ones, our proposed global channel attention mechanism can identify the global channel ranking across all different samples in the training set, particularly through a learn-to-rank regularization. In detail, to make the static GlobalPru approach the maximum image-specine compression ratio of dynamic pruning as well as stabilize the training process, we first use majority-votin voting-based strategy to specify the global ranking to make the static Gro approach the maximum image-specific compression ratio of dynamic pruning. Then, given certain ranking, all the image-specifie channel rankings are forced to agree on the ranking via tearn-to-rank regularization. When all the image-specific channel rankings are the same as the given ranking, the ranking becomes the global ranking. As a result of exposing global ranking for all images during training stages, GlobalPru can also avoid the disadvantage of existing dynamic pruning which needs to store the entire model for deciding image-specife channel ranking during inference and perform more efficient pruning on globally ordered channels.
Our contributions are summarized follows:
We propose GloablPru, static network pruning method. GlobalPru tackles the issue of image-specific channel redundancy faced by existing static methods by learning global ranking of channels w.r.t redundancy. GlobalPru produces pruned network such that GlobalPru is more memory-efficient solution than existing dynamic methods.
To the best of our knowledge, we for the first time propose a global channel attention mechanism where all the images share the same ranking of channels w.r.t. importance. Instead of repeatedly computing image-specific channel rankings under existing local attention mechanisms, our proposed global attention enriches the representation capacity of models and therefore greatly improves the pruning efficiency.
Extensive experimental results show that GlobalPru can achieve state-of-nn of-the-art performance with almost all popular convolution neural network architectures.
RELATED WORK
2.1 STATIC PRUNING & DYNAMIC PRUNING
As the most traditional and classic model pruning method, static pruning shares a compact model among all different samples Wen et al. (2066) Liu et al. (001a)): Liebenwein et al. (019)) Molchanov et al. 2019 Tang et al. (2020). To be specific, static methods select pruning results through trade-offs on different samples, which leads to final compact models having limited representation capacity and thus suffering an obvious accuracy drop with large pruning rates. Recently, some works turn their attention to the pursuit of the ultimate pruning rate and focus on excavating sample-wist model redundancy, named dynamic pruning. Dynamic pruming generates different compact models for different samples Dong et al. (2017): Gao et al. (2018); Hua al.; (2019); Rao et al. (2018); Taag et al. (2021b). Actually, dynamic methods learn path-decision module to find the optimal model path for each input during inference. For example, state-of-the-ar work Liu et al, (2019) investigates feature decay regularization to identify informative features for different samples, and therefore achieves an intermediate feature map to the model sparsity. Tang et al. (202 further improves the dynamic pruning efficiency by embedding the manifold information of all samples into the space of pruned networks. Despite dynamic methods achieve higher compression rate, most of them are not memory-efficien cause most of them requires to deploy the full model even in inference stage.
2

---

Under review as a conference paper at ICLR 2023
2.2 CHANNEL ATTENTION
Channel attention is similar to the scale coefficients added on feature channels to enhance the important features and also weaken the unimportant ones. It is usually implemented through extra channel attention modules, which exploit the inter-channel relationship for imput feature maps. An typical channel attention mechanism is SENet et al. (2017), it uses the "Squceze-and aeuuooze-and-EEcitaaioor (SE) block which be stacked together to adaptively recalibrate channel-wise feature responses on each convolutional layer by explicitly modelling inter-dependencies between channels. [Li et al. (2019) proposes dynamic selection mechanism CNNs that allows each neuron adaptively adjust its receptive field size based on multiple scales of input information, which is named "Selective Kernel" (SK) unit. By dynamically calculating channel attention on the different kemnlls, SK realizes parameter sharing and thus improves the model efficiency. In summary, nearly all the existing channel attention are local in the sense that the channel saliency image-specific and can not identify a global channel attention over the entire dataset.
3 GLOBAL ATTENTION-BASED CHANNEL PRUNING
In this section, will give detailed formulation and theoretical explanation of the proposed Global Channel Attention Pruning (GlobalPru). As illustrated Scction 3.1, the GlobalPru is formulated as static alternative regularized simultaneously by local channel-wise saliency and the global channel rank loss and can be optimized in an end-to-end manner. Then, the training process of GlobalPru divided into two stages including Global Channel Attention Election (stage 1) and Learn-to rank Regularization Pruning (stage 2), which would be illustrated and proven theoretically in Section 3.2 and Section 3.3, respectively. In stage 1, GlobalPru explicitly elects the global channel attention that benefits most samples from the observations of the locally image-specifis channel dependencies. After that, stage 2, the proposed learn-to-rank algorithm forces the channels to ordered towards the global channel attention concurrently during the model training process.
Loss
Conv
Prediction Loss
Mask
Score
Score
Rank
Vote Pool
Global Attention
Global Channel Attention
Rank
(a) Global channel attention.
(b) Learn ta-rank regularization.
Figure 1: The overview of the training process of Channel Rank Network.
3.1 PROBLEM FORMULATION
Given dataset with N samples as X = [xr) with the corresponding labels Y = { For convolution neural network (CNN) with L convolution layers and parameters set θ W E gexx xx xxxxxxx'** denotes the convolution parameters in the I-th layer, where el the channel numbers of I-th layer and k represents the corresponding kemnel size. F'(x;) E Rbxe' xhxx is the output feature map of the I-th layer for the input Iı, where b, h, the output feature map, respectively. Given the input feature F (æ,). the output of layer can be calculated as F' ('(x;) Relul Bn(Wi / do pl-1)), where Relu and Bn represent the activation and
the batch size, height and width of
3

---

Under review as a conference paper at ICLR 2023
batch normalization operation respectively. Finally, use f(x, )) to represent the output of the neural network over imput samples X.
Static channel pruning climinates the same redundancies and discovers compact network among all images. And most static methods are guided by empirical risk minimization and numeric- based regular terms, iee, parameter magnitudes, channel saliencies reconstruction errors, etc. Typical static methods can be formulated as follows:
min Σ L (J )): + Norm(O),
where L denotes the loss function and Norm(-) is the regularization term which is usually human-designed criterion for inducing model sparsity. And À used as a knob to strike the different trade-offs between model accuracy and sparsity ratio.
Conversely, dynamic channel pruning discovers effective sub-nes networks for each input dependently during the inference stage, which is usually implemented through additional model path-finding functions. Define a scoring module S (a squeeze and excitation module (Hu al. (2017) in our work),
for a specific input 24 the channel saliencies on layer can be computed as n (zx) = (xx) E R where x (1r;) indicates the relative importance score of the j-th channel, and a smaller m indicates that channel j is less important. Given a pruning threshold e', the variable mask), to indicate the removal of the j-th channel of layer / is initialized to I and set to 0 when (2)) <''. c. The channel saliencies for the pruned model would be re-calculated as:
# (x:) = n (2:) ) @ mask'(;)
And the feature output of the (-th layer would be:
F((,)) Relu BN ((mask (x:) (x;) & W
Thus, a general dynamic pruning paradigm could be formulated as EQ. wherein the regularization used to induce the instant-wise network sparsity.
min ( O), v1)
Σ || (x:)l
Dynamic pruning has achieved higher compression rates than static methods by removing instance- specific model redundancy, however, the calculation of n (2xz) depends on the complete referenced model along with additional path-addressing units being deployed on resource-limited devices. These unavoidable dependencies make dynamic methods not memory and computation-efficien thus without great applicability.
To utilize advantages from both static and dynamic methods while avoiding the corresponding bottlenecks, GlobalPru expects all these samples to reach an agreement on the channel importance ranking simultaneously with model training, thus achieving a dynamic-like static channel pruning. To this end, GlobalPru first investigates the local channel attention of different samples via preheated scoring module S, based on which it could identify global channel attention the channel importance ranking prior through the Majority Vote Mechanism. After that, GlobalPru forces all image-specific channel rankings to converge to the certain global channel ranking by the novel Learn-to-Be Rank regularization. The detailed formulation of GlobalPru is as follows.
Let R denote the medium results of channel importance ranking of layer I most voted by all input samples during the training process, and T the corresponding channel ranking prior. Hence he channel ranking loss could be computed 4 R₁, Ti), wherein & is the learn-to-rant regularization defined in Eq. 7 The optimization target of GlobalPru can be formulated as:
N
L
min
(ffez, v. + +a]))(2)))) + TT),
(5)
1=1
Let T. and Rt represent the importance ranking index of the j-th channel the prior and the global attention during training, respectively. We elaborately use the previously defined channel score |</]] to measure the similarity between the current position and the target position of the j-th channel on
4

---

Under review as a conference paper at ICLR 2023
layer 1, and have P(R) = TT)
emp(x})
Hence + can be formulated in Bayesian perspective
Σ,=1 E,=1dexp(at)) exp(as)
as:
d'
P(T = -T)P(()
(TT" R²') = -oo II
P(R) 1112;)
log
(6)
P(T))
==1
Since 7 is defined as the prior of the channel rank and independent the current channel ranking. we have P(T") = and P(T R = = 1, hence we can rewrite EQ. as:
d'
exp (TT)
" R²)
log P(R) = Tブ)
log
Σ 1j=1 exp(TT))
To simulate the impact of model pruning and reduce computational overhead, we use ft instead of Ty in actual optimization.
3.2 GI.OBAL CHANNEL ATTENTION
Definition 1 Channel local attention refers to the channel attention of single input (channel importance ranking), while channel global attention refers to the channel attention consistent with the most samples for the current data domain.
Lots previous works have proven that the importance ranking of channels Tang et al. (2021b) lb) is highly sample-dependent, which means the model redundancy for different samples could be quite different with high probability. fact, natural images are mixture of some intrinsic features such as color, edges, textures, etc., in different ratios, resulting in different requirements of different samples for the model. Channel attention actually adjusts the weight of each channel the output features for each sample approximate the target feature mapping. Thus, we give the following corollary.
Corollary Conventional channel attention could indicate the quantitative channel redundancy for each sample through the learnable feature scaling
fcalingfacttrrs. factors.
Corollary 2 Local channel attention is highly sample-related, which implies that redundant channels are disordered and unpredictable for the whole sample domain.
To implement global channel attention across samples and therefore eliminate the dependency of dynamic pruning on sample-wise path- futth-findin,, we propose to explore the "Global Channel Attention" for all different samples. As shown in Figure [a). the global channel attention is identified by the majority vote mechanism. Specifically, a deep neural network with channel attention modules first pre-warmed. After that, in the training stage, every image-specific local channel attention is collected as the electoral college to be voted for global channel attention, which would be the prior of the channel importance ranking in the learn-to-rank regularization. In this way, as a static alternative, Global obtains the most supported channel importance ordering in the current data domain to achieve high-gu qualltul dynamic- pruning, achieving a higher model compression rate and reducing the precision decline from pruning.
3.3 LEARN-TO-RANK REGULARIZATION
The learn-to -oontto-rak regularization explicitly modeled from Bayesian perspective as a joint probability maximization problem. As shown in Figure [bb) and EQ. specifically, regarding the global channel attention as the prior of the channel importance ranking, channel ranking loss can be formalized as the negative value of the probability that the current channel ranking is equal to the given prior. Thus, in addition to empirical risk minimization, GlobalPru trains a model distribution towards the probabilistic model which is most likely generate the target channel ranking. However, the huge number of channels CNNs brings unique challenges for computing the ranking loss for each channel. We use the unique advantages of model pruning to address the problem of huge computational costs in EQ.8] given the pruning rate p. the computation cost could be further reduced by maximizing the probability that only the first (1 p)c channels in T are ranked correctly. The
5

---

Under review as a conference paper at ICLR 2023
proposed learn-to-ran to-nkk regularization could be reformatted to Eq.
and:
(1-ple²
(1-p)e²
Σ
log (xpp(pt))) log
Σ exp(x))
(8)
4 EXPERIMENT
In this section, GlobalPru is empirically tested the most representative neural network architectures: plain architecture, residual structure, and lightweight depthwise convolution networks to verify the effectiveness of the proposed "Global Attention Mechanism" and "Leam-To-Rank" method.
Datasets:
The effectiveness of the proposed GlobalPru is verified on most popular image datasets of model pruning, including MinilmageNet, CIFAR-10, CIFAR 100 and SVHN. Diverse datasets can help effectively demonstrate the broad applicability of our GlobalPru.
Models: We explore the effectiveness of GlobalPru on three mainstream neural network architec- tures includes: plain architecture- VGG-16, residual structure ResNet with different depth, and lightweight depthwise convolution MobileNet-V2. These three representative architectures cover almost all the most popular convolutional networks currently.
Tasks: Most model pruning works only demonstrate the accuracy of the pruned models on the image classification task, and ignores the robustness of the pruned networks. To further demonstrate the applicability of our method, we also evaluate the performance of the pruned network against adversarial perturbations on the adversarial samples detection task.
Implementation Details: Standard data argumentation RandomSizedCrop and RandomHorizon- talFlip are used in all datasets we used. The coefficient to regulate the channel saliency score is set as 0.0001 our work. And another coefficient, 33 to balance the weight the channel rank loss also set 0.0001 empirically. The stochastic gradient descent (SGD) is used for all training processes. Two pruning modes are tested in GlobalPru, named "Fixed" and "Mixed", respectively. Under the fixed mode, all layers use the same pruning rate with the Lottery Ticket of model pruning which corresponds to the highest test accuracy Frankle Carbin (2019). For ResNet-1 8 on CIFAR 10 in this work, the range of pruning rates which achieve higher test accuracy than the original model is [38.9%, 100.0%). and the optimal pruning rate corresponds to the highest test accuracy is 61.1% Under the mixed mode, the layer-wise pruning rates are set according the sensitivity of each layer to model perturbation. Dong et al. (2020) has theoretically proved that the right sensitivity metric for model perturbation is the average Hessian trace. Align the value of the average Hessian trace of all layers with the optimal pruning rate, and the fluctuation of the pruning rate of each layer relative to the optimal pruning rate set according to the standard deviation of the Hessian trace of this layer relative to the model average The larger the Hessian trace value, the lower the pruning rate. All of the experiments are conducted on NVIDIA GeForce GPUs.
4.1 COMPARISON ON PLAIN ARCHITECTURE
The proposed method is first compared with the state-of-the-ar network pruning algorithm on the most intuitive plain architecture, which is the initially successful way construct a convolutional neural network. The typical plain network VGG-16 we used consists of a few building blocks (convolutional layers, activation layers, etc.) stacked in vertical manner. A "Squeezz-anddEExiiaatio module is added following the convolutional layer to measure its relative channel saliency. Both static methods including Luo al. (2017); [i al. (017); He et al. (2017a): [Lu al.] (2017b). and the dynamic methods including Lin et al. (2017): Gao et al. (2019) are compared. All of the algorithms are tested on the Cifar- dataset and the comparison results are shown in Table 1 The pruning performance is reported over three main metrics: Memory Access Cost (MAC), FLoating point Operations (FLOPs), and Classification Accuracy Drop. When using the fixed pruning rate for all the computing layers, our method achieves 55.2% FLOPs decrease with negligible accuracy drop of -0.21 %. And when using the empirically mixed pruning rate for each layer, our method can prune more 1.5% FLOPs than the fixed mode, while with less accuracy degradation. The results show that our approach achieves the highest compression ratio on VGG-16 with competitive model performance. Compared to the SOTA static pruning method, undoubtedly, our method wins

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison of pruning performance VGG-16 architecture and Cifar-10 dataset. Our method is compared with different channel pruning algorithms including both traditional static methods and sample-wise dynamic methods. "Dy?" means the current method is sample-wise dynamic method. "Fixed" and "Mixed" represent the above-mentionel fixed pruning mode and mixed pruning mode, respectively. "*" denotes the new static pruning paradigm, which approximates
the maximum compression ratio of dynamic methods.
Method
Cite
Dy?
LMAC
FLOPs(M)
LFLOPs
-Acc
Baseline
0
626.4
0
0
LIWS
Wang al.
(2020)
Y
53.2%
184
70.6%
-0.8%
ThiNet
Luo
2017
N
50%
313
50%
-0.14%
LI-norm
Li et
2017
N
34%
413.2
34%
-0.5%
CP
He et al.
2017a
50%
313
50%
-0.32%
NS
Liu et al.
2017b
51%
306.8
51%
-0.19%
RNP
Lin et al.
(0177)
50%
313
50%
-0.85%
FBS
Gao et al.
(2019)
50%
310.4
50.4%
-0.47%
GlobalPru(Fixed)
Ours
55%
280.87
55.2%
-0.21%
GlobalPru(Mixed)
Ours
56.7%
271.26
56.7%
-0.15%
Table 2: Comparison of pruning performance Residual architecture: ResNet-20 architecture and
Cifar-10 1:aar-10aa dataset.
Method
Cite
Dy?
FL.OPs(M)
JFLOPs
-Acc
Baseline
43.17
0
SFP
He et al.
(2018)
N
24.95
42.2%
-1.39%
FPGM
He et al.
(011
N
19.86
54%
-1.78%
DSA
Ning et
2200
N
21.46
50.3%
-0.84%
Hinge
et al.
2020a
N
23.53
45.5%
-0.38%
DHP
Li et al.
(020b)
N
20.80
51.8%
-0.68%
Maninp
Tang et al.
(021a)
Y
19.77
54.2%
-0.17%
FBS
Gao et al.
(0199)
Y
19.97
53.1%
-1.25%
GlobalPru(Fixed)
Ours
15.77
60.84%
-0.14%
GlobalPru(Mixed)
Ours
16.07
60.1%
-0.26%
absolute advantage in pruning rate and only has a slight accuracy drop, i.e., -0.07% and -0.01 respectively for fixed and mixed modes, than state-of-the pioneering dynamic pruning method, our method gains and obviously less accuracy loss than the newest Liu et al. (2019), and outperforms Lin et al. (007)) Gao et al. (2019) in both compression rate and accuracy.
ThiNet. And when compared to the
4.2 COMPARISON ON RESIDUAL ARCHITECTURE
We further explore the effectiveness of GlobalPru the residual neural networks, where GlobalPru explores global attention inside residual blocks by adding the vanilla input feature representations to the output feature maps scaled by channel attention. To be specific, we introduce compression each convolutional layer inside the residual block without changing the transfer residual information.
The compared methods include both SOTA static method He al. 2018 2019); Ning et al. (2020);
7

---

Under review as a conference paper at ICLR 2023
Table 3: Comparison of pruning performance Wist architecture: MobileNet-V2 architecture and Cifar-10 dataset.
Method
Cite
Dy?
FLOPs(M)
JFLOPs
-Ace
Baseline
89.9
WM
Zhuang et al.
(0018)
N
66.53
26%
-3.68%
DCP
Z.uang et al. (0118)
66.53
26%
-3.16%
NPPM
Gao et al. (2021
N
47.65
47%
-3.1%
GlobalPru(Fixed)
Ours
47.47
47.2%
-0.69%
GloxalPru(Fixed)
Ours
35.83
60.14%
-0.55%
ct al. (2020a by and state-of-the-art dynamic pruning method Tang et al. (2021a) and Gao et al. (2019). The results are tested on ResNet-20 and C0. As shown in Table 2 the FLOPs, FLOPs decrease relative to the baseline and post pruning accuracy drop are reported. Our method under both fixed pruning rate and mixed pruning rate setting achieve the highest compression ratio at the with the most negligible accuracy drop. For example, even under the highest pruning rate 68% for all layers, the accuracy of method still reaches 98.01% % with only -0.34 degradation. Besides, also compare the performance of GlobalPru with the classic three-stage train-prune-fine-uur pruning different pruning rates. According Figure ??, we choose 40.1% 1% and 61.1% 1% as model pruning rates. They correspond to the point where the generalization performance is the highest and the point where the generalization accuracy starts to suffer, respectively. All of the comparisons show that Gru can work efficiently on residual neural networks.
time
static
Accuracy (%))
Accuracy
30
Lottery
Lottery
GlobalPru
GlobalPru
200
300
380
400
100
Interation
Interation
(a) Pruming rate 40.1%
(b) Pruning rate 61.1%.
Figure 2: Comparison of pruning performance under different pruning rates ResNet-20 and Cif "Lottery" represents an typical three-stage static pruning paradigm.
4.3 COMPARISON ON DEPTHWISE ARCHITECTURE
Another popular model architecture using "Depthwise Separable Convolution" is also included in our experiment to cover another class of models in the spotlight. Also, the depthwise architecture is originally designed for resource-limited mobile devices and already very light-weight. To validate whether our method can further compress these already compressed models, we compare it with the most widely used efficient MobileNet- V2. As shown in Table 3 while outperforming all methods in computing reduction, model achieves the closest model accuracy to the baseline. The advantage of our method on lightweight Depth Wise architecture is obvious which proves that our model enables further model compression in almost lossless mode current wate-of-the-ar methods.
8

---

Under review as a conference paper at ICLR 2023
Table 4: Comparison of pruning performance MinilmageNet (M-ImageNet) and SVHN datasets
with popular model pruning SOTAs.
Accuracy (%)
Dataset
Method
Top
Top
J.Size
PolarPrun
85.857
96.068
0.59
SlimPrun
70.06
90.44
0.60
M M-ImageNet
GlobalPru
89.227
96.97
0.60
PolarPrun 97.099
98.199
0.59
SlimPrun
96.848
99.228
0.50
SVHN
GlobalPru
97.543
99.3
0.60
Table 5: Comparison AUROC (%) of GlobalPru with typical adversarial detection methods on Fast Gradient Sign Method (FGSM)-based CIFAR100 and SVHN datasets.
Non Prune
Prune
AUROC (%)
MAHA
FBS
DPIC
GlobalPru
FGSM
SVHN
99.63
99.95
99.96
100.00
CIFAR100
99.77
100.00
100.00
100.00
4.4 GENERALIZATION VERIFICATION
In addition model accuracy, good pruning method also needs to have good generalization and robustness, which usually ignored in previous work. To show that GlobalPru has these favorable properties, we test the performance of GlobalPru on more diverse vision dataset and extra visual tasks with added adversarial samples.
compare the performance of GlobalPru and two static pruning SOTAs typical image datasets SVHN and MinilmageNet. SVHN contains large-scaled labeled data (over 600,000 digit images) for more difficult, unsolved real-world problem. MinilmageNNe,, drawn from ImageNet, has complex samples and multi-classes, more suitable for prototyping and experimental research. Obviously, as static alternative, GlobalPru achieves higher model compression/task loss ratio than static pruning SOTA by removing the global redundancy of the model for all samples.
As shown in Table 4
As shown in T we verify the robustness of our method by observing the performance Glob- alPru when faced with adversarial perturbations. It can be seen that GlobalPru exhibits extraordinary perturbation resistance, even surpassing some typical adversarial sample detection methods. We speculate that this is because the model redundancy removed by GlobalPru is selected by majority vote of all samples, thereby weakening the effect of small image disturbances and making GlobalPru more robust than the previous pruning criterion only guided by task loss.
CONCLUTION
To sum up, we propose a novel pruning method, i.e., the Channel Attention-based Leam-to-Rank Network. based on the shortcomings of current dynamic pruning methods, such the need save the complete model locally and repeat the forward computation. Our approach first explores the channel saliency rank of each sample, and then selects the most suitable channel rank supported by all current inputs through majority voting mechanism. We define this channel rank as global channel attention. Next, we obtain a Channel Rank Network by proposing an efficient channel sorting algorithm to incorporate the knowledge of global channel attention into the training of the model, so as to quickly give an appropriate pruning response when new samples or sparse requirements arrive. While obtaining the advantages of dynamic pruning, method avoids the defects of the original dynamic pruning method, and achieves better pruning performance than most state-of-the-an methods.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jifeng Dai, Kaiming He, and Jian Sun. Instance- aware semantic segmentation via multi-task network cascades. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 3150-3158, 2016.
Xuanyi Dong, Junshi Huang, Yi Yang. and Shuicheng Yan. More is less: A more complicated network with less inference complexity. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5840-5848, 2017.
Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. HAWQ-V2 V2: hessian aware trace-weighted quantization of neural networks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Shangqian Gao, Feihu Huang, Weidong Cai, and Heng Huang. Network pruning via performance maximization. In IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 9270-9280. Computer Vision Foundation IEEE, 2021.
Xitong Gao, Yiren Zhao, Lukasz Dudziak. Robert Mullins, and Cheng-zhong Xu. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint arXiv:1810.05331, 2018.
Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert D. Mullins, and Cheng-Zhong Xu. Dynamic channel pruning: Feature boosting and suppression. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Song Han, Huizi Mao, and William Dally, Deep compression: Compressing deep neural networks 2015a.
with pruning, trained quantization and huffman coding. arXiv preprint
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In Proceedings of the Twenty-Seventh Seventh International Joint Conference Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pp. 2234-2240. jjcai.org. 2018
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceddings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 4340-4349 2019.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pp. 1389-1397, 201 7a.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pp. 1389-1397, 2017b.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and- waweeee--andee--tttiiooo networks. CoRR, abs/1709.01507. 2017.
Weizhe Hua, Yuan Zhou, Christopher M De Sa, Zhiru Zhang, and Edward Suh. Channel gating neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet. and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 08710, 2016.
10

---

Under review as a conference paper at ICLR 2023
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 510-519, 2019.
Yawei Li, Shuhang Gu, Christoph Mayer, Luc Van Gool, and Radu Timofte. Group sparsity: The hinge between filter pruning and decomposition for network compression. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8018-8027, 2020a.
Yawei Li, Shuhang Gu, Kai Zhang, Luc Van Gool, and Radu Timofte. Dhp: Differentiable meta pruning via hypernetworks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII 16, pp. 608 624. Springer, 2020b.
Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. In International Conference Learning Representations, 2019.
Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 1178-1188, 2017.
Chuanjian Liu, Yunhe Wang, Kai Han, Chunjing Xu, and Chang Xu. Learning instance- wise sparsity for accelerating deep models. In Proceedings of the Twenty- Eighth International Joint Conference on Art Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 3001-3007. ijcai.org, 2019.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. Proceedings of the IEEE international conference on computer vision, pp. 2736-2744, 2017a
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 2755-2763. IEEE Computer Society, 2017b.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-5066, 2017.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Turi Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1164--12772 2019.
Xuefei Ning, Tianchen Zhao, Wenshuo Li, Peng Lei, Yu Wang, and Huazhong Yang. Dsa: More efficient budgeted pruning via differentiable sparsity allocation. In Computer Risio-EECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pp. 592-607. Springer, 2020.
Yongming Rao, Jiwen Lu, Lin, and Jie Zhou. Runtime network routing for efficient image classification. IEEE transactions on pattern analysis and machine intelligence, 41(10):291-2304, 2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Yehui Tang, Yunhe Wang. Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. Advances in Neural Information Processing Systems,
2020.
Yehui Tang, Yunhe Wang, Yixing Xu, Yiping Deng, Chao Xu, Dacheng Tao, and Chang Xu. Manifold regularized dynamic network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5018-5028, 2021a.

---

Under review as a conference paper at ICLR 2023
Yehui Tang. Yunhe Wang, Yixing Xu, Yiping Deng, Chao Xu, Dacheng Tao, and Chang Xu. Manifold regularized dynamic network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5018 5028,, 2021b.
Yulong Wang, Xiaolu Zhang, Xiaolin Hu, Bo Zhang, and Hang Su. Dynamic network pruning with interpretable layerwise channel selection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34. pp. 6299-6306, 2020.
Wei Wen, Chunpeng Wu, Yandan Wang. Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in neural information processing systems, 29, 2016.
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jin-Hui Zhu. Discrimination-aware channel pruning for deep neural networks. In Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 883-894, 2018.
12


Reviews:
[{"review_id": "iX8LQlHyTqX", "reviewer": "Reviewer_sMpW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a new network pruning framework via global channel attention. In particular, it first computes the channel importance for the whole dataset via a SE block. Then 'global channel attention' is computed over the whole dataset. Then, it forces all samples to learn the same rank as the 'global channel rank'. Experiments on different network architectures and datasets are conducted to verify the effectiveness of the proposed method.", "summary_of_the_review": "A new pruning framework is proposed in this paper. However, one of key component is not novel and key results on ImageNet are missing. More recent and high-performance SOTAs are recommended.", "strength_and_weaknesses": "The strengths of the proposed method are listed as below:\n+ The paper is well organized and easy to follow.\n+ Experiments on different network architectures are conducted.\n+ Both static pruning and dynamic pruning methods are compared in experiments.\n\nThe weaknesses of the proposed method are listed as below:\n- There are two key components of the method, namely, the attention computation and learn-to-rank module. For the first component, it is a common practice to compute importance using SE blocks. Therefore, the novelty of this component is limited. \n- Some important SOTAs are missing and some of them as below outperform the proposed method:\n(1) Ding, Xiaohan, et al. \"Resrep: Lossless cnn pruning via decoupling remembering and forgetting.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021. \n(2) Li, Bailin, et al. \"Eagleeye: Fast sub-net evaluation for efficient neural network pruning.\" European conference on computer vision. Springer, Cham, 2020. \n(3) Ruan, Xiaofeng, et al. \"DPFPS: dynamic and progressive filter pruning for compressing convolutional neural networks from scratch.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 3. 2021.\n- Competing dynamic-pruning methods are kind of out-of-date. More recent works should be included.\n- Only results on small scale datasets are provided. Results on large scale datasets including ImageNet should be included to further verify the effectiveness of the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dw1a-tN1X7w", "reviewer": "Reviewer_ywfs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a channel pruning method which aims to enhance global consistence among samples for network channels. \nIt claims integrating both static and dynamic information into the formulation. \nComparison on small datasets demonstrate the effectiveness of the proposed methods. ", "summary_of_the_review": "See previous comments.", "strength_and_weaknesses": "Strong points\n+  The bayesian based formulation for learning-to-rank is somewhat novel to me. \n\n\nWeak Points\n- This is a very rushing paper with a lot of typos and errors. \n\n       - There are plenty of repeated references. i.e., many identical references are listed more than once in the reference section. \n       - page-8, there are missing ref for Figure ??\n       - Eq-7, there are two minus after =. \n       - learn-to-rank should be \"learning-to-rank\" generally. \n\n- Some descriptions are not complete, for instance\n      - what are the network architecture for M-ImageNet and SVHN in Table-4? And what is the original performance?\n\n- Some confused points:\n      - in the contribution claims, \"GlobalPru\" is a static pruning method. However, in Table-1/2, you marked it as *, which means it is both static or dynamic, right? And in eq-5, you include the \\pi(x) term which is dynamic, right? If it is static, you should exclude this term. \nIf not, the contribution of those two terms need investigated. \n     \n     - It claims \"global\". From description, it is still pruned layer-by-layer? especially for the \"fixed\" scenario. For the \"mixed\" case, it is unclear how it was realized. Just according to the ranking of $\\pi_j$ from all layers?\n\n- It lacks experiments on large-scale network on large-scale dataset like ImageNet, which this is done for most of the compared citations in the paper. \n\n- page-5, for the corollary-1/2, do you have any mathematical proof or deductive process? Otherwise, they are just some of your claims. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "88nLCaDKdy7", "reviewer": "Reviewer_3k5Z", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores a static alternative pruning method for dynamic pruning methods. They propose channel attention-based learn-to-rank algorithm and  channel attention prior among all sample-specific channel saliencies. A Bayesian-based regularization is further introduced to enhance the performance.", "summary_of_the_review": "The proposed method seems to share some novelty. The experiments are too weak. The acceleration on hardware is missing.", "strength_and_weaknesses": "Weaknesses\n1. The major concern is that the experiment comparison. I find the paper does not compare with sufficient recent works. Thus I may doubt why the performance is SOTA or not.\n2. There is no experiments on ImageNet. \n3. Experiments are too weak.\n4. Are dynamic pruning methods benefitting from acceleration on real-world devices?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0I_Tf3piz6B", "reviewer": "Reviewer_VwQA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The proposed work initially obtains a majority vote-based prior on the global rank of channel saliencies before forcing each sample-level channel saliency to match the global prior. In this way, the proposed work aims to use the platform of static pruning yet match the high pruning levels similar to dynamic pruning while maintaining a common channel saliency across all samples.", "summary_of_the_review": "Justifying and addressing the points addressed in the weaknesses mentioned above could serve to highlight interesting comparisons between the proposed and existing methods.", "strength_and_weaknesses": "Strengths\n- The context and explanation provided for static and dynamic pruning are well done.\n\nWeaknesses\n- Quantitatively, on a channel to channel comparison, could the authors provide more insight in to the difference in performance and saliency between dynamic pruning approaches and GlobalPru? This could expose both the positive and negative aspects of both approaches.\n- Channel level attention spans multiple ideologies and cannot be solely categorized as methods that are \"local\", especially over datasets, since certain methods learn inter-channel relationships over the dataset as opposed to sample-specific properties. Could the authors justify their statement in Pg. 3, Section 2.2, Lines 8-10?\n- I encourage the authors to take a closer look at Figure 1 and revise it slightly so that it can be a common reference to the underlying process, especially across later sections. As constituted currently, there are certain missing elements and the flow of processes in the diagram is confusing.\n- Equation 1 emphasizes objective functions which learn the mask to be applied on the weight matrices. Could the authors clarify if subsequent comparisons in the experiments section maintain this characteristic?\n- The nomenclature of \"prior\" and \"global attention during training\" need to be clearly defined before being put to use. As constituted currently, they are clarified just before Section 3.3. I encourage the authors to revise the explanation in Section 3 to ensure preliminary terms are well defined before they are put to use.\n- Could the authors clarify in some detail the reasoning behind the choice of expressions and formulation for the $\\phi()$, and all relevant information beyond Equation 5? On first glance, there seems to be some inconsistency in notation in Equations 6 and 7.\n- After establishing $\\alpha, \\beta$ as balancing coefficients in the main loss function, the experimental setup highlights their values to be 0.0001. Could the authors justify the choice of small values, including a comparison of the impact of varying them across a range of values?\n- Results from Table 1 consistently compare against ThiNet. However, there exist a number of more advanced methods, even static pruning, which improve upon ThiNet. Could the authors provide comparisons against current works that improve upon the performance of ThiNet?\n- Could the authors clarify the baseline performance across the result tables provided and whether they match the relative drop in accuracy values across baselines?\n- Figure 2, the X-axis is incorrectly labelled \"Interation\". Please revise the label.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject