Model Compression via Hyper-Structure Network
Authors: Shangqian Gao
Feihu Huang
Heng Huang
Venue:  (2021)

Abstract: In this paper, we propose a novel channel pruning method to solve the problem of compression and acceleration of Convolutional Neural Networks (CNNs). Previous channel pruning methods usually ignore the relationships between channels and layers. Many of them parameterize each channel independently by using gates or similar concepts. To fill this gap, a hyper-structure network is proposed to generate the architecture of the main network. Like the existing hypernet, our hyper-structure network can be optimized by regular backpropagation. Moreover, we use a regularization term to specify the computational resource of the compact network. Usually, FLOPs is used as the criterion of computational resource. However, if FLOPs is used in the regularization, it may over penalize early layers. To address this issue, we further introduce learnable layer-wise scaling factors to balance the gradients from different terms, and they can be optimized by hyper-gradient descent. Extensive experimental results on CIFAR-10 and ImageNet show that our method is competitive with state-of-the-art methods. 

Under review as a conference paper at ICLR 2021
MODEL COMPRESSION VIA HYPER-STRUCTURE NET- WORK
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we propose a novel channel pruning method to solve the problem of compression and acceleration of Convolutional Neural Networks (CNNs). Previous channel pruning methods usually ignore the relationships between channels and layers. Many of them parameterize each channel independently by using gates similar concepts. To fill this gap, a hyper-structure network is proposed to generate the architecture of the main network. Like the existing hypernet, our hyper- structure network can be optimized by regular backpropagation. Moreover, we use a regularization term to specify the computational resource of the compact network. Usually, FLOPs is used as the criterion of computational resource. However, if FLOPs is used in the regularization, it may over penalize early layers. To address this issue, we further introduce learnable layer-wise scaling factors to balance the gradients from different terms, and they can be optimized by hyper-grae gradient descent. Extensive experimental results CIFAR-10 10 and ImageNet show that our method is competitive with state-of-the-art methods.
INTRODUCTION
Convolutional Neural Networks (CNNs) have accomplished great success in many machine learning and computer vision tasks (Krizhevsky et al. 20122] Redmon et al.) 2016 Ren et al. 2015;; Simonyan & Zisserman 2014a| Bojarski al.] (2166). To deal with real world applications, recently, the design CNNs becomes more and more complicated in terms of width, depth, etc. (Krizhevsky et al. 2012
Simonyan Zisserman 2014b] He al. 2016 Huang et al. 2017). Although these complex CNNs benchmark tasks, their computational and storage costs increase dramatically. As a result, a typical application based on CNNs can easily exhaust an embedded or mobile device due to its enormous costs. Given such costs, the application can hardly be deployed on snource-limited platforms. To tackle these problems, many methods (Han et al.) 22055b a) have been devoted to compressing the original large CNNs into compact models. Among these methods, weight pruning and structural pruning are two popular directions.
can attain better performance
Unlike weight pruning or sparsification, structural pruning, especially channel pruning, is effective way to truncate the computational cost of a model because it does not require any post- post-processing steps achieve actual acceleration and compression. Many existing works (Liu al. 2017] Ye et al. 20188; Huang & Wang 2018 Kim et al., 2020: You et al. 2019) try to solve the problem of structure pruning by applying gates or similar concepts on channels of a layer. Although these ideas have achieved many successes in channel pruning, there are some potential problems. Usually, each gate has its own parameter, but parameters from different gates do not have dependence. As result, they can hardly learn inter-channel or inter-layer relationships. Due to the same reason, the slimmed models from these methods could overlook the information between different channels and layers, potentially bringing sub-optimal model compression results.
To address these challenges, we propose a novel channel pruning method inspired by hypernet (aa et al. 2016). In hypernet, they propose to use hyper network to generate the weights for another network, while the hypernet can be optimized through backpropagation. We extend a hypernet to htper-structure network to generate architecture vector for a CNN instead of weights. Each architecture vector corresponds to a sub-network from the main (original) network. By doing so, the inter-channel and inter-layer relationships can be captured by our hyper-structure network.

---

Under review as a conference paper at ICLR 2021
Besides the hyper-structuse network, we also introduce regularization term to control the computa- tional budget of a sub-network. Recent model compression methods focus on pruning computational FLOPs instead of parameters. The problem of applying FLOPs regularization is that the gradients of the regularization will heavily penalize early layers which can be regarded as a bias towards latter layers. Such bias will restrict the potential search space of sub-networks. To make our hyper-structure network explore more possible structures, we further introduce layer-wise scaling factors to balance the gradients from different losses for each layer. These factors can be optimized by arper-gradient descent.
Our contributions are summarized follows:
1) Inspired by hypernet, we propose to use a hyper-structure network for model compression capture inter-channe and inter-layer relationships. Similar to hypernet, the proposed hyper-structure network can be optimized by regular backpropagation.
2) Gradients from FLOPS regularization are biased toward latter layers, which truncate the potential search space of sub-network. To balance the gradients from different terms, layer- wise scaling factors are introduced for each layer. These scaling factors can be optimized through hyper-gradient descent with trivial additional costs.
3) Extensive experiments on CIFAR-10 and ImageNet show that our method can outperform both conventional channel pruning methods and AutoMI. based pruning methods on ResNet and MobileNetV2.
RELATED WORKS
2.1 MODEL. COMPRESSION
Recently, model compression has drawn a lot of attention from the community, Among all model compression methods, weight pruning and structural pruning are two popular directions.
Weight pruning eliminates redundant connections without assumptions on the structures of weights. Weight pruning methods can achieve very high compression rate while they need specially designed sparse matrix libraries to achieve acceleration and compression. As one of the early works, Han al. 2015b) proposes to use L1 L2 magnitude as the criterion prune weights and connections. SNIP Lee et al. 2019) updates the importance of each weight by using gradients from loss function. Weights with lower importance will be pruned. Lottery ticket hypothesis (Frankle Carbin 2019) sub-networks within the large network at initialization time. They then retrain the sub-network with the same initialization. In rethinking network pruning (Liu et al. 2019)), they challenging the typical model compression process (training, pruning, fine-tuning), and argue that fine-tune tine-tuning is not necessary. Instead, they show that training the compressed model from scratch with random initialization can obtain better results.
assumes there exist high-performane
One of the previous works (Li et al. 2017) in structural pruning uses the sum of the absolute value of kernel weights as the criterion for filter pruning. Instead of directly pruning filters based on magnitude, structural sparsity learning (Wen et al., (0166) is proposed to prune redundant structures with Group Lasso regularization. On top of structural sparsity, GrOWL regularization is applied to make similar structures share the same weights (Zhang et al. 2018). One of the problems when using Group Lasso is that weights with small values could still be important, and it's difficult for structures under Group Lasso regularization to achieve exact zero values. As a result, Louizos et al. (2018) 2018) propose to use explicit Lo regularization make weights within structures have exact zero values. Besides using the magnitude of structure weights as a criterion, other methods utilize the scaling factor of batchnorm achieve structure pruning, since batchnorm (loffe Szegedy, 2015) is widely used in recent neural network designs (He et al. 2016 Huang et al. [017].. A straightforward way to achieve channel pruning to make the scaling factor of batchnorm to sparse (Liu al. 2017). If the scaling factor of channel fell below a certain threshold, then the channel will be removed. The scaling factor can also be regarded as the gate parameter of a channel. Methods related to this concept include (Ye et al. 2018 Huang & Wang 2018 Kim et al. 2020 You et al.) 2019). Though it has achieved many successes in channel pruning, using gates can not capture the relationships between channels and across layers. Besides using gates, Collaborative channel pruning (Peng et al.) 2019) try to prune channels by using Taylor expansion. Our method is also related to Automatic Model Compression(AMe) (He et al. 2018b). In AMC, they use policy gradient to update the policy
2

---

Under review as a conference paper at ICLR 2021
Q1
Generate Architecture Vector
Layeri
Imput
a2
GRU
Sub-network Architecture
Channels
Output
It
map
min L + AR
Weight Temsor
Optimize HSN by evaluating the sub-network
Layer [+]
Hyper-Structure Network
Convolutional Neural Network
Figure 1: Overview of our proposed method. The width and height dimension of weight tensors are omitted. The architecture vector is firstly generated from fixed input a; i == 1...., L. Then, sub-network sampled according the architecture vector v. The parameters of HSN are updated by using gradients from the loss function when evaluating the sub-network.
network, which potentially provides both inter-channel and inter-layer information. However, the high variance policy gradient makes it less efficient and effective compared to our method. In this paper, focus channel pruning, since it provides natural way to reduce computation and memory costs.
Besides weight and channel pruning methods, there works from other perspectives, including bayesian pruning (Molchanov et al. 2017 Neklyudov et al. 2017), weight quantization (Courbariaux et al. 2015 Rastegari al.) 2016). and knowledge distillation (Hinton et al. 2015).
2.2 HYPERNET
Hypernet (Ha al. 2016) introduced to generate weights for a network by using hyper network.
Hyper networks have been applied to many machine learning tasks. yon Oswald et al. (2020) uses hyper network generate weights based on task identity to combat catastrophic forgetting in continual learning. MetaPruning (Liuu et al. 2019a) utilizes hyper network to generate weights when performing evolutionary algorithm. SMASH (Brock et al., 2018) is neural architecture search method that can predict the weights of network given its architecture. GPN (Zhang et al.) 2019) extends the idea SMASH and can be used on any directed acyelic graph. Other applications include Bayesian neural networks (Krueger et al. 2017), multi-task learning (Pan et al. 2018),, generative models (Suarez, 2017) and so on. Different from original hyper network, the proposed hyper-structure network aims to generate the architecture of sub-network.
PROPOSED METHOD
3.1 NOTATIONS
To better describe our proposed approach, necessary notations are introduced first. In a CNN, the feature map of ith layer can be represented by F, E xC; number of channels, W; and H, are height and width of the current feature map, the number of layers. The mini-batch dimension of feature maps is ignored to simplify notations. sigmoid(-) is the sigmoid function. round(-) rounds inputs to nearest integers.
I,... L. where C; is the
3.2 SYYPER-STRUCTOR NETWORK
In the context of channel pruning, we need to decide whether a channel should be pruned or not. We use 0 or I to depict the removal or keep of channel. Consequently, the architecture of sub-network can represented as concatenated vector (containing or 1) from layers. Our goal then to neural network generate this vector represent the corresponding sub-network.
3

---

Under review as a conference paper at ICLR 2021
x
layer-wise setting
unreachable
[a] Original FLOPs regularization
(b) FLOPs Layer-wise scaling
Figure 2: (a) For the original FLOPs regularization, some architectures may become unreachable. (b) After layer-wise scaling, the potential search space of architectures for sub-network is increased.
For ith layer, the following output vector is generated:
0 HSN(a;;;),
where HSN is our proposed hyper-structure network composed of gated recurrent unit (GRU) (Cho
et al. 2014) and dense layers, a; is a fixed random vector generated from uniform distribution U(, 1). and è is the parameter of HSN. The detailed setup of HSN can be found Appendix C. In short, GRU is used to capture sequential relationships between layers, and dense layers are capable of capturing inter-channel relationships. Note that a₂ is constant vector during training, a, is randomly sampled, it will make learning more difficult and result in sub-optimal performance.
Now we have the output Og, we need to convert it to a 0-1 vector to evaluate the sub-network. The binarization process can be demonstrated by the following equations:
2j si +
(2)
0 (2;)d and U2 E {0, {0,1)
where y follows Gumbel distribution: y Gumbel 1), is the architecture vector of ith layer, Since the round operation is not differentiable, we use straight through estimator (STE) (Bengio et al. 2013) to enable gradient calculation: 07 åz This process can be summarized as using ST Gumbel-Softmal (Jang et al. (016) with fixed temperature to approximate Bernoulli distribution. The idea of HSN can also be viewed as mapping from constant vectors {a, V to the architecture of sub-network. When evaluate sub-network, the feature map of ith layer is modified as follows:
and t is the temperature
f 0, Fi,
where @ is element-wise multiplication, 0, is the expanded version of v, and ú; has the same size of F. The feature map J; is from the output of Conv-Bn-Relu block. The overall loss function is:
min J(0) == ((((;; (,v v) + XR(T(v))
(4)
where v = (ei,..... Ur)). T(v) the current FLOPs decided by the architecture vector v. Tittl is the total FLOPs of the original model, p E (0, 1] is predefined parameter deciding the remaining fraction of FLOPs, À is the hyper-parameter controlling the strength of FLOPs regularization, / (x;; W, v) is the CNN parameterized by W and the sub-network structure is determined by architecture vector v. L the cross entropy loss function and is the regularization term for FLOPs, and θ again is the parameters of HSN. The regularization term R. used in this paper is R(T(v). (T).pTTi.i pTinal) log([T(v) waa + 1).
3.3 LAYER- WISE SCALING
The FLOPs regularization considered in Eq. will heavily penalize layers with larger amount of FLOPs (early layers for most architectures). Consequently, the resulting architecture from the original FLOPs regularization will have larger pruning rate at early layers. The alternative architectures with similar FLOPs could be omitted. This phenomenon is also demonstrated in Fig. 2 Further analysis is provided in Appendix B.
To alleviate the problem caused by original FLOPs regularization, we introduce layer-wise scaling factors dynamically balance gradients from the regularization term R. and the loss term L. Only
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Model Compression via Strper-Structure Network
Input: dataset for training HSN: DiSK: perversed rate FLOPs: p; hyper-parameter: training
epochs: ng: pre-trained CNN: f: learning rate & when updating {ax)==1.
Initialization: initialize randomly; initialize O₁ = 1, 11.=1,.... 1,...; freeze W
for e 1 to ng do
Shuffle(Dyss)
for a batch (z, v) in DHSN do
1. produce architecture vector from HSN (Eq. and
2. calculate gradients w.r.t θ (Eq. 5).
3. calculate hyper-gradient for O; (Eq. 6
4. update layer-wise scaling factor of a 1-1
-3
-......
Ba
5. update θ by ADAM optimizer.
end end
return HSN with the final e
gradients in dense layers are balanced since GRU is shared by all layers. The gradients w.r.t the parameters of ith dense layer can be written in the following equation:
aJ
dc
04 00,
(5)
00,
00,*
where 0; is the parameter of ith dense layer, as is the layer-wise scaling factor for ith layer. If no layer-wise scaling is applied, a, = 1. ag can be regarded as a balancing factor between 20, and ac 2e, @it
og only appears in gradient calculation, as a result, it can not be directly optimized. To optimize ag, we follow similar deriving process from (Baydin et al. 2018). We first define the update rule of = u(00 af) and can be applied to any optimization algorithms. For example, under stochastic aR 2R.z). Ideally, our goal is to update gradient descent, u(6" af) = 8%- naa; ac + X 50 a0
so that the corresponding architecture can obtain lower loss value with loss function J. To do so, want to min (u(((-)) a{)) before update 6,, For simplicity, the expectation is omitted. The hyper-gradient with respect to og can be calculated by:
aJ(u(0)-11 af)) da:
0J yr Ou
Ou(00 .aa)
0J 0u(00-2,a)) p
6
Oa:
00
Oog
Given the hyper-gradient of a,, it can be updated by regular gradient descent method. In experiments, the update rule is ADAM optimizer (Kingma & Ba 2014), and the detail derivation of aJ(a(e - .a()) for ADAM optimizer is described in Appendix E.
day
3.4 MODEL COMPRESSION VIA
NETWORK
In Fig. we provide the flowchart of HSN. The overall algorithm of model compression via hyper- structure network is shown in Alg. [ As shown in Alg. [ our method can prune any pre-trained CNNs without modifications. should be emphasized again that the gradient of GRU not affected by as, which is simply 82 Zonu + X HR
80am and OGRU is the parameter for GRU. Moreover, HSN does not need whole dataset for training, and small fraction of the dataset is enough, which makes the training of HSN quite efficient. After the training of HSN, we then use HSN to generate an architecture vector v, and prune the model according to this vector. Also, note that there is certain randomness (Eq. aaproximmates Bernoulli distribution) when generating v, but we find that there no need to generate the vector multiple times, and average them conduct majority vote. When generating the vector multiple times, most parts of vectors are the same, the different parts are trivial and do not have impacts on the final performance.
5

---

Under review as a conference paper at ICLR 2021
Method
Architecture
Baseline Acc
Pruned Acc
A-Acc
FL.OPs
Channel Pruning (He et al. 2017
92.80%
91.80%
-1.0%
50.0%
AMC (He et al. 20186
92. 90%
91.90%
-0.90%
50.0%
Pruning Filters (La ct al. 2017
93.04%
93.06%
+0.02%
27.6%
Soft Prunings (He
2018a
93.59%
93.35%
-0.24%
52.6%
ResNet-56
DCP (Zhuang et al. 2018
23.80%
93.59%
-0.31%
50.0%
DCP-Adapt (Zhuang ct al. 2018
23.80%
93.81 %
+0.01 %
47.0%
CCP Peng ei al. 2019
93, 93.50%
93.42%
-0.08%
52.6%
MCul(ours)
92.99%
93.23%
+0.24%
50.0%
WM (Zhuang ct al. 2018
94,47%
94.17%
-0.30%
26.0%
DCP
Zhuang el al. 2018
MobileNetV2
94.47%
94.69%
+0.22%
26.0%
MCI](ous))
94.23%
94.68%
+0.38%
40.0%
Table 1: Comparison results on CIFAR-10 dataset with ResNet-56 and MobileNetV2. A-Acc
represents the performance changes before and after model pruning. +/- indicates increase decrease compared to baseline results.
4 EXPERIMENTAL RESULTS
4.1 IMPLEMENTATION DETAILS
Similar many model compression works, CIFAR-I0 (Krizhevsky & Hinton 2009) and Ima- geNet (Deng et al. 2009) are used to evaluate the performance of our method. Our method requires one pypee-rrr p to control the FLOPs budget. The detailed choices of p are listed in Appendix F.
For CIFAR-10, 10, we compare with other methods on ResNet-56 and MobileNetV2. For ImageNet, we select ResNet-34, ResNet-50, ResNet-101 101 and MobileNetV2 as our target models. The reason we choose these models is because that ResNet (He et al. 2016) and MobileNetV2 (Sandler et al. 2018) are much harder to prune than earlier models like AlexNet (Krizhevsky al.) 2012) and VGG (imonyyn & Zisserman, 2014b). À decides the regularization strength our method. We choose À 4 in all CIFAR- 10 experiments and À for all ImageNet experiments.
For CIFAR-10 models, we train ResNet-56 from scratch following the pytorch examples. After pruning, finetune the model for 160 epochs using SGD with start learning rate 0.1, weight decay 0.0001 and momentum 0.8, the learning rate is multiplied by 0.1 at epoch 80 and 120. For ImageNet models, directly the pre-trained models released from pytorch (Paszke et al. [017] 2019). After pruning, we finetune the model for 100 epochs using SGD with start learning rate 0.01, weight decay 0.0001 and momentum 0.9, and the learning rate scaled by 0.1 at epoch 30, 60 and 90. For MobileNetV2 on ImageNet, we choose weight decay as 0.00004 which is the same with the original paper (Sandler et al. 2018).
For the training process of HSN, we use ADAM (Kingma & Ba 2014 optimizer with a constant learning rate 0.001 and train HSN for 200 epochs. Eq 2 is set as 0.4. The 3 for a; is chosen as 0.01, and a; is updated as shown in Alg. 10, 000 samples for for CIFAR-10 10 and ImageNet separately. stand-alone validation set is not necessary, all samples in DESN come from the original training set. codes this paper are implemented with pytorch (Paskke et al. 2017 2019). The experiments conducted on machine with Nvidia Tesla P40 GPUs.
To build dataset DHSN we random sample 2, 500 and the experiments, we found that
4.2 CIFAR-10 RESULTS
In Tab. we present the comparison results on CIFAR-10 dataset. Our method abbreviated as MCH (Model Compression via Hyper Structure Network) in the experiment section. For ResNet- 56, our method can prune 50% of FLOPs while obtain 0.24% performance gain accuracy. On MobileNerV2. method can obtain 0.38% gain in accuracy. Compared to all other methods, our method can achieve the best results. Our method can outperform the second best method (DCP-Adapt) method outperform the the second best method by 0. 0.16% while pruning 14% more FLOPs. For both models, our method performs much better than
by 0.23% ResNet-56. On MobileNetV2,
early methods (He et al. 2017
2018h 8b
Li et al. 2017
He al. 2018a). Our method can outperform

---

Under review as a conference paper at ICLR 2021
Method
Architecture
Pruned Top-1
Pruned Top-5
A Top-1
A Top-5
FLOPs
Pruning Filters di ctal
2017
72.17%
-1.0%
24.8%
Soft Prunings He
20188
71.84%
89.70%
-2.09%
-1.92%
41.1%
IE Molchanov
2019
ResNet-34
72.83%
-0.48%
24.2%
PPPM [He et al.
2019
72.63%
91.08%
-1.29%
-0,54%
41.1%
MCF]
72.85%
91.15%
-0.45%
-0.27 %
44.0%
11: Molchanuy
2019
74.50%
1..8%
45.0%
FFGM [Bc al.
2019
74.83%
92.32%
-1.32%
-0.55%
53.5%
GAL [ al 2019
71 80%%
80%
-4,5%
-2.015%
55.0%
DCP Zhuang a. 2018
74.95%
92.32%
-1.06
-0.61%
55.6%
CCP [Feng ai.
75.21%
92.42%
-0,94%
-0.45%
54.1%
ResNet-
MetaPruming
2071
75.40%
-1.20%
51.25%
GBN You ct nl.
2019
75. 78%18%
92.41%
-0.67%
-0.26%
55.1%%
HRank
74.98%
-1.17%
43.8%
Hinge LTeeaa
2120
74.70%
-1,40%
54.4%
LeGR (Chin et
2020
75. 75.30% 30%
-0.80%
54.0%
MCHio
75.60
92.67%
-0.55%
-0.20 %
56.0 %
Rethinking (C IE Molchanov et al.
2018
77.37%
-2.10%%
47.0%
2019
ResNet-101
77.35%
-0.02%
39.8%
FPG (He el al. 2019
77.32%
93.561
-0.05%
0.00%%
41.1%
MCHuuurs)
77.58
+0.21
+0.25 %
56.0%
MobileNetV2 0.75 (Sander al. AMC (He el al. 201
2018
69,80%
89.60%
-2.00%
-1.40%
30.0%
70.80%
-1.00%
30.0%%
MetaPruning (L.u0
20
71.20%
-0.80%
30.7 %
MobileNetV2
LeGR (Chin
71.40%
-0.40%
30.0%
MCH(ours)
71.54%
90.08%
-0.58%
-0.33%
30.1%
MCH(cos scheduler)
71.73%
90.17%
-0.39%
-0.24 %
30.1
Table 2: Comparison results on ImageNet dataset with ResNet-34, ResNet-50, ResNet-101 and MobileNetV2. A-Acc represents the performance changes before and after model pruning. +/- indicates increase decrease compared to baseline results.
CCP by 0.32% terms of A-Acc, which demonstrate that learning both inter-layer and inter-channel relationships better than only considering inter-channel relationships.
4.3 IMAGENET RESULTS
In Tab. 2. the results on ImageNet are presented, Top-1/Top-5 accuracy after pruning are presented.
Most our comparison methods comes from recently published papers including IE (Molchanoy et al. 2019), FPGM (He et al. 2019), GAL (Lin et al. 2019), CCP (Peng et al. 2019), MetaPrun-
ing (l al. 2019a). GBN You et al. 2019).
Hinge (Li et al., 2020), abd HRank (Lin al..
2020).
ResNet-34. Our proposed MCH can prune 44.0% FLOPs with 0.45% and 0.27% performance loss on Top-1 and Top-5 accuracy. Such result is better than any other method. Proposed MCH performs similarly compared to IE (Molchanov et al. 2019) in Top-1 and A Top- accuracy (72.85% 0.45% 72.83% 0.43%). while method can prune almost 20% more FLOPs. Given similar FLOPs pruning rate, our method achieves better results compared to FPGM (He et al., (009)) (-0.45%/-0.27% -1.29% -0..4% for A Top-1/A Top-5). Besides IE and FPGM, The margin between our method and rest methods are even larger.
ResNet-50. ResNer-50 very popular model for evaluating model compression methods. With such intense competition, our method can still achieve the best Top-1/Top- and A Top- I/A Top-5 results. The second best method in
of Top-1 accuracy MetaPruning (Liu et al. 2019a). which can achieve 75.40% Top-1 result after pruning. Our method outperforms MetaPruning by 0.20% in Top-1 accuracy while our method can prune 5% more FLOPs. MetaPruning utilizes hypernet to generate weights when evaluating sub-networks, however, such design paradigm prohibits MetaPruning be directly used on pre-trained models. The weights inherited from the pre-trained model might be one of the reasons why method can outperform MctaPruning. GBN (You et al., 2019) obtains the second-best Top-1 accuracy, however, the accuracy after pruning is quite low compared to other methods. Our method outperform GBN by 0.42% in Top-1 accuracy. Besides GBN and MetaPruning, our method can outperform two recent methods HRank (Lin et al. 2020) and Hinge (Li et al.) 2020) by 0.62% 0.90% Top-1 accuracy.
ResNet-101. For ResNet-101, our method can increase the performance the baseline model by 0.21% 6 and 0.25% Top-1 and Top-5 accuracy, while removes 56% of FLOPs. The second best
7

---

Under review as a conference paper at ICLR 2021
150
(a) A. ResNet
(b) A. MobileNetV2
(C) LWS. RosNet
(d) LWS. MobileNet22
Figure 3: (a.b): Effect of on the performance sub-networks. (c,d): Effect of layer-wise scaling on the performance of sub-network. All experiments are done on CIFAR-10.
Layens
(a) ResNet-56
(b) MohileNesV2
Figure 4: Layer-wise preserved rate with without layer-wise scaling (LWS) for ResNet-56 and MobileNetV2 on CIFAR-10.
method FPGM (He et al., 2019) can maintain the performance and reducing 41% of FLOPs. In short, compared to FPGM, our method can obtain performance gain while pruning 15% more FL.OPs.
Vo On Mob,leNetV2, we mainly compare with AMC (He et al. 2018b) and MetaPrun- ing (Liu et al., 2019a). Both of them can be regarded as representative works for AutoML related model compression methods (AMC uses reinforcement learning: MetaPruning uses evolutionary algorithm and hypernet.. Our method can achieve 71.5 71.54% Top-1 accuracy while pruning around 30% of FLOPs, which is 0.34% and 0.74% higher than MetaPruning and AMC. These results show that our method can outperform AutoML based methods.
In summary, method outperform these comparison methods and achieve the state-of the-art performance. These experimental results also indicate that inter-channel and inter-layer relationships should be considered when designing model compression methods.
4.4 EFFECTS OF LAYER-WISS WISE SCALING
We further study the impact of À and layer-wise scaling (LWS) when training HSN on CIFAR-10. In Fig. 3 ba,,), can see that changing does have a large impact the final performance of a sub-network, and our method is not sensitive to it. One possible reason is that adapts to when using ADAM optimizer. In general, we do not spend much time on tuning In Fig. (c,d), it shows that using LWS can improve the final performance of a sub-network and obtain lower loss. Moreover, early layers usually have larger preserved rate with LWS as shown in Fig 4 indicating that alternative nub-network architectures can be discovered from LWS. Without LWS, the final performance of ResNet-56 will decrease 0.19%, achieves 93. 93.04% final accuracy on CIFAR-10. Similar observations hold for MobileNetV2 (94.45% final accuracy and the relative gap is 0.16%). These observations show that LWS indeed helps the training of the HSN.
4.5 DETAILED ANALYSIS
In this section, provide detailed analysis to answer the following questions: (1) why fixed inputs for a;? (2) Can we replace HSN with dense layers? (3) Does LWS work for different learning rate settings? (4) Does LWS still work for other optimization methods?
8

---

Under review as a conference paper at ICLR 2021
(a) as. ResNet
(b) MobileNetV2
(c) Settings, ResNet
(d) Settings, MobileNetV2
Bacile
(c) Ir, ResNet
f lr, MobileNetV2
(g) SGD/LARS, RosNer
(h) SGDILARS, MobileNetV2
Figure 5: (a.b): Effect of different scheme for the inputs of HSN a;. (c,d): Effect of different settings of HSN. (e.f) Effect of different learning rates with LWS. (g.h): Effect of different optimizer on with LWS. For plots in (c.d.g.h), shaded areas represents variance from 5 trials.
To answer the first question, examine three different settings: learnable inputs, fixed inputs, and randomly generated inputs from the uniform distribution. From Fig. [5 (a,b). we observe that fixed inputs have similar performance to learned inputs, and both of them outperform random inputs. The idea of using fixed inputs is that we want to project the optimal sub-network to fixed vectors in the input space, which generally simple (compared to learned inputs) and easy train (compared to random inputs). The above results justify why we use fixed inputs.
To verify the effectiveness of different components of HSN, we use three different settings: vanilla HSN, HSN only with dense layers and gates (definition is given in Appendix). From Fig. (c.d), .), it can be shown that HSN has the best performance, which again shows that we should separately treat each channel or each layer.
In Fig. (e,f), we plot training curves for different learning rates with without LWS. It can seen that LWS can lead to better performance, given different learning rates. Finally. in Fig 5 (g.h). we examine whether LWS still useful given two additional optimizer: SGD and LARS You al.. 2017). LARS applies layer-wise learning rates overall gradients, which can be complementary to LWS. When applying LWS on these two methods, still improves performance. SGD is not good
choice when the optimization involves discrete values, as suggested by the previous study (Alizadeh
ctt al. 2019).
CONCLUSION
In this paper, we proposed hyper-structure network for model compression to capture inter-channel and inter-layer relationships. An architecture vector can generated from HSN to select sub- network from the original model. At the same time, we evaluated this sub-network by using classification and resource losses. The HSN can be updated by the gradients from them. Moreover, also identified the problem of FLOPs constraint (bias towards latter layers), which limits the final search space of HSN. To solve it, we further proposed layer-wise scaling to balance the gradients. With the aforementioned novel techniques, method can achieve ofte-off-the-rts performance ImageNet with four different architectures.
REFERENCES
Milad Alizadeh, Javier Fernández- Nicholas D. Lane, and Yarin Gal. A systematic study of binary neural networks' optimisation. In ICLR, 2019. URL https: /ttts:://ppezeevee. net/ foorm?? der

---

Under review as a conference paper at ICLR 2021
Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood. On- line learning rate adaptation with hypergradient descent. In International Conference on Learning Representations, 2018. URL
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 2013.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End end learning for self-driving cars. arXiv preprint arXiv: arXiv: 1604.03316, 2016.
Andrew Brock, Theo Lim, J.M. Ritchie, and Nick Weston. SMASH: One-shot model architecture search through hypemetworks. International Conference on Learning Representations, 2018.
URL WKKLLttpps//ppeereveee.. http:/ieewweee///oeeeeeee Eorum? id=rydecEss- ror.
Ting-Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculeseu. Towards efficient model compres- sion via learned global ranking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1518-1528, 2020.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv: 1409. /259, 2014.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances neural information processing systems, pp. 3123-3131, 2015.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Patlern Recognition, 2009, CVPR 2009. IEEE Conference on, pp. 248-255. leee, 2009.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https
//openreview.co net/ httt:/foommmdddr---oo0
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint 1xXx::6609009000 2016.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffiman coding. arXiv preprint arXiv:1510.00145 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015b.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In International Joint Conference Artificial Intelligence (IJCAI), pp. 2234-2240, 2018a.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4340-4349, 2019.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017.
Yihui He, Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration mobile devices. In Proceedings of the European Conference Computer Vision (ECCV), pp. 784-800, 2018b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint /503.00 2015.
10

---

Under review as a conference paper at ICLR 2021
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1700-4708, 2017.
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pp. 304-320, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning Volume 37, ICML, pp. 448-456. JMLR.org, 2015.
URL http://dl. acm. org/ citation...... cfmmii==345111 3045167
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatioo with gumbel-softmax. arXiv preprint arXiv: 1.01 144, 2016.
Jaedeok Kim, Chiyoun Park, Hyun-Joo Jung, and Yoonsuck Choe. Plug-in, trainable gate for streamlining arbitrary neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:14/2.698) 2014.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks. arXiv preprint arXiv: 2017.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based connection sensitivity. ICLR, 2019.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. ICLR, 2017.
Yawei Li, Shuhang Gu, Christoph Mayer, Luc Van Gool, and Radu Timofte. Group sparsity: The hinge between filter pruning and decomposition for network compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8018-8027, 2020.
Mingbao Lin, Rongrong Ji, Yan Wang. Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Shaohui Lin, Rongrong Ji, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, and David Doermann. Towards optimal structured enn pruning via generative adversarial learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2019.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE International Conference on Computer Vision. pp. 3296-3305, 2019a.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019b. URL
net. / forum? wiw.mmmdddrnnnncC
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through lo regularization. In International Conference on Learning Representations, 2018. URL https
7/operrevies net./ ott.r//eewwe..eeeoooooomoooooooe
11

---

Under review as a conference paper at ICLR 2021
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2498-22917 JMLR. org, 2017.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2264-11272, 2019.
Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry P Vetrov. Structured bayesian pruning via log-normal multiplicative noise. In Advances in Neural Information Processing Systems, pp- 6775-6784, 2017.
Zheyi Pan, Yuxuan Liang, Junbo Zhang, Xiuwen Yi, Yong Yu, and Yu Zheng. Hyperst-net: Hyper- networks for spatio-temporal forecasting. arXiv preprint arXiv: /809. 10889, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems, pp. 8024 8035, 2019.
Hanyu Peng, Jiaxiang Wu, Shifeng Chen, and Junzhou Huang. Collaborative channel pruning for deep networks. In International Conference on Machine Learning, pp. 5113-5122, 2019.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, pp. 525-542. Springer, 2016.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chied Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510 45200, 2018.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in neural information processing systems, pp. 568-576, 2014a.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.15566 2014b.
Joseph Suarez. Language modeling with recurrent highway hypernetworks. In Advances in neural information processing systems, pp. 3267-3276. 2017.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. Continual learning with hypernetwokss. In International Conference Learning Representationss 2020. URL
net wtwp://wnnneeetoo.iieeee.....
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.
Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the www.-noos------eeeeeee-e assumption in channel pruning of convolution layers. In International Conference on Learning Representations, 2018.
12

---

Under review as a conference paper at ICLR 2021
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv 2017.
preprint
Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 2130-2141, 2019.
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. In International Conference on Learning Representations, 2019. URL /RLLittts://peeeeeeiee net pet/fooa 2id-rkg0oAA-r-1
Dejiao Zhang, Haozhu Wang, Mario Figueiredo, and Laura Balzano. Learning to share: Simultaneous parameter tying and sparsification deep learning. 2018.
Zhuangwei Zhuang. Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-awen channel pruning for deep neural networks. Advances in Neural Information Processing Systems, pp. 875-886, 2018.
13

---

Under review as a conference paper at ICLR 2021
A VISUALIZATION OF PRUNED ARCHITECTURES
In Fig 6 we visualize the pruned architecture for ResNet-50 and MobileNetV2.
B BIAS OF FLOPS REGULARIZATION
We briefly discuss two types of FLOPs regularization used in our paper and trainable gate (TG) Kim al. 2020). First, we provide the specific definition of Tw::) (FLOPs of ith layer):
T(w:) = K? 12110, Us 1,,,,,,
9i
where Gü is the number of groups in a convolution layer, K; is the kernel size, is a all one vector, and 1 1" is the number of perversed channels in ith layer. With T(u;). T(v) E/= T(u:). In TG, they simply mean square enror (MSE) the regularization term, and in their paper RMSE(T(V) pT = (T(v) - pT (aal)) ). The gradients w.r.t 0 is:
ÖRMS 2(((Tv) pT
oT(s))
(
Ov:
For the regularization used in our method: w.r.t 0 is:
C(T(V),PTTonl log([T(() - pPTioal | + 1), the gradients
0R
((v) wotl
(T(e))
(9)
0c. ((() - ioaal + [T(v) pfoioaal du
For both regularization functions, the ratio between the gradients w.r.t 8 two layers k,j is
K²- -1 W.H. Take ResNet-50 K2 ----ww,, W,H; of a bottleneck block, and we random initialize HSN. If j is in the first block, and le is in the last block, then K2 Kj 3. W Hj 56, We H4 7, 1 U3-1 0.5 x 64 (due to random initialization). 1 1fus- Ux-1 ~ 0.5 x 512, finally, (T(ve) (v)) 3x3x256x7w7 trivial.
(T(v)) Dea
oT(e))
example, let j, l be the middle layers
Deg
R g. which is not
de
des
3x3x32x56x56
When calculating the gradients w.r.t O,, we have 2R 80,
(r Ses 0, share the same eR decided = CR de, 30,
by the regularization function. Without loss of generality, assume the magnitude of äv. is similar given different layers. The assumption is based the following derivation (to simplify derivation, we omit weight norm in dense layers):
Dv, dz,
00,
a0,
dz do:
Do: 00,
do: 00,
1 47
+ ( (ggmoid((:++9)/1))
<
bj.
where sigmoid(z)(1 sigmoid(z)) 1 and 6; is the input ith dense layer, which is also the outputs of GRU. Since all 6; have the same shape, and weights in GRU are normalized, we can assume all b, have similar magnitude. Since + of is a upper bound of 8mg similar assumptions can be made.
88,
Following this assumption, the relative magnitude of gradients w.r.t 0, and 0g for layers j. k can be roughly represented by (T(m)) oT(e,) After training for a while, the ratio might be smaller, however, it only indicates that early layers are more aggressively pruned. Thus, when applying FLOPs regularization, penalizes early layers much heavier compared to latter layers.
des
One should also note that this is general problem when using gradient based model compression methods with the FLOPs regularization. It's quite hard to circumvent calculating @T(e))
in
De,
TG (Kim et al. 2020) and our paper.
14

---

Under review as a conference paper at ICLR 2021
040
Charrll Chatt
020
300
46.0
Layers
Layens
(a) ResNet-50
(b) MobileNetV2
1063
523
300
803
Demaaee: Charmes
713
600
Laye'S
Lavers
(c) R5
(d) MobileNetV2
Figure 6: (a,b): visualization of pruned architectures for ResNet-50 and MobileNetV2. (c,d): mean and variance of 20 generated sub-networks for pruning.
(2) Acc,
MobileNetV2
(c) ResNet
MobileNetV2
Figure 7: (a,b): Performance of sub-networks when using HSN or not using HSN (the setting in Eq. Th.. (c,d): Regularization loss for the same settings.
C DETAILED SETUP OF HYPER-STRUCTURE NETWORK
In Tab. 3 we present the architecture of HSN. The forward calculation is:
by h; = GRU(a, h4-1)
(10)
dense;(b.)
where h; and are hidden states and outputs of GRU step o; is the final output of HSN. GRU also requires hidden layer input at time-step /g. In the experiment, the ho a all zero tensor. As mentioned in Tab, 3 the dimension of a, 64. Since a; is single input instead of mini-batch. we cannot apply batchnorm. To make the training more stable, we use weight norm (Salimans & Kingma 2016) both GRU and dense layers.
Initially, we tried to use a huge dense layer (input size 64, output size C, + C2 +... Ct)) as HSN. However, find that the huge dense layer is hard optimize and also parameter heavy.
To verify the strength of the proposed HSN, we can instead use a simplified setting to prune neural networks, which shown as follows:
& sigmoid - +
(11)
6 round(2;), and 0, E {0.1)
15

---

Under review as a conference paper at ICLR 2021
Inputs 0; i
128)6 WeightNorm, Relu
dense;(128,C.), WeightNorm, i=1, L
Outputs o,, i=l,
Table The structure of HSN used our method.
Acc. ResNet
MobilaNetV2
ResNet
MobileNetV2
Figure 8: (a.b): Performance of sub-networks when training HSN given forward (c=0) and backward pruning (c=3). (c,d): Regularization loss of sub-networks when training HSN given forward (c=0) and backward pruning (c=3). All experiments are done on CIFAR-10.
where the architecture vector is parameterized by 0. Under this setting, the parameter for each channel does not have relationships. We use this setting prune ResNet-56 and MobileNetV2 CIFAR-10, 10, the results are shown in Fig. [ From the figure, we can see that the performance and convergence speed of using HSN much better. Under high dimensional setting, like MobileNetV2. the simplified setting shown in Eq. 11 can not learn efficiently, which demonstrate that capturing inter-channel and inter-layer relationships are crucial for pruning deep neural networks.
D FORWARD AND BACKWARD PRUNING
Here, we refer forward pruning as start pruning from a random sub-network, and refer backward pruning as start pruning from the original large model. Many model compression methods use backward pruning. We also provide simple way to extend our method to backward pruning. When we binarize the output of HSN, we can add constant
24 sigmoid((o: + (g + c))/7),
(12)
V; ro;nd(:;), and v, 1
where g ~ Gumbel(O, 1), and the Gumbel(O, 1) distribution can be sampled using inverse transform sampling by drawing
24(0, 1) and computing 9 log( log(u)) When the constant c is big enough, will make U; become an all one vector, thus the sub-network produced by HSN will start from the original large CNN. If we show the results of forward and backward pruning. It can be seen that they can achieve similar sub-network performance, but the changes in regularization loss various dramatically.
O, then it will start from random sub-network. In Fig. 8
E DERIVATIVE OF HYPER-GRADIENT WITH ADAM OPTIMIZER
The update rule of ADAM for 0, is shown Alg. and is:
u(0 o 0.-1 mprne (vri/ +))
(13)
16

---

Under review as a conference paper at ICLR 2021
Algorithm 2: ADAM optimizer for 0;
Input: 7), 31, 82 E [0, 1): learning rate and decay rate for ADAM. Initialize mg, no, f
Update rule at step
mh 31m--1 + (1 (a)(a} 0c
OR
00
80
T2 = 82n4-1 + (1 - (a)
000 -1 +
DC
OR
)
"00
rit m²/( (1 1 B4)
f. nt/C (1 (5)
O (0, ,0,) 0:-11 m/rt// (Viig +
10,--,
Then the derivation of Ou(0: 1,a!)
is:
day
0u(05-1,00
0 (mr//
(V +c)
-1
da;
dag
W/ntte 1h + Briy (v +
Do:
Da.
(viz + c)²
Bia me Ba:
=-7( 111 +
Do:
2V/ul Vin + +)2
(1 (1) a6
OC
(1- (2)
OC
2 + X OC
3R
)ri
00:
20;
a00
(1 Bi) 14 +E)
v Vhllv + e)
32)
(1 91)
2(1 (21-8)((1)
(a)1
where Om
20
and
0A Dac
06 80
Recall that when updating
Das
1-#[
1-33
0 to a have compute:
= (J((( 2 aa)
aJ T
0.-- 3(
(14)
da,
-00-1
Oo:
We need af when updating 0: to 0f. Thus at each update step, it requires an extra copy of 0 and parameters of ADAM to compute
au(g a. 1 The cost of the extra storage is trivial.
8ea;
F CHOICE OF p GIVEN DIFFERENT DATASETS AND ARCHITECTURES.
Dataset ResNet-56
CIFAR-IO
ImageNet
Architecture
MobileNetV2
ResNet-34 0.55
ResNet-50
ResNct-101
MobileNetV2
0.50
0.60
0.38
0.42
0.64
Table 4: Choice of p for different models. p is the remained FLOPs divided by the total FLOPs
In Tab. we list the choices of p for different models and datasets used our experiments.
REFERENCES
Jaedeok Kim, Chiyoun Park, Hyun-Joo Jung, and Yoonsuck Choe. Plug-in, trainable gate for streamlining arbitrary neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Tim Salimans and Durk Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in neural information processing systems, pp. 901-909, 2016.
17


Reviews:
[{"review_id": "wgwuqqkJ_nZ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThis paper proposes hyper-structure network for model compression (channel pruning). The idea is to have a hyper-network generate the *architecture* of the network to be pruned. To do so, the proposed approach use Gumbel softmax together with STE to get around the non-differentiability issue of such design. Additionally, the proposed approach dynamically adjust the gradient for each layer so that earlier doesn't get over-regularized due to the FLOPs regularizer. Empirical results are presented showing better performance compared to alternatives with ablation study on the proposed components (hyper-network and layer-wise scaling).\n\n## Reasons for score\nI'm leaning toward a rejection. I like the idea of both layer-wise scaling and using GRU to model inter-layer dependencies. However, I find them under-studied as the novel components of the paper. While this paper provides seemingly good performance compared to prior methods, they are not really apple-to-apple comparisons, which makes them relative weak signals. Moreover, some of the recent papers that are closely related to this submission are missing. I'm willing to raise my score if the weaknesses I listed below are properly addressed during the rebuttal period.\n\n## Strengths\n\n- The paper proposed a novel formulation towards the channel pruning problem. Specifically, the novelty lies in using GRU with the proposed layer-wise scaling.\n- The results seem good compared to prior literature (with a caveat of having longer training iterations)\n- Ablation of the proposed method ($\\lambda$ and LWS)\n\n## Weaknesses\n\n- The novel aspect of the work, namely the layer-wise scaling, can be discussed in more detail. For example, why do we expect modifying the scaling of layer-wise gradient to make a better gradient than analytical gradient? In my understanding, analytical gradient gives you the steepest descent direction. In this case, why do we expect we can do better by tuning $\\alpha_i$? It seems this is suggesting that we need layer-wise learning rate and such learning rates can be optimized via gradient descent by meta-gradient. It is not clear to me why such a formulation is specific to pruning. Can it benefit training vanilla network by setting $\\lambda=0$? Overall, it is a bit mysterious to me why such a formulation *accelerates* training for equation (4), which is empirically shown in Figure 3 (c)(d). A more in-depth theoretical analysis would be very helpful. Without theoretical analysis, I'd be curious to see more settings empirically. If we sweep multiple learning rates, can LWS stop being better? If we use a different optimizer (say SGD), is LWS still better? Most importantly, if we use LARS optimizer [1], is LWS still better?\n\n- Missing AutoML-based related methods that have strong performances [2-5]. Discussion with these prior methods are needed to better position the proposed method. Comparing with these methods, the proposed method is only comparable. Specifically, DMCP [4] has 47% reduction with top-1 of 76.2 for ResNet-50 and 30% reduction with top-1 of 72.2 for MobileNetV2. Putting these methods into the table and discussion is necessary.\n\n- Comparison with AMC is limited to numbers from the previous paper. From the formulation of this paper, AutoML-based methods are highly relevant. It would be better if AMC is compared with this paper by using the same empirical setup. This is in fact doable as AMC has open source code. Without such a fair comparison, it is hard to understand what are the benefits of the proposed approach over AMC that solves the exact same problem. The paper has argued in the related work that policy gradient is noisy without really providing the quantitative evidences. The numbers from the paper is a really weak signal as AMC only fine-tunes for 30 epochs while this paper fine-tunes for 100 epochs for ImageNet.\n\n- The other novel aspect of the paper is using GRU for designing the network. However, it is not clear if GRU is necessary and why it is a good design choice. The argument for GRU is cross-layer dependences. I'm wondering what the results would look like if we simply use FC for each layer independently. This can better motivate the so-called cross layer dependencies and better motivate the adoption of GRU.\n\n\n[1] You, Yang, Igor Gitman, and Boris Ginsburg. \"Large batch training of convolutional networks.\" arXiv preprint arXiv:1708.03888 (2017).\n\n[2] Yu, Jiahui, and Thomas Huang. \"AutoSlim: Towards One-Shot Architecture Search for Channel Numbers.\" arXiv preprint arXiv:1903.11728 (2019).\n\n[3] Berman, Maxim, et al. \"AOWS: Adaptive and optimal network width search with latency constraints.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n[4] Guo, Shaopeng, et al. \"DMCP: Differentiable Markov Channel Pruning for Neural Networks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n[5] Chin, Ting-Wu, et al. \"Towards Efficient Model Compression via Learned Global Ranking.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n\n--------- Post rebuttal ---------------\n\nI've read the rebuttal and I appreciate the additional efforts by the authors.\n\nSpecifically, the authors have addressed my concerns comparing LWS (the proposed method) and LARS. Additionally, the authors have addressed my concerns regarding LWS by conducting more experiments. With another detailed read, I figured LWS updates $\\alpha$ in a lookahead fashion. Specifically, $\\frac{\\partial \\mathcal{J}}{\\partial \\mu}$ in Eq. 6 essentially requires one to compute the loss after the gradient update is being made, which gives it the potential to outperform the analytical gradient.\n\nMoreover, the authors have run additional experiments to demonstrate the usefulness of GRU, which makes the proposed method more convincing.\n\nWhile the authors argued that it is not fair to compare to AutoSlim, AOWS, and DMCP, I disagree. They are all relevant and strong channel pruning methods and the authors should have cited them and discuss the main differences (can be used to prune a pre-trained model or not) in the related work as opposed to omit them entirely.\n\nOverall, I find the paper interesting and it provides several novel aspects: GRU and LWS. Both are empirically verified to be useful in the channel pruning setting. However, the related work section can be further improved. As a result, I raised my score to weak accept.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "V1wxfEp22oo", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper propose to utilize a hyper-structure network to generate the architecture of the main network, which can help to capture inter-channel and inter-layer relationships. The learnable layer-wise scaling factors are introduced to balance the gradients from different terms. BasisNet can be applied to any network architectures. MCH shows state-of-the-art ImageNet performance in pruning setting.\n\nMy main concern is about the novelty of the paper. The idea to use a hypernet to help pruning is not new. MetaPruning uses a hypernet for automatic channel pruning [1]. GaterNet use a gater network to generate pruning mask for each input image [2]. This paper use a hypernet to generate mask to prune the main network, which can be viewed as a weak version of GaterNet where the mask for each sample is the same. \n\nIn addition, Eq.5 involves 2 hyperparameters, i.e. $\\lambda$ and $\\alpha_i$. $\\lambda$ is set by hand and $\\alpha_i$ is learnable. The two hyperparameters may be deprecated, that is, you only need to preserve $\\alpha_i$ and $\\lambda$ can be merged into $\\alpha_i$?\n\nOverall, I think this paper below the bar of ICLR due to the above weakness.\n\n[1] MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning. ICCV 2019.\n[2] You Look Twice: GaterNet for Dynamic Filter Selection in CNNs. CVPR 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SHxFtsKOFZ0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a channel pruning method to compress and accelerate pre-trained CNNs. This paper proposes to use gated recurrent unit (GRU)-based hyper-structure networks (HSN) to generate channel importance, and Gumbel-rounding operations are subsequently applied to the importance values to generate stochastic masks to prune a CNN. The authors further introduced layer-wise trainable scaling with a modified update rule to accelerate convergence. While the tricks employed above are not original on their own, the combined method is still a novel amalgam, which learns inter-channel and inter-layer relationships in a unified package. Experiments on CIFAR-10 and ImageNet with ResNet and MobileNet-V2 showed that this method is competitive with existing state-of-the-art (SOTA).\n \n### Advantages \n* The combined method is novel, as it:\n\t* uses HSN to learn to prune without retraining models; and\n\t* learns inter-channel and inter-layer relationships in a unified HSN.\n* Good results that are competitive with the SOTA. \n \n### Issues\n1. Section 3.2, “This process can be summarized as using Gumbel-Softmax distribution (Jang et al., 2016) to approximate Bernoulli distribution.” This is not true, as equation (2) clearly uses the “round” operation which gives discrete outputs, while the Gumbel-softmax approximation by Jang et al. is continuous with temperature annealing.\n2. Section 3.2, the authors mentioned that $a_i$ is drawn from a uniform distribution but remains constant during training. What is the purpose of having only constant values that are not even hyperparameters as the inputs to the HSN? This seems unnecessary and introduces unwanted variance.\n3. Section 3.4, “When generating the vector multiple times, most parts of vectors are the same, the different parts are trivial and do not have impacts on the final performance.” Why? More results are necessary to verify this.\n4. The experiments are not run multiple times to give mean and standard deviation results.\n \n### Other minor issues\n* Section 2.1, “A straight forward way to”, should be “straightforward”.\n* Figure 1 is not mentioned in text.\n* In Section 4.4, “In Fig. 4 (c,d),”, should be Fig.3 (c,d).\n* It is not apparent if layer-wise scaling was used for ImageNet models.\n\n### Summary\nWhile this method proposed in this paper is novel and the results are competitive, the reviewer believes that in its current state, this paper lacks additional experiments to address issues 2, 3 and 4 mentioned above.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Gkg5k0fRqZF", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work explores the model compression problem with a hyper-network to adaptively determine the preservation of inter-channel and inter-layers. To this end, they use a shared-GRU layer to explore the relations in consecutive layers and then FCs to generate the coefficient vectors which indicate the preserved rates within each layer. As a result, the inter-layer relation can be given from GRU and inter-channel relation can be given from FCs. In this paper, a learnable factor in each layer is also involved in the network optimization to better balance the classification performance and FLOPs regularization. Experimental results also validate the performance of the proposed method.\n\nOverall, I think the paper is easy to follow and the derivation is also clear to understand. Nonetheless, there are still some problems that need to be further explored, given as follows:\n\n1. This paper only investigates the channel pruning, but the title uses the \"model compression\". This might be not very specific since model compression serves as a general topic and consists of many approaches, such as channel pruning, quantization, knowledge distillation and tensor decomposition. \n\n2. This paper also uses FCs to generate the coefficient vectors (ideally 0-1 code) to indicate the importance for each channel. However, this practice has been investigated in other channel pruning papers and is thus not new. Just to name a few,\n\n[NeurIPS2019] AutoPrune - Automatic Network Pruning by Regularizing Auxiliary Parameters  \n[Pattern Recognition] AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inference\n\n3. In subsection 4.4, the experiments on Figure (a)-(b) reveal that “changing $\\lambda$ does not have a large impact on the final performance of a sub-network, and our method is not sensitive to it”. I think the reason why the method is not sensitive toward $\\lambda$ should be further analyzed. As in the appendix (E) shows, in Eq.(8), the update of $\\alpha$ contains $\\lambda$, so $\\alpha$ can actually be reviewed as the adaptive variable, and for different $\\lambda$ during the network training, the value of will gradually converge to the corresponding scale which considers $\\lambda$.\n\n4. I think there are some mistakes in analyzing the bias of FLOPs regularization. As stated in appendix (B), the relative magnitude of gradients w.r.t  $\\theta_k$ and $\\theta_j$ can be approximately estimated. However, since the ratio value of $v_k$ and $v_i$ is 8, the conclusion about \"the gradient in the early layers are larger than that of the latter layers\" does not hold. I wonder whether it is a typo here. Furthermore, can you explain why the assumption “the magnitude of  $\\partial v_i \\partial \\theta_i$ is similar given different layers” holds herein? \n\n5. In subsection 4.4, “In Fig.4(c,d) should be changed into “In Fig.3(c,d)” instead.\n\n6. How to choose the hyper-parameter p?  \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a channel pruning method to compress and accelerate pre-trained CNNs.
The reviewers suggest further analysis of the experimental results to help explain the gains in performance, as well as point out some errors in the formulation. The paper is also found similar to meta-pruning method. The authors are encouraged to re-submit the paper after adding the analysis and improving the related work section. 
Decision: Reject