Locally Free Weight Sharing for Network Width Search
Authors: Xiu Su
Shan You
Tao Huang
Fei Wang
Chen Qian
Changshui Zhang
Chang Xu
Venue: ICLR 2021 Spotlight (2021)

Abstract: Searching for network width is an effective way to slim deep neural networks with hardware budgets. With this aim, a one-shot supernet is usually leveraged as a performance evaluator to rank the performance \wrt~different width. Nevertheless, current methods mainly follow a manually fixed weight sharing pattern, which is limited to distinguish the performance gap of different width. In this paper, to better evaluate each width, we propose a locally free weight sharing strategy (CafeNet) accordingly. In CafeNet, weights are more freely shared, and each width is jointly indicated by its base channels and free channels, where free channels are supposed to locate freely in a local zone to better represent each width. Besides, we propose to further reduce the search space by leveraging our introduced FLOPs-sensitive bins. As a result, our CafeNet can be trained stochastically and get optimized within a min-min strategy. Extensive experiments on ImageNet, CIFAR-10, CelebA and MS COCO dataset have verified our superiority comparing to other state-of-the-art baselines. For example, our method can further boost the benchmark NAS network EfficientNet-B0 by 0.41\% via searching its width more delicately.

Published as a conference paper at ICLR 2021
LOCALLY FREE WEIGHT SHARING FOR NETWORK WIDTH SEARCH
Xiu Su', Shan Tooo Huang", Fei Wang", Chen Qian", Changshui Zhang', Chang Xu' 1 School of Computer Science, Faculty Engineering, The University of Sydney, Australia TemeseTime Research
'Institute for Artificial Intelligence, Tsinghua University (THUAI)
Beijing National Research Center for Information Science and Technology (BNRist)
Department of Automation, Tsinghua University, Beijing, P.R.China xisu59 (ou55922uuna 92@uni. sydney edu, au, youshan@senset ime. com,
(huangLao, wangfei, qianchen @senseauto. com, zesêmall ts inghua. edu.en, c. xuuusydnyy, , edu. au
ABSTRACT
Searching for network width is an effective way to slim deep neural networks with hardware budgets. With this aim. one-shot supernet is usually leveraged as a performance evaluator to rank the performance w.r.t different width. Nevertheless, current methods mainly follow a manually fixed weight sharing pattern, which is limited to distinguish the performance gap of different width. In this paper, to better evaluate each width, we propose a loCAlly FrEe weight sharing strategy (CafeNet) accordingly. In CafeNet, weights more freely shared, and each width is jointly indicated by its basc channels and free channels, where free channels are supposed to locate freely in a local zone to better represent each width. Besides, we propose to further reduce the search space by leveraging our introduced FLOPs-sensitive bins. As a result, our CafeNet can be trained stochastically and get optimized within min-min strategy. Extensive experiments MS COCO dataset have verified our superiority comparing to other state-of-the-art baselines. For example, our method can further boost the benchmark NAS network EfficientNet-BO by 0.41 % via searching its width more delicately.
ImageNet, CIFAR-10, CelebA and
INTRODUCTION
Deep neural networks are easily constructed by stacking multiple layers of non-linearitin on top of each other. But it is much more delicate job than it seems. Before going into the deep, we have to determine the width each layer' (Chen et al., 2020) be stacked, especially given the consideration of the computational budgets (He et al., 2016; Han et al., 2015; loffe & Szegedy, 2015; Huang et al., 2017) (e.g., FLOPs, and latency) and the resulting network performance. The width of a pretrained peeepparameterieed neural network can be further trimmed by classical channel pruning techniques (Liu al., 2019b; He al., 2017: Tang et al., 2019) climinate the redundancy. Inspired by the spark of neural architecture search (NAS) (Liu et al.; Cai et al., 2019; Hu et al., 2020; Yang et al., 2020; Huang ct al., 2020; Yang et al., 2021), recent methods attompt to directly scarch for the optimal network width for given network, and have achieved remarkable performance in various FLOPs levels accoleration rates, such as AutoSlim (Yu & Huang, 2019), MctaPruning (Liu al., 2019a) and TAS (Dong& & Yang, 2019). Then the obtained models can deployed in edge devices, or even boosted by other compact techniques, such as quantization and knowledge distillation Hinton al. (2015); You et al. (2017); Kong et al. (2020); Du et al. (2020).
To determine the optimal network width, we need to evaluate and compare the performance of different network width. One of the most straightforward yet authentic approaches is to examine their training-fromm from-scratch performance and traverse all possible settings of network width (not exceeding a maximum allowable width). However, exhaustive training is computationally unaffordable due to
"Corresponding author. Otter literature also use the number of channels/álteer indicate the network width.

---

Published as a conference paper at ICLR 2021
Inpus layer
Lager with
c-l
612
c-5
weights patiom
Convolutionl layer
Bas channels
weight
Output loyer
(a) Toy example
(b) Channel assignment sub-network w.r.f. differemn weight sharing patterns
Figure 1:
(a) A toy example of the supernet. We only search for the width for its one convolutional layer with all 6 channels. (b) Examples of different weight sharing patterns. For the fixed weight sharing pattern, leftmost channels are assigned for the sub-network of width For the locally free weight sharing patter,, the offsct of free zone set to r 1.
the huge search space. Taking MobileNetV2 as an example, 100 channels in each of the 25 layers result in 100gh possible network architectures. For sake of scarching efficiency, current methods Chen al. (2020); Yu Huang (2019); Dong & Yang (2019) usually follow weight sharing strategy by leveraging one-shot supernet, with various sub-networks sharing the same weights with the supernet. Then each network width can be elficiently evaluated by querying the performance of its corresponding sub-network in the supernet.
In this way, how to specify the sub-network for each network width indicates the search space and matters for the performance evaluation. Without loss of generality, we count channels in a layer from the left. A popular weight sharing strategy for the supernet then follows a fixed weight sharing pattem. which simply assigns the left e channels as the sub-network for the width e (see the gray dots in Fig. 1(b)). In this manner, the search space of a layer with channels is O(n). However, this fixed pattern imposes inherent constraint the search space. Any two sets of width fully share their weights, limiting the accurate evaluation ability of supernet and affecting the searching performance accordingly. Ideally, each channel from the supernet shall enjoy the freedom to be selected, i.e., any channel has a probability to occur in the subnetwork, no matter it locates at the leftmost or the rightmost of layer. But the price of this full channel freedom is the dramatic increase of the search space size to O(2"), which affects the feasibility of width search. Instead of this unaffordable full freedom, we prefer restrictions and regularization over the search space to keep the channel freedom within an acceptable range, which benefits the efficiency of width search.
In this paper, we propose loCAlly FrEe weight sharing strategy of the supernet (i.e., CafeNet) for network width search. For each width c, our goal is specify sub-network within supernet by assigning channels, indicated by an index set I(c). To balance the weight sharing freedom and the scarching efficiency, wc split the index set I(c) into two parts, i.e., G. base channels Iy(c) and ej free channels I (c). with Z(c)= [c) (I) and = G + cy. (see Fig. ((b)). In detail, base channels follow the fixed weight sharing pattern, while free channels
encouraged select freely from a given zone, which is the neighborhood of the e-th channel. this way, each width e can be better evaluated by multiple locally free sub-subnetworks. The supernet CafeNet can be trained by optimizing the sub-network with the minimum loss for each width and scarch for the one with the maximum accuracy. To further ease the burden of scarching cost, we take group of channels (i.e., bin) as the minimal searching unit. The bin sizes are carefully set by encouraging FLOPSs to be more evenly allocated over bins. We conduct extensive experiments on the benchmark ImageNet (Deng et al., 2009) and CIFAR-10 (Krizhevsky ct al., 2014) datascts to validate the superiority of our proposed CafeNet. The effectiveness of our CafeNet is also examined on the face attribute recognition lask CelebA (Liu et al., 2015) dataset and the transferability of our searched backbone for object delection task on MS COCO dataset (Everingham et al., 2010).
2 PROBLEM FORMULATION
With given network structure, the aim of the network width search to optimize its width for each layer under certain budgets. Suppose the target network has L layers, then the network width can be represented as a L-length tuple of layer-wise width = (C1, cz, in which represents the width at the i-th layer. Without loss of clarity, we simply focus searching for the optimal layer width of c. Denote the maximum allowable width at the i-th layer as ny. Then the number
of all sub- networks in the supernet amounts to I12_ (2 1), which is absolutely huge. Current
2

---

Published as a conference paper at ICLR 2021
methods mainly use weight sharing stralegy to boost the searching efficiency. In the lypically fixed weight sharing pattern, each layer width & E [1: nj] = 2,, ?4 is indicated by one and only one sub-network (see (bb), and all layers within the sub-network adopt the same weight sharing pattern, then the size of search space C is reduced to I R2.
After specifying each width with its corresponding sub-network, we can evaluate it by examining the performance of its sub-network in the supemeet with shared weights. Then the whole searching consists of two steps, i.e., superet training and searching with supemet. Usually, the original training datasct is split into two datasets, i.e., training datasct D,, and validation dataset Deat. The one-shot supernet N with weights W is trained by randomly sampling a width c and optimizing its corresponding sub-network with weights we C W W.e.,
W arg min
E WcCW W CEU(C)
[Ceraian (Crraen(Ws::N c, DirMl
(1)
where (C) is a uniform distribution of network width in search space, and [8[] the expectation of random variables. After the supemet is trained, the performance (e.g., classification accuracy) of various network width is indicated by evaluating its sub-network on the trained supernet W And the optimal width corresponds to that of the highest performance on the validation dataset, e.g.
e" arg max Accuracy (w",, W":", Deat), S.L. FLOPs(c) S Fe. aEC
where Pb is constraint on the FLOPs, here we consider FLOPs rather than latency as the hardware constraint since we are not targeting any specific hardware device like EfficientNet Tan & Le (2019) and other width search or channel pruning baselines Zhuang et al. (2018); You et al. (2020); Dong & Yang (2019); He et al. (2018b); Tang et al. (2020); Su et al. (2020). The searching of (2) can be fulfilled efficiently by various algorithms, such random or evolutionary search (Hancock, 1992: Whitley et al., 1990; Liu et al., 20)9.).
3 CAFENET
3.1 LOCALLY FREE WHIGHT SHARING
As illustrated before, oho-shot supernet V is usually trained an evaluator to reflect the perfor- mance of different settings of network width. Assume we count channels in a layer from the left. To locate the sub-network in the supernet for width at certain layer" 2, conventional fixed weight sharing pattern assigns the leftmost e channels as the sub-network as Fbb. Formally, for width c, its corresponding sub-network is specified by fixed channel assignment with index set Z(c) of channels, i.e.,
fixed weight sharing pattern: I(c) = [1 : c].
However, this hard assignment imposes an inherent constraint on the search space. Channel assign- ment of smaller width will always be a subset of that of larger width. Actually, this fixed pattern forces the weight sharing of two different width to the greatest extent. To quantify this property, we define a simple weight sharing degree d of two any width c and ē by examining the overlap of their channel assignment, i.e.,
IT(c) I(AI
d(c, (in( [Z(e] & [0.
(4)
Intuitively, if the degree d(c, 2) is larger, then sub-networks of width e and ē will share more weights, and this implies that the supernet will enforce width e and ē to have more similar performance. And for the fixed pattern, the weight sharing degree reaches its maximum as Theorem 1 3.
Theorem 1. The weight sharing degree of fixed pattern will always be I for any two width.
However, ideally, each width should have full freedom in the supernet to select its channel assignment. In this situation, sub-networks of two diffcrent layer width can be fully shared, partially shared, or even have overlap, thus the weight sharing degree freely varies from O to 1. And each width can be better evaluated by some sub-networks, instead of a manually assigned sub-network. Nevertheless,
'eere onit the subscript i for indicating any width at i-th layer.
II holds naturally. Suppose c č, then have I(c) (() I(c) for fixed pattem..
3

---

Published as a conference paper at ICLR 2021
this induces the search space to proliferate from the fixed O(n) to O(2"), which is computationally unaffordable for practical search.
Therefore, we bridge these two extreme situations, and propose locally free weight sharing strategy. Basically, to specify sub-network of width e from supernet for evaluation, we need to assign e channels from all z channels per layer, with the channel assignment index set I(c). We split the index set [(c) into two parts, Le., G base channels [c)c) and ef free channels [c),,, with I(c)- = Ix(c)UZf(c) and c = Ca + c.. In detail, for free chamnels, we encourage to select of channels from a given zone, which is the neighborhood of the e-th channel, iee. B(c;r) [e- : e + r), and is a preset allowed offset to control the search space with ey r 1. Besides, base channels follow the fixed weight sharing pattern with (e) = [0 re]: let [ge) (e) = 0, then we can have O = max(c- r - 0). In this way, in our formulation, we can specify multiple channel assignment for width to better evaluate its performance as Fig. (bb), ie..
locally free weight sharing pattern: I(c) = [1: c+] (Z)( where (c(c) c [c--r: cc-r:cct (5)
According to (4), the weight sharing degree of our method is between ca/c S d(c, Ø} S 1. By properly setting offset r. we can achieve a tradcoff between the fixed pattern and the fully free patterm. Given r, now our search space scales at ( 11)) which is only constant times (e.g.. O(3n) forr 1, see Fig. ((b)) of fixed weight sharing search space. In this way, each width can be hetter represented, thus boosts the performance ranking ability of supernet.
Training with min-min optimization. With the above search space, we can train our CafeNet a stochastic setting as (l) which simply samples a network width e first, and then optimizes its sub-network. Nevertheless, in CafeNet width is specified more freely by several sub-networks, and the performance of these sub-networks can be different. In this way, we propose to indicate the performance for each width by examining its sub-network with the best performance. In training CafeNet, instead of randomly optimizing sub-network, we optimize the sub-network with the smallest training loss. To achieve this, we need to traverse all sub sub-networks for width However, this can be efficiently fulfilled since our job is to identify sub-network with the minimum loss, which can be quickly calculated a feed-forward manner. Afterward, we only need to implement a backward update for the target sub-nette sub-network with minimum loss. A detailed explanation of this min-min optimization is elaborated in Appendix A.I.
Searching with max- max selection. After the CafeNet / is trained, we can evaluate each network width by examining its performance (e.g. classification accuracy) CafeNet, for each width c, we use the sub-network with the highest performance to indicate the performance. Then we search for the optimal c* as (2).. Note that since scarching itself is much faster than the supernet training, the increased computational cost is subtle and acceptable in real practice. A detailed description of this max-max selection is presented in Appendix A.I.
the validation dataset D
Similar the training
3.2 REDUCING SEARCH SPACE WITH SOOPS-EENSITIVE BINS
Based on the locally free weight sharing pattern, each network width can be more flexibly indicated and then evaluated by the CafeNet. However, previously illustrated, the size of the search space is considerably large and brings tough challenge for the subsequent searching algorithms, such random search and evolutionary search. In this way, current methods usually choose to reduce the search space by initializing the minimum searching unit as a group of channels (called bin) instead of a single channel. Concretely, all channels at each layer are partitioned into K groups, with (n:/ K) channels in each group for the i-th layer.
However, this uniform partition neglects the extra information of channels within layers, i.c., kernel size and feature map resolution, and all of these factors are involved in the calculation of FLOPs. This inspires us that we should build bins based on the FLOPs. For simplicity, we encourage all bins in our search space to have the same FLOPs. Note that FLOPs is up to the layer width between two adjacent layers; for example, for convolution with resolution H x W and kerrel size K x K, its FLOPs can be calculated as
(6) where Cin and Cout are the width for the input and output of this layer, respectively. In this way, channels at different layers contribute differently to the overall FLOPs of the supernet. And if a
FLOPs Cond) Cin X Cont H W x K2
4

---

Published as a conference paper at ICLR 2021
channel at a layer contributes more to the FLOPs, then we can say it is more sensitive w.r.t. FLOPs. This FL.Ps-sensitivity can he simply reflected by examining the real FLOPs of single channel at layer. Concretely, for each channel at the i-th layer, it serves as both the output channel of the (i 1j-th layer and imput channel of the (i 1)-th layer. Thus the sensitivity & for the i-th layer can he represented as
(1- + (1,
where 1 1 and 1 are the width of the (i - 1) and the (i 1)-th layer, respectively.
As result, we simply encourage all bins to have the same FLOPs--ensitiiityy To do so, propose initialize the bin size (i.e., number of channels in bin) more adaptively. The intuition is that the larger FLOPs sensitivity layer has, the fewer channels should be allocated in a bin, and vice versa. As a result, the bin size b, for i-th layer should satisfy
by x 1/2;-
(8)
In practice, we usually specify minimum bin size (i.e., with maximum sensitivity e) as 8, then the bin size for each layer is
max, Ej
b; 3
(9)
Actually, the minimum bin size 3 in (9) controls the total number of bins, thus also determines the size of the search space. Specially, b, should be an integer and greater than or equal to 1. If 5 is set to be smaller, the search space will be much larger and finc- ane-grained since our search unit gets smaller and delicate accordingly. We will discuss a multi-staee search strategy by controlling / actively in Section 3.3, and present empirical investigations in Appendix A.
3.3 FURTHER BOOSTING PERFORMANCE WITH MULTI-STAGE SEARCH
As previously illustrated, setting minimum bin size in (q. be smaller will bring in smaller searching unit, and thus a more fine- graiggraiedd structure will be expected. However, this is accompanied by a larger search space and is challenging for both the ranking ability of supernet and searching performance. In this way, we can leverage multi-stage searching strategy to avoid the search space being too large but still search fine-grained level. In detail, we propose to search for the width under the multi-stage FLOPs budget, which is decayed linearly, ie..
(LOPs(() + FLOP)(() - (FLOPs(0) Fb) x T +(++1)+ 3(t)/a
(10)
where FLOPs(t) represents the FLOPs budget 1-th stage and FLOPs(0) is the FLOPs of the supernet. F6 is the target FLOPss after all T stages as (13). Besides, at each stage, we shrink the minimum bin size 3 with evolving speed > 0. Thus the search space will evolve from the coarse- grained to the fine aneinnrd but with controllable size. More details explored in Appendix A.14 144 and A.15. However, searching with multi-stage strategy inevitably introduces more computation consumption. Therefore, we only implement this method in the small dataset (i.e. CIFAR-10).
EXPERIMENTAL RESULTS
In this section, we perform extensive experiments to validate the effectiveness of our CafeNet. For all architectures, we use the SGD optimizer with momentum 0.9. Parameters 3 in Eq.(9) and in (10) are set to and 2, respectively. Offiset for locally free channels is set for all experiments. Detailed experimental settings are elaborated Appendix A.2.
We include various state-of-the-art width search methods for comparison, e.g. AutoSlim (Yu & Huang, 2019), MetaPruning (Liu et al., 2019a). TAS (Dong & Yang. 2019), GBN (You et al., 2019) and FPGM (He et al., 2019a). Besides, for comprehensive comparison, we perform searching using both evolutionary search and random search, named CafeNet-E and CafeNet-R, respectively. In addition. we also consider two vanilla baselines, i.e., Uniform and Random. Uniform: we shrink the width of each layer uniformly with a fixed factor to meet the FLOPs budget. Random: we randomly sample 20 networks under FLOPs constraint, and train them by 50 epochs, then we continue training the one with the highest performance and report its final result.
5

---

Published as conference paper at ICLR 2021
Table I: Results on ImageNet of ResNet50 and MobileNetV2 compared with state-of tefte-of-hhe-aeee of--th-art arr methods. References of baseline methods summarized in the Appendix A.4.
Molck-NeV2
Methods
FLOPs
Param
Tap-1
Top-5
Michhod
PLOPs
Param
Top-1 [
Top-5
AutoSlim
76.0%
MictaPruning
71.2%
MetaPruning
3.0G
76.2%
LEGR
210M
71.4%
LEGR
3.0G
76.22%
AutaSlim
207M
4.1M
73.0%
3G
Random
3.0G
19.1M
75.2%
92.5%
Uniform
Random
217M
2.7M
70.3%
89.1%
CafeNet-R
3.0
22.6M
77.1%
94.3%
CafeNet-R
217M
3.0M
73.3%
91.1%
CafeNet-E
GBN
24G
3LXM
76.2%
92.8%%
LEGR
180M
70.8%
LEGR
2.4G
75.7%
92.7%
TAS
150M
70.9%
HPGM
2.4G
75.6%
AMC
150M
70.8%
TAS
2.3G
76.2%
93.1%
LEGR
150M
69.45
AutoSlim
2.0G
20.6M
75.6%
150M
MuffNet
149M
63.7%
2G
Uniform
2.G
13.3M
75.1%
92.7%
Unitorm
150M
2.0M
60.355
88.
Random
2.0G
74.6%
92.29
Random
150M
68.8%
88.7%
CafeNet-R
76.5%
95.1%
CafeNet-R
2.7M
CafeNet-E
2.0G
18.4M
76.9%
93.3%
CafeNet-E
150M
3.0M
72.4%
90.4%
MetaPruning
1.0
73.4%
MetaPruning
105M
05,0%%
AutoSlim
Uniform
Lmifarm
L0G
6.6M
73.1%
91.8%
Random
105M
63.9%
IG
CafeNet-R
Random
1.G
11:2M
72.2%
91.4%
CafeNet-E
CafeNet-R
106M
2.2M
2.1M
68.2%
88.2%
CafeNet-R
1.G
12M
75.1%
92.6%
100M
MuffNer
50M
50.3%
AutoSlim
370M
72.2%
MetaPruning
43M
58.39
Unifarm
570M
90.6%
Uniform
S0M
59.7%
570M
Random
570M
69.4%
90.3%
Random
SUM
57,455
61.2%
CafeNet-R
570M
11.3M
72.7%
90.9%
CafeNet-R R
S0M
1.7M
64.3%
85.2%
CafeNet-E
570M
12.0M
73.3%
91.2%
CafeNet-E
50M
1.6M
64.9%
85.458
Table 2: Searching results of EfficientNet-Bi with xx and 0.5x FLOPs ImageNet dataset.
LithoienNee--00
0.x
LchecienNer--00
Methods
FLOPs
Faram
Top 1
Tap-5
Methods
FLOPs
Param
Top
Tap
Uniform
385M
5.3M
76.42%
92.24%
Uniform
192M
2.8M
74.16%
91.48%
Random
385M
5.1M
75.96%
92.11%
Random
192M
3.0M
73.36%
1..33%
CafeNet-R
385M
6.2M
76.59%
92.74%
CafeNet-R
192M
3.7M
74.47%
91. 1..65%
CafeNet-F
385M
6.9M
76.83%
92.96%
CafeNet-F.
192M
3.9M
74.62%
91.87%
4.1 RESULTS ON IMAGENET DATASET
We perform experiments ResNet50 (He et al., 2016), MobileNetV2 (Sandier et al., 2018) and EfficientNet-B0 (Tan Le, 2019) examine the performance CafeNet on heavy and light models, as summarized in Table and In detail, ResNet50 and MobileNetV2 are searched from the original size, and for EfficientNet-BB. the 0,5 x FLOPs is also searched from the original model. But the 1.0x FLOPs is from supernet with 1.5x FLOPS from uniform width scaling. The original EfficieniNet- BO, ResNet50 and MobileNetV2 has 5.3M, 25.5M and 3.5M parameters and 386M, 4.1G and 300M FLOPs with 76.4%, 77.8% and 73.7% Top-1 accuracy, respectively. Note that we also adopt KD the final training of MobileNetV2 for fair comparison with baseline methods, i.e., AutoSlim (Yu Huang, 2019), TAS (Dong & Yang, 2019). More detailed experiments and visualization results of our searched network are shown in Appendix A.8 and A.
As shown in Table 1, CafeNet achieves the highest performance on ResNet50 and MobileNetV2 w.r.t. different FLOPs, which proves the superiority of our method compared to other width search algorithms. Besides, CafeNet obtains remarkable performance with tiny FLOPs budgets. For example, our 570M ResNet50 achieves 73.3% Top-1 accuracy, surpassing AutoSlim (Yu & Huang. 2019) by 1.1%. In addition, shown Table for the NAS- network Efficiee(Net-BO.. the performance can further improved using CafeNet. It indicates that our CafeNet can optimize its width more fine grained manner.
4.2 RESULTS ON CIFAR- 10 DATASET
We also investigate CafeNet with MobileNetV2 (Sandler et al., 2018) and VGGNet (Simonyan & Zisserman, 2014) on the moderate CIFAR-10 10 dataset. Our original VGGNet (MobileNetV2) has 20M (2.2M) parameters and 399M (297M) FLOPs with accuracy of 93.99% (94.81%). As shown in Table 3, our CafeNet enjoys significant superiority to other algorithms. In detail, our 44M MobileNetV2 achieves 95.31% % accuracy and outperforms MuffNet (Chen et al., 2019) by 2.2%. For VGGNet,
6

---

Published as conference paper at ICLR 2021
Table 3: Results of MobileNerV2 and VGGNet on CIFAR-IH 10 dataset. References of baseline methods summarized in Appendix A.4.
MobikNNaV2
Groups
Methods
FLOPs
Parameters
Accurscy
Groups
Methods
FLOPs
Paramidicss
Accuracy
DCP
94473%
GAL
80.8
CafeNet-R
188M
1.4M
95.44%
DCP
199M
104M
94.16%
100M+
CafeNet-R
188M
1.5M
95.56%
200M
Sliming
199M
10.4M
93.80%
MuffNet
CafeNet-R
CafeNet-R
144M
1.2M
95.28%
CafeNet-F.
189M
8.0M
94.36%
CafeNet-K
144M
1IM
95.44%
PS
136M
V3 637
93.220%
CafeNet-R
AutoSlim
59M
0.7M
93.00%
CafeNet-E
154M
3.1M
94,23%
MußNett
45M
93.12%
100M
AOFP
124M
93. 93.84%
100M-
CafeNet-R
95.16%
CafeNet-R
115M
2.4M
93.87%
CafeNet-I
4-M
0.4M
95.31 %
CafeNet-E
115M
2.1M
94,01 %
AutoSlim
28M
0.3M
92.00%
ClNNets
92M
CafeNet-R
28M
0.2M
93.87%
76M
CafeNet-R
76M
2.2M
93.36%
CafeNet-E
28M
0.2M
94.11%
CafeNet-E
76M
1.4M
93.67%
Table 4; Results of MobileNetV2 and ResNet18 on CelebA datasct.
MobileNeIV2
ResNei18
106M
FLOPs
316M
619M
Uniform 91.63%
71.73%
91 92%
91.97%
Uniform
91.66%
91.79%
91.939
92.03%
Random
91.42%
91.52%
63.6
91.76%
Random
91.51%
91.62%
91.67%
91.84%
CafeNet-R 91.71%
92.03%
92.09%
92.12%
CafeNet-R
91.83%
92.07%
92.13%
92.17%
CafeNet-E
91.85%
92.13%
92.16%
92.19%
CafeNet-E
91.92%
92.16%
98.18%
92.25
Table Results of ResNet50 and MobileNerV2 on MS COCO dataset (Lin et al., 2017b;a).
ResNet50
MobieNNaVV
Framework Original 1,0x
CafeNet
Uni 0.5x
Framework RetimaNet
Original L.0x
0.5x
Uni 0.5x
RetinaNet
36.3%
34.9%
34.2%
31.2%
29.2%
27.5%
Faster R-CNN
37.2%
36.0%
35.5%
Paster R-CNN
31.7%
29.1%
28.5%
our CafeNet achieves 94.36% accuracy with 189M FLOPs, which surpasses DCP (Zhuang et al., 2018) by 0.2%. Similarly, CafeNet shows great advantages w.r.. tiny models, e.g. with 2.11%
accuracy improvement than AutoSlim (Yu & Huang, 2019) MobileNetV2 with 28M FLOPs. More experiments on CIFAR-10 datasct are investigated Appendix A.9.
4.3 EXPERIMENTS ON FACE ATTRIBUTE TASK WITH CELEBA DATASET
To futher investigate the effeetiveness of CafeNet, we perform experiments multi-label classifiatn task on CelebA (Liu et al., 2015) dataset with MobileNetV2 (Sandler al., 2018) and ResNet18 (He et al., 2016). CelebA dataset has 162K training and 20K testing images with 40 attribute labels from 2 categories. We report the average accuracy of all 40 labels. The original MobileNetV2 (ResNet18) 8) has 213M (1.3G) FLOPs and 2.2M (1 22) parameters with 92.07% 93%% average accuracy.
As shown in Table 4, our CafeNet can even achieve higher accuracy with FLOPs reducing bit. For example, our 162M-FLOPs MobileNetV2 has 92.19% 19% average accuracy and outperforms the original model by 0. 12%. Besides, our algorithm still enjoys great superiority in tiny FLOPs; for example, our 130M-ResNell achieves 91 91.92% average accuracy. which is 0.26% higher than the uniform baseline. Besides, comprehensively compare the classification performances on all 40 labels, we present comparison results for each label in Appendix A.9.
4.4 TRANSPERABILITY OF THE SEARCHED WIDTH TO OBJECT DETECTION TASK
For object detection tasks, its backbone is usually initialized by pretrained model ImgeNet dataset. this way, we use MobileNetV2 and ResNet50 with 0.5x FLOPs as the backbone feature extractors to examine the transferability of our searched network width. Besides, we leverage both the two-stage Faster R- R-CNN with Feature Pyramid Networks (FPN) (Lin et al., 2017a; Ren et al., 2015) and the one-stage RetinaNet (Lin et al., 201 7b) frameworks for verification. We train models using the trainval split MS COCO (Everingham et al., 2010) as training data and report the results in mean Average Precision (Lin et al., 201 7a: Ren et al., 2015) on minival split. As shown in Table 5, the backbones obtained by our method (CafeNet 0.5 x) can achieve higher performance compared to the uniform baseline (Uni 0.5x).
7

---

Published as conference paper at ICLR 2021
95.2
94.2
MobileNetV2
95.1
94.1
Acouracy (56
r42
Accuracy (%6
Accuracy 996
94.8
95
94
1-1
94.6
93.9
94.8
MobileNet2
VGGNet
93.8
94.2
F=0
94.7
93.7
10
15
20
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9 1.0
Time cost (GPU hours)
lambda
(a) Performance of CafeNet under different offsets
(b) Ablations of min-min optimization
Figure 2: (a) Performance with different offset r of free channels for the 0.5x-FLOPs MobileNetV2 and VGGNet CIFAR-10 dataset. (b) Ablations of min-min optimization with 0.5x-FLOPs MobileNetV2 on CIFAR-10 dataset.
4.5 ABLATION STUDIES
Effect of off'st for free channels. As described before, free channels freely locate in the local zone indicated by an offset r. Since we search on bins, r controls the size of search space; larger r implies more accurate evaluation of width but induces larger search space, corresponding to more computational cost. To investigate its effect, compare the performance and time cost between different r on --LPs MobileNetV2 and VGGNet on CIFAR- 10 dataset. accuracy performance benefits from the increase of r while the larger r also aggravates the burden training, For a tradc-off between performance and time cost, we set r = in all experiments. Note that in the training of CafeNet, larger will have more sub networks; however. identifying the one with the minimum Joss only involves foed-forward calculation and does not increase training cost dramatically, More results refer to Appendix A.S.
shown in Fa),
Effect min-min optimization on supernet. During training CafeNet, we only optimize the sub-network with minimum training loss for cach sampled width. To investigaae the effect of this optimization strategy, supposc have all T iterations, then we implement min-min optimization only on the last (1 X) 7 iterations with À [0, 1]. For the first X T iterations, we simply optimize one sub-network randomly for each sampled width. As shown in Fig.2(b), our 0.5x-FLOP MobileNetV2 on CIFAR-10 improves 0.92% accuracy from A = to = O, which means that min-min optimization does help to better evaluate each width and boost the searching performance accordingly. We further record the performance of IK sampled width; details refer Appendix 1.1
Effect of performance improvement with more training and searching cost. Since CafeNet needs extra training and searching cost than fixed weight sharing patter,, one natural question comes to whether a fixed pattern can benefit from more training and searching cost and even surpass Cafe:et result. With this aim, we implement fixed pattern (i.e., 0 for CafeNet) with times (I (Ix+33x) of training epochs and generations using evolutionary search search for 0L5X-FLOPs MobileNetV2 and VGGNet on CIFAR-I0 10 dataset.
Table 6: Performance with more training and searching on CIFAR-10 dataset. Note CafeNet has 95.44% and 94.36% accuracy for O.5X- MobileNetV2 and VGGNet, respectively.
MobsleNerV
VoGNet
Training
Training
Souching
Searching
1x
94.53%
94.57%
94.55%
1x
93.62%
93.65%
93 668
94.618
04.67%
04.69%
2x
93.08%
93.73%
93.71%
94.72%
94.75%
94.74%
3x
93.09%
93.00%
9337456
As shown in Table only improve searching results slightly, while more times training cost for supernet has almost no effect on the searching performance. By comparing with Table and Fig.2(a) our CafeNet does efficiently improve the performance of scarching results with the same or a little extra training and searching cost.
that searching with more generations by evolutionary search
8

---

Published as a conference paper at ICLR 2021
CONCLUSION
In this paper, we introduce a new weight sharing pattern for network width search. In detail, our locally free pattern enables cach width is jointly indicated by its base channels and free channels, where free channels are used to better and more flexibly evaluate the performance. Besides, we leverage FLOPs--ensitive bins reduce the search space and allow FLOPs distribute more evenly for different layers. CafeNet can be trained slochastically using min-min optimization and searches the optimal width by max-max selection. Extensive experiments have been conducted on benchmark ImageNet, CIFAR-10 10, CelebA, and COCO datasets to show the superiority of our CafeNet to other state-of-the-art methods.
ACKNOWLEDGMENTS
This work is funded by the National Key Research and Development Program of China (No. 2018AAA0107011 and the NSFC 61876095. Chang Xu supported in part by the Australian Research Council under Projects DE180101438 and DP210101859.
REFERENCES
Han Cai, Chuang Gan, Tianzhe Wang. Zhekai Zhang, and Song Han. Once-for-all: Train network and specialize for efficient deployment. arXiv preprint arXiv:1908.09799 097911 2019.
Hesen Chen, Ming Lin, Xiuyu Sun, Qian Qi, Hao Li, and Rong Jin. Muffnet: Multi-layer feature federation for mobile deep learning. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Zhengsu Chen, Jianwei Niu, Lingxi Xie, Xuefeng Liu, Longhui Wei, and Qi Tian. Network ad- justment: Channel search guided by flops utilization ratio. arXiv preprint arXiv:2004.00 02767, 2020.
Ting-Wo Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculescu. Legr: Filter pruning via learned global ranking. arXiv preprint arXiv: 12368, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition. pp. 248-255 255. Ieee, 2009.
Xiaohan Ding, Guiguang Ding, Yuchen Guo, Jungong Han, and Chenggang Yan. Approximated oracle filter pruning for destructive ean width optimization. pp. 1607-1616, 2019.
Xuanyi Dong and Yi Yang. Network pruning via transformable architecture search. In Advances in Neural Information Processing Systems. pp. 759-770, 2019.
Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan. More is less: A more complicated network with less inference complexity. pp. 1895-1903, 2017.
Shangchen Du Shan You, Xiaojie Li, Jianlong Wu, Fei Wang, Chen Qian, and Changshui Zhang Agree to disagree: Adaptive ensemble knowledge distillation in gradient space. Advances in Neural Information Processing Systems, 33, 2020.
Mark Everingham, Luc Van Gool, Christopher K I Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (vac) challenge. International Journal of Computer Vision, 88(2): 303-338, 2010.
Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng zhhng Xu. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint arXiv:1810.055 181.0333...331 2018.
Shaopeng Guo, Yujie Wang, Quanquan Li, and Junjie Yan. Dmop: Differentiable markov channel pruning for neural networks. arXiv preprint arXiv:2005.08:: 033354 2020.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv preprint arXiv:15 2015.
9

---

Published as a conference paper at ICLR 2021
Peter JB Hancock. Pruning neural nets by genetic algorithm. In Artificial Neural Networks, pp. 991-994, Elsevier, 1992.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yang He, Guoliang Kang. Xuanyi Dong, Yanwei Fu, and YI Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint 1808 06866, 80.
Yang He, Ping Liu, Ziwei Wang. Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition, pp.
2019a.
Yang He, Ping Liu, Linchao Zhu, and Yi Yang. Meta filter pruning to accelerate deep convolutional neural networks. arXiv: Computer Vision and Patlern Recognition, 2019b
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference Computer Vision, pp. 1389-1397, 2017.
Yihui He, Lin, Zhijian Liu, Hanrui Wang. Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 784 800, 2018b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge a neural network. arviv preprint arXiv: 02333331,., 2015.
Yibo Hu, Xiang Wu, and Ran He. Tf-nas: Rethinking three search freedoms of latency- differentiable neural architecture search, arXiv preprint arXiv:2008. 05314, 2020.
Weizhe Hua, Yuan Zhou, Christopher M De Sa, Zhiru Zhang, and G Edward Suh. Channel gating neural networks. In Advances in Neural Information Processing Systems, pp. 1884-1894, 1894, 2019.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Wein- herger. Multi-scalc dense networks for resource efficient image classification. arXiv preprint wwXiv:1033.09844, 2017.
Tao Huang, Shan You, Yibo Yang, Zhuozhuo Tu, Fei Wang, Chen Qian, and Changshui Zhang. Explicitly learning topology for differentiable neural architecture search. arXiv preprint arXiv:2011.08 1.09300, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint urXiv:1502.0311 2015.
Shumin Kong, Tianyu Guo, Shan You, and Chang Xu. Learning student networks with few data. In AAAL, pp. 4469-4476, 2020.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar- 10 dataset. online: http://www es. toronto. edu/kriz/cifar himl, 55, 2014.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv: Computer Vision and Puttern Recognition, 2016.
Tuanhui Li, Baoyuan Wu, Yujiu Yang. Yanbo Fan. Yong Zhang, and Wei Liu. Compressing convolutional neural networks via factorized convolutional filters. pp. 3977-3986, 2019.
Shaohui Lin, Rongrong Ji, Chengian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang. and David Doermann. Towards optimal structured enn pruning via generative adversarial learning. pp. 2790 2799, 2019.
Tsungyi Lin, Piotr Dollar, Ross Girshick. Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. pp. 936 944, 2017a
Tsungyi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. pp. 2017b.
10

---

Published as a conference paper at ICLR 2021
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang- Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE International Conference on Computer Vision. pp. 3296-3305, 2019a.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2736-2744, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. pp. 3730-3738, 2015.
Jian-Hao Luo and Jianxin Wu. Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference. arXiv preprint 1805 08941, 2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: towards real-time object detection with region proposal networks. 2015:91-99, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chier Chen. Mo- bilenetv2: Inverted residuals and linear boutenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 45 15-04520, 2018.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. pp. 525-536, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
Xiu Su. Shan You. Tao Huang. Hongyan Xu, Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu. Data agnostic filter gating for efficient deep networks. arXiv preprint arXiv:2010.15041 2020.
Mingxing Tan and Quoc V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. pp. 6105-6114, 2019.
Yehui Tang, Shan You, Chang Xu, Boxin Shi, and Chao Xu. Bringing giant neural networks down earth with unlabeled data. arXiv preprini arXiv:1907.0660 2019.
Yehui Tang, Shan You, Chang Xu, Jin Han, Chen Qian, Boxin Shi, Chao Xu, and Changshui Zhang. Reborn filters: Pruming convolutional neural networks with limited data. In AAAL, pp. 5972-5980, 2020.
Yulong Wang, Xiaolu Zhang. Lingxi Xie, Jun Zhou, Hang Su, Zhang, and Xiaolin Hu. Pruning from scratch. arviv: Computer Vision and Pattern Recognition, 2019.
Darrell Whitley, Timothy Starkweather, and Christopher Bogart, Genetic algorithms and neural networks: Optimizing connections and connectivity. Parallet computing, 14(3):347-361, 1990.
Yibo Yang, Hongyang Li, Shan You, Fei Wang, Chen Qian, and Zhouchen Lin. Ista-nas Efficient and consistent neural architecture search by sparse coding, arXiv preprint a7Xiv:2010.06/76, 2020.
Yibo Yang. Shan You. Hongyang Li, Fei Wang, Chen Qian, and Zhouchen Lin. Towards improving the consistency, efficiency, and flexibility of differentiable neural architecture search. arXiv preprint arXiv:2/01.11342 2021.
Mao Ye. Chengyue Gong, Lizhen Nie, Denny Zhou, Adam R Klivans, and Qiang Liu. Good subnetworks provably exist: Pruning via greedy forward selection. arXiv: Learning, 2020.

---

Published as conference paper at ICLR 2021
Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1285-1294, 2017.
Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, and Changshui Zhang. Greedynas: Towards fast ont-shot nas with greedy supernet. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1999-2008, 2020.
Zhonghui You, Kun Yan, Jimmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. In Advunces in Neural Information Processing Systems. pp. 11300-144,, 2019.
Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for chamnel numbers. arXiv preprint arXiv: 1903. 1728, 8, 2019.
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-awe -wware aware channel pruning for deep neural networks. In Advances in Neural Information Processing Systems, pp. 875-885-88 2018.
12


Reviews:
[{"review_id": "rRx_eGlyhXN", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce in this submission a locally-free weight sharing strategy for selecting effective network width. The rationale and intuition behind are well-grounded. Experiments on various datasets and pruning setups prove the validity. \n\nStrength:\n+ The approach is well motivated and makes sense. The problem studied here is also important and could be of interest to a large audience.\n+ Experiments are sufficient. The results are promising and well support the claim.\n+ FLOPs-sensitivity bin considers factors including feature size and kernel size and seems to be independent of the total channel number, which, without douts, brings values.\n\nWeakness:\n- The proposed approach seems to be a  compromise between completely free weight and fixed weight, right? As a result, it would be good if the authors could elaborate the relation between the two.\n- By utilizing the methods, my understanding is that the search space scales from O(N) all the way up to O(C_{2r+1}^{N}), no? This is a considerable amount of time required as compared to the single network width. Please provide some discussion along this line.\n- The influence of the super network should be detailed. Intuitively, higher degrees of freedom will lead to better results. The authors should provide more analysis along this line.\n- The writing can be enhanced. Please go over the manuscript and make sure all the grammar errors have been taken care of. \n\n\n\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "km2Sy1SBvZw", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the weight sharing schema in one-shot width search and proposes a locally free weight sharing strategy (CafeNet). By splitting each width candidate into base channels and free channels, CafeNet makes a compromise between fixed weight pattern and full freedom pattern. Such strategy can reduce the search complexity and improve the performance ranking, w.r.t. different width, in the supernet. Experiments on various tasks, including classification, detection and attribute recognition, are well provided to support the effectiveness of the proposed method. The final results are quite promising.\n\nStrengths:\n1) The paper is well written and easy to follow. The motivation is clearly explained by an example and the problem formulation.\n2) The idea of locally free weight sharing is interesting. Such a solution for the previous fixed weight sharing seems sound.\n3) Experiments with additional analyses are well provided.\n\nI have the following concerns and suggestions:\n1) Missing some relevant papers. OFA[1] and TF-NAS[2] introduce width search by dynamically choosing the channels. The authors should cite and explain the differences.\n2) Is there any correlation between the degree in Eq. (4) and the searched accuracy under a fixed FLOPs?\n3) The bin size of Eq. (9) makes me confusing. As shown in experiments, β can be less than 1 and the second term in the right side of Eq. (9) is also less than 1. Thus, the bin size bi (i.e., number of channels in a bin) is less than 1 channel. Please explain it in detail.\n4) Why lambda=0 achieves the best accuracy in Fig. 2(b)? It is in conflict with the statement “As shown in Fig. 2(b), our 0.5-FLOPs MobileNetV2 on CIFAR-10 improves 0.92% accuracy from lambda=0 to lambda=1”.\n5) I suggest the authors to split 1G group in Table 1, as done in Table 11.\n6) In Algorithm1, both the supernet and the total number of epoch are defined as N.\n\nAlthough some details make me a little confusing, the experimental analyses and the intuitive solutions of locally free weight sharing in one-shot width search are quite informative and helpful to the NAS community. I suggests the authors to release their code and would like to see the authors’ responses. \n\n\n[1] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, Song Han. Once-for-All: Train One Network and Specialize it for Efficient Deployment. ICLR, 2020.\n\n[2] Yibo Hu, Xiang Wu, Ran He. TF-NAS: Rethinking Three Search Freedoms of Latency-Constrained Differentiable Neural Architecture Search. ECCV, 2020.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y7n2WWlQhRl", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors introduce a new weight sharing pattern to search for the width in a network layer. Besides, FLOPs-sensitive bins is proposed to measure the real FLOPs of a single channel at a layer and further reduce the search space. The paper proposes a locally free weight sharing mechanism where the channels in a layer are split into base channels and free channels. Compared with conventional fixed weight sharing pattern where the leftmost channels are assigned as the sub-network, the proposed locally free pattern increases more flexibility while the search space also scales at O(n). The proposed FLOPs-sensitive bins forces the layers with larger FLOPs sensitivity to have fewer channels, thus reducing the search space at a fixed FLOPs. Experimental results on several datasets show that the proposed CafeNet outperforms many other width search algorithms. The searched network experimentally achieves high performance with tiny FLOPs budgets.\n\nWhat I like about this paper in that: \n1.\tThe motivation and intuition are reasonable, which is to design a more flexible weight sharing pattern for network width search. \n2.\tExperiments are sufficient, thorough and carefully designed. Experimental results can support the objective of proposed methods. The searched network achieves remarkable performance with tiny FLOPs budgets.\n3.\tThe paper is well written and organized. The work is easy to follow and be reproduced.\n4.\tThe proposed methods have high generality and might be used on any convolutional network.\n\nSome minor concerns or suggestion about this paper:\n1.\tThe searching and training algorithms (max-max selection and min-min optimization) should be described in more detail.\n2.\tThe free channels are the neighborhood of the c-th channel in this paper, but I think more channels on the right should be included in the zone.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ARfspQ3zWtj", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Most existing methods follow a manually ﬁxed weight sharing pattern, leading to the difficulty that estimates the performance of networks with different widths. To address this issue, this paper proposes a locally free weight sharing strategy (CafeNet) to share weights more freely. Moreover, this paper further proposes FLOPs-sensitive bins to reduces the size of the search space. Specifically, this paper divides channels into several groups/bins that have the same FLOPs-sensitivity and searches for promising architectures based on the divided groups. Extensive experiments on several benchmark datasets demonstrate superiority over the considered methods. However, some important details regarding the proposed method are missing. My detailed comments are as follows.\n\nPositive points:\n1. Compared with the manually ﬁxed weight sharing pattern, this paper proposes a locally free weight sharing strategy (CafeNet), which allows more freedom in the channel assignment of a sub-network.\n\n2. To reduce the size of the search space, this paper proposes to divide channels into several groups/bins (also called minimum searching unit) that have the same FLOPs-sensitivity.\n\n3. The experimental results on image classification and object detection tasks show that the proposed method outperforms the existing methods by a large margin.\n\nNegative points:\n1. When training the super network, why the authors optimize the sub-network with the smallest training loss? More explanations are required.\n\n2. Why the sensitivity of a layer should be calculated as Eqn. (7)? It would be better to provide more details about that.\n\n3. Given a FLOPs constraint in Eq. (2), how to select a suitable width for each layer? Please discuss more and make it clearer.\n\n4. Is it possible to find a sub-network with zero width ($c=0$) for a layer? If so, how to deal with this case when evaluating the sub-network?\n\n5. The experimental results are inconsistent with the descriptions. In Figure 2(b), the performance of the proposed method goes worse with the decreasing of the $\\lambda$. However, the authors state that “MobileNetV2 on CIFAR-10 improves 0.92% accuracy from $\\lambda$ =0 to $\\lambda$=1”.\n\n6. The experimental comparisons in Table 1 are unfair. Compared with other methods (e.g. AutoSlim), the proposed method trains the models on ImageNet for more epochs (100 v.s. 300). More experiments under the same settings are required.\n\nMinor issues:\n1. In appendix A.13, “… and the bin evolving speed α in Section 3.4” should be “… and the bin evolving speed α in Section 3.3”.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed locally free weight sharing strategy (CafeNet) for searching optimal network width. The proposal is a nice tradeoff between manually fixed weight sharing pattern (too small search space) and completely free weight sharing pattern (too large search space). The *originality* and *significance* are clearly above the bar. The paper is related to the general interests of deep learning research and its *applicability* deserves a spotlight presentation.

It seems the *clarity* can still be improved, so please carefully revise the paper following the reviews. BTW, I am very curious, why "locally free weight sharing strategy" goes to a short name CafeNet? I went over the paper but I didn't find the answer. Perhaps the name of the proposal should also be explained...
Decision: Accept (Spotlight)