BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget
Authors: Jack Turner
Elliot J. Crowley
Michael O'Boyle
Amos Storkey
Gavin Gray
Venue:  (2020)
Keywords: model compression
architecture search
efficiency
budget
convolutional neural networks

Abstract: The desire to map neural networks to varying-capacity devices has led to the development of a wealth of compression techniques, many of which involve replacing standard convolutional blocks in a large network with cheap alternative blocks. However, not all blocks are created equally; for a required compute budget there may exist a potent combination of many different cheap blocks, though exhaustively searching for such a combination is prohibitively expensive. In this work, we develop BlockSwap: a fast algorithm for choosing networks with interleaved block types by passing a single minibatch of training data through randomly initialised networks and gauging their Fisher potential. These networks can then be used as students and distilled with the original large network as a teacher. We demonstrate the effectiveness of the chosen networks across CIFAR-10 and ImageNet for classification, and COCO for detection, and provide a comprehensive ablation study of our approach. BlockSwap quickly explores possible block configurations using a simple architecture ranking system, yielding highly competitive networks in orders of magnitude less time than most architecture search techniques (e.g. under 5 minutes on a single GPU for CIFAR-10).

Published as a conference paper at ICLR 2020
BLOCKSWAP: FISHER-GUIDED BLOCK SUBSTITUTION FOR NETWORK COMPRESSION ON A BUDGET
Jack Turner; Elliot J. Crowley; Michael O'Boyle, Amos Storkey, Gavin Gray School of Informatics University of Edinburgh
(jack, 11 lot.. Grow ley Red.ao. uk, nobe inf edd.ac.ck, Ia. storkey, g-d .......gg.... grayl Bed.ac .c.uk
ABSTRACT
The desire to map neural networks to varying-capacity devices has led to the devel- opment of a wealth of compression techniques, many of which involve replacing standard convolutional blocks in a large network with cheap alternative blocks. However, not all blocks are created equally; for required compute budget there may exist potent combination of many different cheap blocks, though exhaus- tively searching for such a combination is prohibitively expensive. In this work, we develop BlockSwap: a fast algorithm for choosing networks with interleaved block types by passing a single minibatch of training data through randomly ini- tialised networks and gauging their Fisher potential. These networks can then be used as students and distilled with the original large network as a teacher. We demonstrate the effectiveness of the chosen networks across CIFAR-10 and Ima- geNet for classification, and COCO for detection, and provide a comprehensive ablation study of our approach. BlockSwap quickly explores possible block config- urations using a simple architecture ranking system, yielding highly competitive networks in orders of magnitude less time than most architecture search tech- niques (e.g. under 5 minutes on a single GPU for CIFAR- 10). Code is available at atlttpps://gtthhi com htwpa//ww...c.caaeeaaacee//rrrrr
INTRODUCTION
Deep Convolutional Neural Networks are extremely popular, and demonstrate strong performance on a variety of challenging tasks. Because of this, there exist large range of scenarios in the wild in which practitioners wish to deploy these networks e.g. pedestrian detection in a vehicle's computer, human activity recognition with wearables (Radu et al. 2018).
These networks are largely over-parameterised;; a fact that can be exploited when specialising networks for different resource budgets. For example, there is a wealth of work demonstrating that is possible to replace expensive convolutional blocks in large network with cheap alternatives e.g. those using grouped convolutions (Chollet 2017 Xie et al., [017]] 10 loannou et al. Huang et al. (0188) or
bottleneck structures (He et al. 2016 Sandler et al. 2018 Peng al. 2018). This creates smaller network which may be used as a student, and trained through distillation (Ba & Caruana 2014
Hinton al. 2015) with the original large network teacher to retain performance.
Typically, each network block is replaced with a single cheap alternative, producing a single-blocktynn network for given budget. By cheapening each block equally, one relies on the assumption that each block is of equal importance. We posit instead that for each budget, there exist more powerful mixxddbbockkyyp networks that assign non- uniform importance each block by cheapening them to different extents. We now have paradox of choice: for a given budget, it is not obvious which cheap alternatives to use for our student network, nor where to place them.
Let's assume we have large network that consists of B expensive convolutional blocks, and candidate pool of cheap blocks that we could substitute each of these out for, and we are given a limit on the number of parameters (and therefore memory) that the network can use. Which of
Equal contribution.
Now at the University of Toronto.

---

Published as a conference paper at ICLR 2020
Candidate pool
Original
A
A
Attention Transfer
An
argmax A1)
Student
1. Sample random architectures
Rank by Fisher potential after one minibatch
3. Train with AT
Figure 1: The three-step BlockSwap pipeline. Beginning with a large network, we sample list of candidate architectures by replacing its blocks with cheap alternatives. In Step 2, we rank each candidate by its Fisher potential after a single minibatch of training data. In Step 3, we select the highest ranked architecture and train it with attention transfer from the original network.
the C mnixed-blocktye networks is appropriate? It rapidly becomes infeasible to exhaustively consider these networks, even for single digit C. We could turn to neural architecture search (NAS) techniques (Zoph & Le 2017 7, Zoph et al. 2018 Pham et al., 2018 Tan et al. [2019] Wu et al. 2019) Liu et al., (019a) these introduce gigantic computational overhead; even one of the fastest such approaches (Pham et al. 2018)) still requires half day of search time on GPU. Moreover, in
real-world scenarios, allocated budgets can change quickly; we would like be able to find good network in matter of minutes, not hours or days.
Our goal this paper is, given a desired parameter budget, to quickly identify a suitable mixed- blocktype version of the original network that makes for a powerful student. We present simple Beethod-Blockat to achieve this. First, we randomly sample a collection of candidate mixed- blockrype architectures that satisfy the parameter budget. A single minibatch is then pushed through each candidate network to calculate its Fisher potential: the sum of the total (empirical) Fisher information for each of its blocks. Finally, the network with the highest potential is selected student and trained through the distillation method of attention transfer (Zagoruyko & Komodakis 2017 with the original teacher. Our method is illustrated in Figure [
In Section 3 we describe the block substitutions used in BlockSwap. distillation via attention transfer, and Fisher information. We elaborate on our method in Section well providing a comprehensive ablation study. Finally, we experimentally verify the potency of BlockSwap on CIFAR- 10 (Section as well ImageNet (Section 6 and COCO (Section 7 Our contributions are as follows:
1. We introduce BlockSwap, an algorithm for reducing large neural networks by performing block-wise substitution. We show that this outperforms other top-down approaches such as depth/width scaling, parameter pruning, and random substitution.
2. We outline simple method for quickly evaluating candidate models via Fisher information, which matches the performance of bottom-up approaches while reducing search time from days to minutes.
3. We conduct ablation studies to validate our methodology, highlighting the benefits of block mixing, and confirming that our ranking metric highly correlated the final error.
2 RELATED WORK
Neural networks tend to be overparameterised: Denil et al. (2013 accurately predict most of the weights in network from small subset; Frankle & Carbin 2019) hypothesise that within large network, there exists a fortuitously initialised subnetwork that drives its performance. However, remains difficult exploit this overparameterisation without taking hit in performance.
2

---

Published as a conference paper at ICLR 2020
One means to combat this is to use large teacher network network to regularise the training of small student network; a process known as distillation. The small network is trained from scratch, but is also forced to match the outputs (Ba & Caruana 2014, Hinton et al. 2015 or activation statistics (Romero et al., 2015 Zagoruyko & Komodakis 2017 Kim et al. 2018) of the teacher using an additional loss term. When utilising distillation one must decide how create student network. A simple approach would be to reduce the depth of the original large network, although this can prove detrimental (Urban al. 2017). An effective strategy is to create student by replacing all the teacher's convolutions with grouped alternatives (Crowley et al. 2018a).
Grouped convolutions are a popular replacement for standard convolutions as they drastically cut the number of parameters used by splitting the input along the channel dimension and applying much cheaper convolution on each split. They were originally used in AlexNet (Krizhevsky et al. 2012) due to GPU memory limitations, and have appeared in several subsequent architectures (Loffe) Szegedy 2015 Chollet 2017] Xie et al.) 2017: Joannou et al. 2017 Huung et al. 2018). However, as the number of groups increases, fewer channels are mixed, which hinders representational capacity, MobileNet (Howard et al. 2017) compensates for this by following its heavily-grouper depthwise convolutions by pointwise (1 x 1) convolution allow for channel mixing.
The increasing complexity of neural network designs has encouraged the development of methods for automating neural architecture search (NAS). Zoph al. (2018) use RNN to generate network block descriptions and filter the options using reinforcement learning. These blocks are stacked to form full neural network. This is extremely expensive process, utilising 450 GPUs over the course of 3 days. To address this, Pham et al. (2018) propose giving all models access to shared set of weights, achieving similar performance to Zoph et al. (2018) with a single GPU in less than 24 hours. Subsequent works have made extensive use of this technique (Liu et al. 2019a Luo et al., 2018, Chen et al. 2018b). However, has been observed that under the constrained architecture search space of the above methods, random architecture search provides a competitive baseline (Li & Talwalkar 2019) Yu et al. 2020). In particular, Yu et al. (2020) show that weight sharing hampers the ability of candidate networks to learn and causes many NAS techniques to find suboptimal architectures.
NAS techniques predominantly take bottom-up approach; they find a powerful building block, and form neural networks using stacks of these blocks. Other works have taken a top-down approach
to find architectures using pruning (Chen et al., 2018 8a Lee et al. 2019: Liu et al. 2019b Crowley ct al. 2018b; Frankle & Carbin 2019). Chen et al. (201 Sa) take a pre- trairrinnd network and apply a principled framework compress it under operational constraints such as latency. In SNIP (Lee et al. 2019). one randomly initialises a large network and quantifies the sensitivity of each connection using gradient magnitudes. The lowest sensitivity connections are removed to produce sparse architecture which is then trained as normal.
3 PRELIMINARIES
3.1 SUBSTITUTE BLOCKS
Here, we will briefly elaborate on the block substitutions used for BlockSwap. A tabular comparison given in Appendix [A] The block choices are deliberately simplistic. We can therefore demonstrate that it is the combination of blocks that is important rather than the representational capacity of specific highly- eegineeed block e.g. one from the NAS literature.
The blocks considered are variations of the standard block used in residual networks. In the majority of these blocks, the input has the same number of channels as the output, so we describe their parameter cost assuming this is the case. This standard block contains two convolutional layers, each using N lots of N x x & filters where le is the kernel size. Assuming the costs of batch-norm (BN) layers and shorteut convolutions (where applicable) are negligible, the block uses total of 2 parameters.
Groopedd+Pointwicn Block - G(g) In a grouped convolution, the N channel input is split along the channel dimension into g groups each of which has N/g channels. Each group goes through its own convolution which outputs N/g channels, and all the outputs are concatenated along the channel dimension. This uses g (N/g) 2k2 222 parameters. To compensate for reduced
3

---

Published as a conference paper at ICLR 2020
channel mixing, each convolution is further followed by pointwise (1 1) convolution, incurring an extra cost of 2 For this block, each full convolution has been replaced with a grouped+pointwise convolution, and so the block uses 2((N2 22) parameters.
Bottleneck Block B(b) In this block, pointwise convolution is used to reduce the number of channels the input by a factor b before a standard convolution is applied. Then, another pointwise convolution brings the channel size back up. This uses (N//)2 + 2N2 parameters.
Bottleneck Grouped+ Prouuped+Pointwiee Pointwise Block - BB g) This is identical to the bottleneck block, except the standard convolution is further split into y groups, and so uses (N/bg)2k² + 2N2 //b parameters.
3.2 DISTILLATION VIA ATTENTION TRANSFER
Attention transfer Zagoruyko & Komodakis 2017) is a distillation technique whereby a student network is trained such that its attention maps at several distinct attention points made to be similar to those produced by a large teacher network. A formal definition follows: Consider a choice of layers ..,, L. in teacher network with I. layers, and the corresponding layers in the student network. At each chosen layer of the teacher network, collect the spatial map of the activations for channel j into the vector afy. Let A collect at, for all j. Likewise for the student network we correspondingly collect into a and Aç. Now given some choice of mapping ((A,) that maps each collection of the form A, into a vector, attention transfer involves learning the student network by minimising
f(A
f(Ag)
=AT = LCE +
(1)
[((A)]
[[f(A)) (2
where 3 is hyperparameter, and LCg is the standard cross-entropy loss. In Zagoruruyko & Komodakis NA; (217)) the authors use f(A.) (1/NA.) Ej4 a?y. where NA, is the number of channels at layer i.
3.3 FISHER INFORMATION
Theis al. (2018) derive a second order approximation of the change loss that would occur on the removal of particular channel activation in neural network; they demonstrate that this is equivalent to calculating an empirical estimate of the Fisher information for binary mask parameter that is used to toggle that channel or off. They use this signal A. identify the least important activation channels, and remove their corresponding weights while pruning. Formally, let us consider a single channel of activation in network due to input minibatch of N examples. Let us denote the values for this channel as a: a x x tensor where W and H are the channel's 's spatial width and height. Let refer to the entry corresponding to example in the mini-batch at location (i, j) as anij- We can backpropagate the network's 's loss L to obtain the gradient of L. with respect to this activation channel &L Un
Let us denote this gradient as and index it as Gnij. A. can then be computed
by
N
W
A.
Σ
ΣΣ
anizgmis
(2)
2N
In this work, we are interested in obtaining the Fisher information for a whole block. We approximate this quantity by summing A. for every output channel a block as
A
A
Using an approximation to the Taylor expansion of a change in loss to gauge the saliency of individual parameters originated in LeCun al. [ (1989) and has inspired many works in pruning (Hassibi &
Stork
1993 Molchanov et al. 2217): Guo et al. 2016 Srinivas Babu 2013) and quantisation (Choi
al. 2017 Hou et al. 2017).
4

---

Published as a conference paper at ICLR 2020
4 METHOD
Let us denote a large teacher network I' composed of / blocks each of type S I' a' = [8|, S2 ..., SB] Each of these may be replaced by a cheap block Cr, chosen from list of candidates C1 C2, ,..., NN of various representational capacities. We wish to construct a smaller model M Cr1 Cr2. CrBl that is powerful, and within given parameter budget. But, the space of possible block configurations is very large (~ 102s in Section 5) Even when using cheap network evaluation strategy is not possible exhaustively search through the available network architectures. What we require is method to quickly propose and score possible configurations with as little training as possible. We develop BlockSwap to achieve this.
First, we obtain candidate architectures through rejection sampling: we generate mixed-blocktypp architecture proposals random and only save those that satisfy our parameter budget. As these are only proposals, this step does not require instantiating any networks; parameter count can be inferred directly from the proposed configuration, making this a very cheap operation.
Second, we score each of our saved candidates by its Fisher potential to determine which network to train. We obtain this score for each candidate as follows: we initialise the network from scratch and then place probes after the last convolution in each block. A single minibatch of training data is then passed through the network, and the resulting cross-entropy loss is backpropagated. The probe measures the total Fisher information of the block A, by summing A. (Equation 2 for each channel in the layer is placed after. For this step, minibatch size is set equal to the size used during training. We then sum this quantity across all blocks to give the Fisher potential.
The intuition for this metric is as follows: the Fisher potential is the trace of the Fisher Information matrix of the activations. It is aggregate of the total information each block contains about the class label (under a simplifying conditional independence assumption). During training it this information about the class that drives learning, and the initial learning steps are key. Hence higher information values tend to result in higher efficiency block utilisation.
Once we have scored each candidate architecture using the Fisher potential, select the one with the highest score and train it using attention transfer. The training hyperparameters can be mirrored from T. We use the following blocks (defined Section 3.1 as candidate blocks:
B(b) for E (2,4)
G(g) for g (2, (2,4,8, 1.448...../166.V88..4......
g) for E {2, 4, 8, 16, M/2
where N is the number of channels in a block, and M is the number of channels after a bottleneck this is 2/2 when = 2. We also use the standard block S as a candidate choice, as not force blocks to reduce capacity where it is imperative.
4.1 WHY MIX BLOCK TYPES?
A reasonable question to ask is whether mixed-bloo blocktype architectures will perform better than the use of single blocktype. As simple check for this, we chose parameter budgets of 200K, 400K, and 800K and at each budget, compared singlebblocttype models to one hundred randomly assembled, mixed- blocktbbocktyppe models. The models are WideResNets (Zagoruyko Komodakis) 2016) with substituted blocks trained on CIFAR-10.
we plot the final test error of the random architectures (blue densities) against their singie-blocktype counterparts (the red dotted lines). We observe that for each of the given parameter budgets there exist combinations mixed blocks that are more powerful than their single blocktype equivalents. However, single-bloktyy random networks. This suggests that, though random search has been shown to work well in other architecture search settings (Yu et al. 2020). it will yield sub-optimal structures in this case.
In Figure 2
networks typically sit below the mean of the density of the
5

---

Published as a conference paper at ICLR 2020
200K
400K Params.
800K Params.
Mixed BG(2,4)
Mixed 1(1
Mixed
Density (%)
((4)
4.5
5.5
4.0
5.0
3.5
4.5
5.0
5.5
Top-1 Erroc (%)
Top-1 Error (36) (b)
Top-1 Error (9)
(s)
(c)
Figure 2: For parameter budgets of (a) 200K, (b) 400K and (c) 800K we train single blocktype network and 100 random mixed blocktype networks. This lets us examine the distribution of mixed blocktype architectures to verify the existence of performant ones. The dotted red line each plot represents models composed of a single block type, and the shaded blue density represents randomly constructed, mixed- beeee-ypoc networks. For example, (a) shows that WRN-40-2 with every block substituted for BG (2, 4) trains to 5.0% error, while the mean error of randomly constructed networks the same parameter budget lies around 5.15% error. This implies that random architecture search will not work here, and that better strategy is needed.
4.2 WHY USE SINGLE MINIBATCH? How DOES YOUR RANKING METRIC COMPARE TO ALTERNATIVES?
Though there are no common metrics for architecture ranking, neural architecture search methods often use information from early learning stages to evaluate possible candidates. In order to assess the fastest, most accurate indicator of final еттоr, we examine our 100 random bixed-blocktype networks for parameter budgets of 200K, 400K, and 800K and record how final error correlates with (i) training accuracy, (ii) total /2 norms of weights, (iii) summed absolute gradient norms of weights (Golub jet al. 2019), and (iv) Fisher potential after 1, 10, and 100 minibatches training in Table 1 We found that Fisher potential was by far the most robust metric, and that there were diminishing returns as the number of minibatches increases.
Table 1: Spearman Rank correlation scores for several ranking metrics when ranking 100 random architectures at three parameter budgets. Negative correlation implies that as the metric score goes up, final error goes down. Summing gradient norms follows similar pattern Fisher potential but appears to be less robust and moderately less accurate. f2-norms are extremely volatile and largely uninformative. Crucially, adding more than I minibatch has diminishing returns.
200K Params.
400K Params.
800K Params
Minibatches
10
100
1
10
100
10
100
Accuracy
-0.004
-0.022
-0.495
-0.068
-0.282
-0.386
0.042
-0.211
-0.257
f2 norms
-0.083
-0.185
-0.289
0.093
-0.213
-0.251
0.368
0.226
0.140
Grad Norms
-0.541
-0.490
0.326
-0.612
-0.444
0.402
-0.608
0.340
0.558
Fisher
-0.602
-0.621
-0.439
-0.685
-0.667
-0.508
-0.635
-0.638
-0.277
4.3
How MANY SAMPLES ARE NEEDED?
Figure suggests that sampling 100 random architectures will reliably yield least one mixed- blocktype network that outperforms a similarly parameterised single blocktype network. However, accepting that there is some noise in our ranking metric, we assume that we will need take more than 100 samples in order to reliably detect powerful architectures. As illustration, at the budget of 400K parameters a single blocktype alternative has a test error of 4.45%, whereas BlockSwap finds networks with final test errors of 4.85%. 4.54%, and 4.21% after 10, 100, and 1000 samples respectively. We empirically found that 1000 samples was enough to be robust to various parameter budgets the tasks we considered.

---

Published as a conference paper at ICLR 2020
4.4 WHAT DO "GOOD NETWORKS" LOOK LIKE?
For each of the three parameter budgets (200K, 400K, 800K) we inspected the most common block choices for "good" and "bad" networks, where good and bad are networks with final error greater two standard deviations below or above the mean respectively. We found that overwhelmingly, the types of blocks used in both good and bad networks was very similar, implying that is the placement of the blocks instead of the types of the blocks that matters. We examine this further in Appendix B
5 CIFAR EXPERIMENTS
Here, we evaluate student networks obtained using BlockSwap on the CIFAR-I0 image classification dataset (Krizhevsky) 2009). We benchmark these against competing student networks for range of parameter budgets. To recapitulate, the BlockSwap networks are found by taking 1000 random samples from the space of possible block combinations that satisfy our constraint (in this case, parameter budget). These points are ranked by Fisher potential after a single minibatch of training data, and the network with the highest potential is chosen.
The structure we use that of a WideResNet (Zagoruyko & Komodakis, 2017), since we can use them to construct compact, high performance networks that are broadly representative of the most commonly used neural architectures. A WideResNet with depth 40, and width multiplier 2 WRN- 40-2-is trained and used teacher. It consists of 18 blocks and has 2.2 million parameters. A student network is generated and is trained from scratch using attention transfer with the teacher. Our BlockSwap wap students WRN-40-2 nets where each of its 18 blocks are determined using our method with the blocks outlined in Section We compare against the following students:
1. Reduced width/depth versions of the teacher: WRN- 1,,-116,,,RNN-122 WRN-40-1
2. Single block-type teacher reductions: every block in the teacher is swapped out for (i) the MBConvó block from MobileNet v2 (Sandler al. 2018 and (ii) the (normal cell) block discovered in DARTS (Liu et al., 2019a).
3. Pruned teacher reductions: we compare against (i) the magnitude-bann based pruning methodol-
ogy from Han et al. (2016) and (ii) SNIP (Lee et al. 2019) versions of the teacher.
We additionally compare against CondenseNet-86 (Huang et al. 2018 distilled with the Born-Again strategy described by Furlanello et al. (2018).
First, we train three teacher networks independently. These are used to train all of our students; each student network is trained three times, once with each of these teachers. Figure shows the mean test errors of BlockSwap students at various parameter counts, compared to the alternatives listed above. Full results with standard deviations are listed in Table 2 along with the number of Aultiply-Accumulate (MAC) operations each network uses.
Our results show that block cheapening is more effective than simple downscaling schemes (reducing width/depth); not only are the reduced models (WRN-40-1, WRN- 16-2, WRN-16-1) 1) inflexible in parameter range, they perform significantly worse than our models at the parameter budgets they can map to. We also show that BlockSwap finds more accurate networks than the other top-down approaches (fl-pruning and SNIP) across the full parameter spectrum. Note that as El fl-pruning and SNIP introduce unstructured sparsity, the parameter counts provided are the number of non-zero parameters (the weight tensors remain the same size as the original teacher).
The mixed block architectures that BlockSwap generates are more accurate at these reduced budgets than all of the single blocktype alternatives we considered. While the performance of DARTS is very similar to BlockSwan wap, it is worth noting that the DARTS architecture search required 24 GPU hours. By comparison, a BlockSwap search for an 800K parameter network took less than 5 minutes using a single Titan X Pascal GPU.
Given the strength of random baselines in architecture search settings (Liu et al., 2019a Yu et al. 2020; Li & Talwalkar 2019), also compare BlockSwap against randomly generated mixed- blocktype configurations in Appendix C BlockSwap consistently outperforms these, demonstrating that our Fisher potential metric is effective at selecting potent block structures.
7

---

Published as a conference paper at ICLR 2020
Table 2: CIFAR-10 top-1 test error for student nets, with parameter count (in thousands, P.(K)) and total MAC operations (in millions, Ops(M)). D-W specifies the number of layers and the width multiplier of the student. We compare BlockSwap reductions that rely on reducing depth (D) and width (W), single-blocktyp networks (MBConv6, DARTS, CondenseNet), and pruning via SNIP (Lee al., 2019). Comparisons to random configurations and él æl-pruning are given in Appendix C We do not report Ops for SNIP since this is dependent on the choice of sparse representation format. BlockSwap able to choose the networks with the lowest mean error all parameter budgets considered.
D-W Method
P. (K)
Ops (M)
EIT. + o
D-W Method
P. (K) Ops (M) ErT. 1z -
40-2
Teacher
2243.5
328.3
3.96 + 0.09
40-2 SNIP
404.2
4.67 + 0.11
40-2 SNIP
289.2
5.04 0.06
16-2
D-Scaled
691.7
101.4
4.78 + 0.13
40-2 SNIP
217.0
5.53 + 0.08
40-1
W-Scaled
563.9
83.6
4.57 +
40-2 SNIP
162.2
6.00 0.16
16-1
D-W-Scaled
175.1
26.8
+
40-2
BlockSwap 811.4
132.5
3.79 0.01
40-2
MBConv6
1500.9
231.7
+
40-2
BlockSwap wap 556.0
89.5
4.17 0.22
40-2
DARTS
321.5
52.8
4.49 0.03
40-2
BlockSwap 404.2
92.8
4.21 0.13
86-
CondenseNet
520.2
65.2
4.95 = 0.05
40-2
BlockS wap 289.2
65.9
4.45 0.18
40-2
SNIP
811.4
4.13 1 0.13
40-2
BlockSwap 217.0
38.8
4.68 = 0.37
40-2
SNIP
556.0
4.32 1 0.10
40-2
BlockSwap 162.2
33.9
5.17 0.00
6.5
BlockSwap
6.0
11 pruning
SNIP
Teat Estor (%)
5.5
MBConve
5.0
Condense
WRN-166 16-N
DARTS
WRN- 40-1
4.5
4.0
WRN-40-2
3.5
200K
400K
600K
800K
1.5M
2.2M
No. Parameters
Figure 3: CIFAR-10 top-1 test error of students versus parameters. BlockSwap models (blue) give lower error for each parameter budget when compared to depth/width reduced or pruned models. They also outperform single blocktype networks (MBConv, DARTS, CondenseNet-86). Note that all networks have been trained using attention transfer, or have been born-again (Furlanello al. 2018 in the case of CondenseNet.
Implementation Details: Networks are trained for 200 epochs using SGD with momentum 0.9. The initial learning rate of 0.1 is cosine annealed (Loshchilov Hutter 2017) to zero across the training run. Minibatches of size 128 are used with standard crop + flip data augmentation and Cutout VDies & Taylor 2017). The weight decay factor is set to 0.0005. For attention transfer | is set to 1000.
IMAGENET CLASSIFICATION
Here, we demonstrate that students chosen by BlockSwap succeed the more challenging ImageNet
dataset (Russakovsky et al. 2015). We use pretrained ResNet-34 (16 blocks, 21.8M parameters) as a teacher, and compare students parameter budgets (3M and 8M). We train BlockSwap student
8

---

Published as a conference paper at ICLR 2020
each of these budgets and compare their validation errors those of reduced depth/width student (8esNet18 and ResNet-18-00 a ResNet- 18 where the channel width in the last sections has been halved) and single-blocktye student (ResNet-34 with G(4) and G(N) blocks). The student networks found by BlockSwap for these two budgets are illustrated Appendix D Top-1 and top-5 validation errors presented in Table At both budgets, BlockSwap chooses networks that outperform its comparators, At 8M parameters it even surpasses the teacher by quite a margin. Specifically, it beats the teacher by 0.49% in top-1 error and 0.82% in top-5 error despite using almost 3x fewer parameters.
Implementation Details: Networks are trained with a cross-entropy loss for 100 epochs using SGD with momentum 0.9. The imitial learning rate of 0.1 is reduced by 10x every 30 epochs. Minibatches of size 256-split across 4 GPUs-are used with standard crop + flip augmentation. The weight decay factor is set to 0.0001. For attention transfer , is set to 750 using the output of each of the four sections of network.
Table 3: Top-1 and Top-5 classification errors (%) on the validation set of ImageNet for students trained with attention transfer from ResNet-34. We that for similar number of parameters, the student found from BlockSwap outperforms its counterparts, and in one instance, the teacher.
Model
Params
MACs
Top-1 err
Top-5 em
ResNet-34 Teacher
21.8M
3.669G
26.73
8.57
ResNet-18
11.7M
1.818G
29.18
10.05
ResNet-34-G(4)
8.1M
1.395G
26.58
8.43
BlockSwap
8.1M
1.242G
26.24
7.75
ResNet-18-0.5
3.2M
909M
37.20
15.02
ResNet-34-G(()
3.1M
559M
30.16
10.66
BlockSwap
3.1M
812M
29.57
10.20
7 COCO DETECTION
Thus far, we have used BlockSwap for image classification problems. Here we observe whether extends object detection on the COCO dataset (Lin al. specialelf,,,,lllly training 2017 train, and evaluating on 2017 val. We consider a Mask R CNN (He et al. 2017) with a ResNet-34 backbone, and apply BlockSwap using COCO images to obtain mixed- blocktyy backbone with 3M parameters. We compare this to single-blocktyy same number of parameters. To avoid conflation with ImageNet, we train everything from scratch. The results can be found in Table We can see that the BlockSwap backhone again outperforms its single-blocktype counterpart.
ResNet-34-G(N) backbone which uses the
Implementation Details:
Networks trained using the default Mask R-CNN settings in Torchvi- sion. We use a batch-size of 16 split across 8 GPUs. models are trained from scratch, and we forgo distillation due to memory constraints.
Table 4: Average Precisions (%) for COCO-2017 val detection for Mask R-CNNs using a ResNet-34. G(N) and BlockSwap backbone (each using 3M parameters).
Backbone
AP
Afto
APrs
AP'
APM
APr
ResNer34-G(N)
22.3
38.9
23.0
12.9
22.9
30.7
BlockSwap
23.4
40.0
24.7
13.6
24.2
31.1
8 CONCLUSION
We have developed BlockSwap: a fast, simple method for reducing large neural networks to flexible parameter targets based on block substitution. We have verified that these reduced networks make

---

Published as a conference paper at ICLR 2020
for excellent students, and have performed comprehensive ablation study. Future work could use BlockSwap to choose networks based on inference time, or energy cost instead of parameter count.
ACKNOWLEDGMENTS
This work was supported in part by the EPSRC Centre for Doctoral Training in Pervasive Parallelism and Huawei DDMPLab Innovation Research Grant, well funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No.732204 (Bonseyes). This work is supported by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number 16.0159. The opinions expressed and arguments employed herein do not necessarily reflect the official views of these funding bodies. The authors are grateful to David Sterratt for his LTEX prowess, and to the BayesWatch Watch team and anonymous reviewers for their helpful comments,
REFERENCES
Lei Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in Neural Information Processing Systems, 2014.
Changan Chen, Frederick Tung, Naveen Vedula, and Greg Mori. Constraint-Aware deep neural network compression. In European Conference Computer Vision, 2018a.
Liang-Chiae Chich Chen, Maxwell Collins, Yukun Zhu, George Papandreou, Barret Zoph, Florian Schroff, Hartwig Adam, and Jon Shlens. Searching for efficient multi-scale architectures for dense image prediction, In Advances Neural Information Processing Systems, 2018b.
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Towards the limit of network quantization. In International Conference Learning Representations, 2017.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE Conference Computer Vision and Patter Recognition, 2017.
Elliot J. Crowley, Gavin Gray, and Amos Storkey. Moonshine: Distilling with cheap convolutions. In Advances in Neural Information Processing Systems, 201 8a.
Elliot Crowley, Jack Turner, Amos Storkey, and Michael Boyle. closer look at structured pruning for neural network compression. arXiv preprint 18X0:00800, 2018b.
Misha Denil, Babak Shakibi, Laurent Dinh, Ranzato Mare' Aurelio, and Nando de Freitas. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems, 2013.
Terrance DeVries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 1708.v:700..444 04552, 2017.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. International Conference on Machine Learning, 2018.
Maximilian Golub, Guy Lemieux, and Mieszko Lis. Full deep neural network training on a pruned weight budget. In Proceedings of the SysML Conference, 2019.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient DNNs. In Advances in Neural Information Processing Systems, 2016.
Song Han, Huizi Mao, and William Dally, Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. In International Conference on Learning Representations, 2016.
Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems, 1993.
10

---

Published as a conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In International Conference on Computer Vision, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0253/ 2015.
Lu Hou, Quanming Yao, and James T. Kwok. Loss-aware binarization of deep networks. In International Conference on Learning Representations, 2017.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
Gao Huang. Schichen Liu, Laurens van der Maaten, and Kilian Q. Weinberger. CondenseNet: An efficient DenseNet using learned group convolutions. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2018.
Yani Ioannou, Duncan Robertson, Roberto Cipolla, and Antonio Criminisi. Deep roots: Improving CNN efficiency with hierarchical filter groups In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.
Jangho Kim, Sconguk Park, and Nojun Kwak. Paraphrasing complex network: Network compression
via factor transfer. In Advances in Neural Information Processing Systems, 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. ImageNet classification with deep convolu- tional neural networks. In Advances in Neural Information Processing Systems, 2012.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, 1989.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. SNIP: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019.
Liam Li and Ameet Talwalkar, Random search and reproducibility for neural architecture search. In Conference on Uncertainty in Artificial Intelligence, 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft COCO: Common objects in context, In European Conference on Computer Vision, 2014.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019a.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019b.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. In Advances in Neural Information Processing Systems, 2018.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Repre- sentations, 2017.
11

---

Published as a conference paper at ICLR 2020
Junran Peng, Lingxi Xie, Zhaoxiang Zhang, Tieniu Tan, and Jingdong Wang. Accelerating deep neural networks with spatial bottleneck modules. arXiv preprint (.Xiv:8899.02000 2018.
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In International Conference on Machine Learning, 2018.
Valentin Radu, Catherine Tong, Sourav Bhattacharya, Nicholas D. Lane, Cecilia Mascolo, Mahesh K. Marina, and Fahim Kawsar. Multimodal deep learning for activity and context recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(4):157, 2018.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for thin deep nets. In International Conference on Learning Representations, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. Int. Journal of Computer Vision (IJCV), 115(3):211-252, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. Mo- bileNetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2018
Suraj Srinivas and R. Venkatesh Babu. Data-free parameter pruning for deep neural networks. In British Machine Vision Conference, 2015.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. MnasNet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferene Huszár. Faster gaze prediction with dense networks and Fisher pruning. arXiv preprint arXiv: 1801005777 2018.
Gregor Urban, Krzysztof J. Geras, Samira Ebrahimi Kahou, Ozlem Aslan, Shengjie Wang, Rich Caruana, Abdelrahman Mohamed, Matthai Philipose, and Matt Richardson. Do deep convolu- tional nets really need to be deep and convolutional? In International Conference Learning Representations, 2017.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. FBNet: Hardware-awand efficient convnet design via differentiable neural architecture search. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluat- ing the search phase of neural architecture search. In International Conference Learning Representations, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference, 2016.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In International Conference on Learning Representutions, 2017.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V. Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018
12

---

Published as a conference paper at ICLR 2020
A BLOCK REPLACEMENTS
Table 5: Block substitutions used in this paper: Conv refers to a k x & convolution. GConv is grouped k x convolution and Conv1 x 1 is pointwise convolution. We assume that the input to each block has N channels and that channel size doesn't ' change unless written explicitly (x + y)- Where applicable, is the number of groups and b is the bottleneck contraction. The convolutional and batch-norm (BN) costs are given, although the latter significantly smaller. BN+ReLU precedes each convolution for WedeResNets and follows each convolution for standard ResNets.
Block
S
C(g)
B(b)
BG(b,g)
Structure
Conv Conv
GConv (g)
Conv1 1(N 5
Conv1 x 1(V N
Conv1 x
Conv
GConv(g)
GConv (g)
Conv1 x 1( N)
Conv1 x 1( N)
Conv1 x
Conv Params
2.
2N
N2 + +)
N + 2
7
ph3
BN Params
4.V
8N
[ + 5
[22 5
B UNDERSTANDING BLOCK PLACEMENT
To understand what makes for "good" block placement, we begin with the more tractable question: if were only cheapen one block, which one should we choose? In Figure happens when we substitute a single standard block with G (4) for each of the possible blocks for our WideResNet. We plot the final test error after training convergence, as well as the total parameter count after the substitution (the higher the bar, the higher the error, therefore the more sensitive the block).
show what
Notice that in Figure D [), at substituting the start and end of each stage appears to have a very detri- mental effect on accuracy, whereas mid-stage blocks can be substituted without hurting performance. For example, block 12 has few parameters, as shown in Figure but substituting it for a cheaper alternative results
drastic drop in accuracy. This implies that safest to substitute blocks the middle of stage but that blocks at the start and end of each stage should have their representational capacity preserved as much as possible.
We used this insight to analyse the block configurations of "good" and "bad" amongst the random architectures at each parameter budget. With several blocktypes, however, we need a method to rank make the simplifying assumption that training the blocktype
the capacity of blocktypes; own is a good indicator of this (though this is not always true, since blocks give varying performance with different input sizes). An example of good and bad network judged this way can be seen in Figure 5} while they do roughly follow our observations, there are clearly more complex interactions
its
4.8
2.25M
2.20M
4.6
2.15M
Error
4.4
2.10M
2.05M
4.2
2.00M
4.0
1.95M
2
10 11
12
13
17
5
12 13
14
15
16
17
Substituled block
Substituled black (b)
(a)
Figure 4: Each bar shows WideResNNet40-22 with one block substituted for G (4), where (a) shows final CIFAR-10 10 error and (b) shows resulting number of parameters, with blocks coloured by network stage. This shows us the sensitivity of each block; for example, block 12 has relatively few parameters, but substituting it for a cheaper block very detrimental final error.
13

---

Published as a conference paper at ICLR 2020
model (5.25% error)
Good mode] (4.62% error)
Capacity Score
90
Capacity Score
80
80
15
Block
Block
(+)
(b)
Figure 5: An example of (a) bad architecture and (b) good architecture for given parameter budget (200K), with blocks coloured by network stage. Capacity score is given by the accuracy that the selected blocktype achieves after being trained on its own. We can see that the good model (b) follows the pattern observed in Figure 4 while the bad model (a) does not.
play (perhaps unsurprising given the random nature of their construction). We found that overall, the models in our ablation set adhere to the pattern described above (for each stage: high capacity blocks at the start and end, increasing capacity throughout), with some variance.
C COMPARING TO RANDOM CONFIGURATIONS AND (1-PRUNING
Table Comparing BlockSwap to randomly configured mixed- blockktp networks and El-pruned versions of the original teacher on the CIFAR-10 dataset. Total MAC operations are not reported for é1 11-pruning because calculating this is highly dependent on choice of sparse tensor representation
format.
D-W
Method
Parameters (K)
MAC Ops (M)
Error (μ + σ)
40-2
BlockSwap
811.4
132.5
3.79 4 0.01
40-2
BlockSwap
556.0
89.5
4.17 + 0.22
40-2
BlockSwap
404.2
92.8
4.21 + 0.13
40-2
BlockSwap
289.2
65.9
4.45 + 0.18
40-2
BlockSwap
217.0
38.8
4.68 0.37
40-2
BlockSwap
162.2
33.9
5.17 0.00
40-2
Random
795.7
81.6
4.26 0.04
40-2
Random
551.1
76.7
4.14
40-2
Random
397.4
59.5
4.91
40-2
Random
285.1
56.7
4.80
40-2
Random
217.6
46.9
5.13 + 0.24
40-2
Random
168.3
33.6
5.75 + 0.09
40-2
21 (1-pruning
894.7
4.12 + 0.03
40-2
{1 11-pruning
760.5
4.32 0.06
40-2
fl-pruning
671.1
4.51 - 0.13
40-2
fl-pruning
536.8
4.98 0.21
40-2
fl-pruning
447.4
5.60 +: 0.23
40-2
f1-pruning
313.2
7.70 0.18
40-2
El-pruning
223.7
11.93 = 0.77
14

---

Published as a conference paper at ICLR 2020
D
BLOCKSWAP NETWORKS FOR IMAGENET
(88)
BGF2
(12)
G(N2)
B(2)
G(16)
BG(2.16)
BG2.M/1
G(N44)
BG(2.2)
BG28
BG(2.M22
G(V4)
BG(2,4)
M12)
(30)
RANu
(22)
SoNA (
G(2)
5
1914
(91
G(2)
G(4)
Figure 6: ResNet-34 block substitutions chosen by BlockSwap at 3M (top) and 8M (bottom) parame- ters on ImageNet.
15


Reviews:
[{"review_id": "rkxUDhtpFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1) in the introduction, it's not clear what attention transfer means \n  2) why did they specifically choose these four blocks? \n  3) (3.1) substitute blocks $S$: ``each using $N$ lots of $N \\times k \\times k$ filters\" should it be ``$N$ number of $k \\times k$ filters\"? \n  4) (3.2) ``consider a choice of layers $i = 1, 2, \\dots, N_L$\" what is $L$ and why does the number of layers depend on $L$? \n  5)  (3.2) Why is that specific $f$ chosen? \n  6)  (3.3) They could elaborate more on why they chose Fisher potential instead of other metrics for architecture selection. Currently, they only provided the intuition of the metric. Since the paper suggests the Fisher potential is a crucial part of their method, they could provide more theoretical justification about this choice. \n  7) (4.0) How did they decide on the hyperparameters? \n  8) In the introduction, they suggest that the major novelty of their method ``assigns non-uniform importance to each block by cheapening them to different extents\", but in their method they only randomly assembled the mixed-blocktype models. \n  9) Section 4.1 suggests good mixed blocktype yields low error but they didn't address how good mixed blocktype can be found ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rkl7kPYxcS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies a fast algorithm for choosing networks with interleaved block types – BlockSwap. It passes a single minibatch of training data through randomly initialized networks and gauging their Fisher potential. \nThe teacher-student network is used here to learn compressed network on a budget. They conduct various experiments, including cifar-10, imagenet, and coco-detection. All the experiments show the advance of proposed model, which is quite remarkable.\nOverall, this paper is well organized, and very well written. The insightful experiments thoroughly discuss and compare the proposed method. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByxSoEyLKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an approach to compressing a deep network so as to satisfy a given budget. In contrast to methods that replace all convolutional blocks with the same cheaper alternative, the authors argue that mixing different types of such cheap alternatives should be effective. To achieve this in a fast manner, they propose to randomly sample architectures with mixed blocktypes that satisfy the given budget and rank these architectures using Fisher information.\n\nOriginality:\n- The approach is simple, but seems effective. Although the different components that make up the final method are known, they are put together so as to address a different task, in a way that I find interesting.\n\nMethodology:\n- From the paper, it is not entirely clear how sampling under budget constraints is achieved. I imagine that one could do it in a naive way, by choosing the first block uniformly randomly, and then removing the candidates that do not fit the budget for the subsequent blocks. This, however, would seem to give a different importance to the early blocks than to the later ones, since the former would essentially not have any budget constraints. I would appreciate it is the authors could explain their strategy in more detail.\n- The assumption that the number of blocks in the student is the same as in the teacher seems constraining. A better architecture might be obtained by having fewer, wider blocks, or more, thinner blocks. Do the authors see a potential way to address this limitation?\n- In principle, it seems that the proposed Fisher Information-based strategy could be used for general NAS, not just for compact architectures. Have the authors investigated this direction?\n\nRelated work:\n- It seems to me that the literature review on compression/pruning is a bit shallow. I acknowledge, however, that most works do not tackle the scenario where a budget is given. However, Chen et al., \"Constraint-aware Deep Neural Network Compression\", ECCV 2018, do, and it would be interesting to discuss and provide comparisons with this work.\n\nExperiments:\n- I appreciate the ablation study, which answered several of my questions.\n- In Table 1, it seems counterintuitive that the (negative) correlation becomes smaller as the number of mini-batches increases. Do the authors have an explanation for this?\n- In Section 5, are the baseline compact networks all trained using the same Attention Transfer algorithm as for the BlockSwap ones?\n- In Table 2, the budget values (P. (K)) seem fairly arbitrary? How were they obtained? They seem to match those of SNIP. Is this because the authors ran SNIP, and then set their budget accordingly?\n- Below Fig. 3, the authors mention that they are generous with sparsity-based methods because they count the number of non-zero parameters. Note that several structured-sparsity compression methods have been proposed (Alvarez & Salzmann, NIPS 2016, 2017; Wen et al., NIPS 2016), which, by contrast with regular sparsity ones, would cancel out entire filters.\n\nSummary:\nOverall, I like the simple idea proposed in this paper. I would however appreciate it if the authors could clarify the way they sample the architectures and address my questions about their experiments.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two reviewers recommend acceptance. One reviewer is negative, however, does not provide reasons for rejection. The AC read the paper and agrees with the positive reviewers. in that the paper provides value for the community on an important topic of network compression.
Decision: Accept (Poster)