BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget

1 INTRODUCTION . Deep Convolutional Neural Networks are extremely popular , and demonstrate strong performance on a variety of challenging tasks . Because of this , there exist a large range of scenarios in the wild in which practitioners wish to deploy these networks e.g . pedestrian detection in a vehicle ’ s computer , human activity recognition with wearables ( Radu et al. , 2018 ) . These networks are largely over-parameterised ; a fact that can be exploited when specialising networks for different resource budgets . For example , there is a wealth of work demonstrating that it is possible to replace expensive convolutional blocks in a large network with cheap alternatives e.g . those using grouped convolutions ( Chollet , 2017 ; Xie et al. , 2017 ; Ioannou et al. , 2017 ; Huang et al. , 2018 ) or bottleneck structures ( He et al. , 2016 ; Sandler et al. , 2018 ; Peng et al. , 2018 ) . This creates a smaller network which may be used as a student , and trained through distillation ( Ba & Caruana , 2014 ; Hinton et al. , 2015 ) with the original large network as a teacher to retain performance . Typically , each network block is replaced with a single cheap alternative , producing a single-blocktype network for a given budget . By cheapening each block equally , one relies on the assumption that each block is of equal importance . We posit instead that for each budget , there exist more powerful mixed-blocktype networks that assign non-uniform importance to each block by cheapening them to different extents . We now have a paradox of choice ; for a given budget , it is not obvious which cheap alternatives to use for our student network , nor where to place them . Let ’ s assume we have a large network that consists of B expensive convolutional blocks , and a candidate pool of C cheap blocks that we could substitute each of these out for , and we are given a limit on the number of parameters ( and therefore memory ) that the network can use . Which of ∗Equal contribution . †Now at the University of Toronto . the CB mixed-blocktype networks is appropriate ? It rapidly becomes infeasible to exhaustively consider these networks , even for single digit C. We could turn to neural architecture search ( NAS ) techniques ( Zoph & Le , 2017 ; Zoph et al. , 2018 ; Pham et al. , 2018 ; Tan et al. , 2019 ; Wu et al. , 2019 ; Liu et al. , 2019a ) but these introduce a gigantic computational overhead ; even one of the fastest such approaches ( Pham et al. , 2018 ) still requires half a day of search time on a GPU . Moreover , in real-world scenarios , allocated budgets can change quickly ; we would like to be able to find a good network in a matter of minutes , not hours or days . Our goal in this paper is , given a desired parameter budget , to quickly identify a suitable mixedblocktype version of the original network that makes for a powerful student . We present a simple method—BlockSwap—to achieve this . First , we randomly sample a collection of candidate mixedblocktype architectures that satisfy the parameter budget . A single minibatch is then pushed through each candidate network to calculate its Fisher potential : the sum of the total ( empirical ) Fisher information for each of its blocks . Finally , the network with the highest potential is selected as a student and trained through the distillation method of attention transfer ( Zagoruyko & Komodakis , 2017 ) with the original teacher . Our method is illustrated in Figure 1 . In Section 3 we describe the block substitutions used in BlockSwap , distillation via attention transfer , and Fisher information . We elaborate on our method in Section 4 as well as providing a comprehensive ablation study . Finally , we experimentally verify the potency of BlockSwap on CIFAR-10 ( Section 5 ) as well as ImageNet ( Section 6 ) and COCO ( Section 7 ) . Our contributions are as follows : 1 . We introduce BlockSwap , an algorithm for reducing large neural networks by performing block-wise substitution . We show that this outperforms other top-down approaches such as depth/width scaling , parameter pruning , and random substitution . 2 . We outline a simple method for quickly evaluating candidate models via Fisher information , which matches the performance of bottom-up approaches while reducing search time from days to minutes . 3 . We conduct ablation studies to validate our methodology , highlighting the benefits of block mixing , and confirming that our ranking metric is highly correlated to the final error . 2 RELATED WORK . Neural networks tend to be overparameterised : Denil et al . ( 2013 ) accurately predict most of the weights in a network from a small subset ; Frankle & Carbin ( 2019 ) hypothesise that within a large network , there exists a fortuitously initialised subnetwork that drives its performance . However , it remains difficult to exploit this overparameterisation without taking a hit in performance . One means to combat this is to use a large teacher network to regularise the training of a small student network ; a process known as distillation . The small network is trained from scratch , but is also forced to match the outputs ( Ba & Caruana , 2014 ; Hinton et al. , 2015 ) or activation statistics ( Romero et al. , 2015 ; Zagoruyko & Komodakis , 2017 ; Kim et al. , 2018 ) of the teacher using an additional loss term . When utilising distillation one must decide how to create a student network . A simple approach would be to reduce the depth of the original large network , although this can prove detrimental ( Urban et al. , 2017 ) . An effective strategy is to create a student by replacing all the teacher ’ s convolutions with grouped alternatives ( Crowley et al. , 2018a ) . Grouped convolutions are a popular replacement for standard convolutions as they drastically cut the number of parameters used by splitting the input along the channel dimension and applying a much cheaper convolution on each split . They were originally used in AlexNet ( Krizhevsky et al. , 2012 ) due to GPU memory limitations , and have appeared in several subsequent architectures ( Ioffe & Szegedy , 2015 ; Chollet , 2017 ; Xie et al. , 2017 ; Ioannou et al. , 2017 ; Huang et al. , 2018 ) . However , as the number of groups increases , fewer channels are mixed , which hinders representational capacity . MobileNet ( Howard et al. , 2017 ) compensates for this by following its heavily-grouped depthwise convolutions by a pointwise ( 1× 1 ) convolution to allow for channel mixing . The increasing complexity of neural network designs has encouraged the development of methods for automating neural architecture search ( NAS ) . Zoph et al . ( 2018 ) use an RNN to generate network block descriptions and filter the options using reinforcement learning . These blocks are stacked to form a full neural network . This is an extremely expensive process , utilising 450 GPUs over the course of 3 days . To address this , Pham et al . ( 2018 ) propose giving all models access to a shared set of weights , achieving similar performance to Zoph et al . ( 2018 ) with a single GPU in less than 24 hours . Subsequent works have made extensive use of this technique ( Liu et al. , 2019a ; Luo et al. , 2018 ; Chen et al. , 2018b ) . However , it has been observed that under the constrained architecture search space of the above methods , random architecture search provides a competitive baseline ( Li & Talwalkar , 2019 ; Yu et al. , 2020 ) . In particular , Yu et al . ( 2020 ) show that weight sharing hampers the ability of candidate networks to learn and causes many NAS techniques to find suboptimal architectures . NAS techniques predominantly take a bottom-up approach ; they find a powerful building block , and form neural networks using stacks of these blocks . Other works have taken a top-down approach to find architectures using pruning ( Chen et al. , 2018a ; Lee et al. , 2019 ; Liu et al. , 2019b ; Crowley et al. , 2018b ; Frankle & Carbin , 2019 ) . Chen et al . ( 2018a ) take a pre-trained network and apply a principled framework to compress it under operational constraints such as latency . In SNIP ( Lee et al. , 2019 ) , one randomly initialises a large network and quantifies the sensitivity of each connection using gradient magnitudes . The lowest sensitivity connections are removed to produce a sparse architecture which is then trained as normal . 3 PRELIMINARIES . 3.1 SUBSTITUTE BLOCKS . Here , we will briefly elaborate on the block substitutions used for BlockSwap . A tabular comparison is given in Appendix A . The block choices are deliberately simplistic . We can therefore demonstrate that it is the combination of blocks that is important rather than the representational capacity of a specific highly-engineered block e.g . one from the NAS literature . The blocks considered are variations of the standard block used in residual networks . In the majority of these blocks , the input has the same number of channels as the output , so we describe their parameter cost assuming this is the case . This standard block contains two convolutional layers , each using N lots of N × k × k filters where k is the kernel size . Assuming the costs of batch-norm ( BN ) layers and shortcut convolutions ( where applicable ) are negligible , the block uses a total of 2N2k2 parameters . Grouped+Pointwise Block – G ( g ) In a grouped convolution , the N channel input is split along the channel dimension into g groups , each of which has N/g channels . Each group goes through its own convolution which outputs N/g channels , and all the outputs are concatenated along the channel dimension . This uses g × ( N/g ) 2k2 = ( N2k2 ) /g parameters . To compensate for reduced channel mixing , each convolution is further followed by a pointwise ( 1× 1 ) convolution , incurring an extra cost of N2 . For this block , each full convolution has been replaced with a grouped+pointwise convolution , and so the block uses 2 ( ( N2k2 ) /g +N2 ) parameters . Bottleneck Block – B ( b ) In this block , a pointwise convolution is used to reduce the number of channels of the input by a factor of b before a standard convolution is applied . Then , another pointwise convolution brings the channel size back up . This uses ( N/b ) 2k2 + 2N2/b parameters . Bottleneck Grouped+Pointwise Block – BG ( b , g ) This is identical to the bottleneck block , except the standard convolution is further split into g groups , and so uses ( N/bg ) 2k2 + 2N2/b parameters . 3.2 DISTILLATION VIA ATTENTION TRANSFER . Attention transfer ( Zagoruyko & Komodakis , 2017 ) is a distillation technique whereby a student network is trained such that its attention maps at several distinct attention points are made to be similar to those produced by a large teacher network . A formal definition follows : Consider a choice of layers i = 1 , 2 , ... , L in a teacher network with L layers , and the corresponding layers in the student network . At each chosen layer i of the teacher network , collect the spatial map of the activations for channel j into the vector atij . Let A t i collect a t ij for all j . Likewise for the student network we correspondingly collect into asij and A s i . Now given some choice of mapping f ( Ai ) that maps each collection of the form Ai into a vector , attention transfer involves learning the student network by minimising LAT = LCE + β L∑ i=1 ∥∥∥∥ f ( Ati ) ||f ( Ati ) ||2 − f ( A s i ) ||f ( Asi ) ||2 ∥∥∥∥ 2 , ( 1 ) where β is a hyperparameter , and LCE is the standard cross-entropy loss . In Zagoruyko & Komodakis ( 2017 ) the authors use f ( Ai ) = ( 1/NAi ) ∑NAi j=1 a 2 ij , where NAi is the number of channels at layer i .

Summary: This paper introduces an approach to compressing a deep network so as to satisfy a given budget. In contrast to methods that replace all convolutional blocks with the same cheaper alternative, the authors argue that mixing different types of such cheap alternatives should be effective. To achieve this in a fast manner, they propose to randomly sample architectures with mixed blocktypes that satisfy the given budget and rank these architectures using Fisher information.