Computation Reallocation for Object Detection

1 INTRODUCTION . Object detection is one of the fundamental tasks in computer vision . The backbone feature extractor is usually taken directly from classification literature ( Girshick , 2015 ; Ren et al. , 2015 ; Lin et al. , 2017a ; Lu et al. , 2019 ) . However , comparing with classification , object detection aims to know not only what but also where the object is . Directly taking the backbone of classification network for object detectors is sub-optimal , which has been observed in Li et al . ( 2018 ) . To address this issue , there are many approaches either manually or automatically modify the backbone network . Chen et al . ( 2019 ) proposes a neural architecture search ( NAS ) framework for detection backbone to avoid expert efforts and design trails . However , previous works rely on the prior knowledge for classification task , either inheriting the backbone for classification , or designing search space similar to NAS on classification . This raises a natural question : How to design an effective backbone dedicated to detection tasks ? To answer this question , we first draw a link between the Effective Receptive Field ( ERF ) and the computation allocation of backbone . The ERF is only small Gaussian-like factor of theoretical receptive field ( TRF ) , but it dominates the output ( Luo et al. , 2016 ) . The ERF of image classification task can be easily fulfilled , e.g . the input size is 224×224 for the ImageNet data , while the ERF of object detection task need more capacities to handle scale variance across the instances , e.g . the input size is 800×1333 and the sizes of objects vary from 32 to 800 for the COCO dataset . Lin et al . ( 2017a ) allocates objects of different scales into different feature resolutions to capture the appropriate ERF in each stage . Here we conduct an experiment to study the differences between the ERF of several FPN features . As shown in Figure 1 , we notice the allocation of computation across different resolutions has a great impact on the ERF . Furthermore , appropriate computation allocation across spacial position ( Dai et al. , 2017 ) boost the performance of detector by affecting the ERF . Based on the above observation , in this paper , we aim to automatically design the computation allocation of backbone for object detectors . Different from existing detection NAS works ( Ghiasi et al. , 2019 ; Ning Wang & Shen , 2019 ) which achieve accuracy improvement by introducing higher computation complexity , we reallocate the engaged computation cost in a more efficient way . We propose computation reallocation NAS ( CR-NAS ) to search the allocation strategy directly on the detection task . A two-level reallocation space is conducted to reallocate the computation across different resolution and spatial position . In stage level , we search for the best strategy to distribute the computation among different resolution . In operation level , we reallocate the computation by introducing a powerful search space designed specially for object detection . The details about search space can be found in Sec . 3.2 . We propose a hierarchical search algorithm to cope with the complex search space . Typically in stage reallocation , we exploit a reusable search space to reduce stage-level searching cost and adapt different computational requirements . Extensive experiments show the effectiveness of our approach . Our CR-NAS offers improvements for both fast mobile model and accurate model , such as ResNet ( He et al. , 2016 ) , MobileNetV2 ( Sandler et al. , 2018 ) , ResNeXt ( Xie et al. , 2017 ) . On the COCO dataset , our CR-ResNet50 and CR-MobileNetV2 can achieve 38.3 % and 33.9 % AP , outperforming the baseline by 1.9 % and 1.7 % respectively without any additional computation budget . Furthermore , we transfer our CR-ResNet and CR-MobileNetV2 into the another ERF-sensitive task , instance segmentation , by using the Mask RCNN ( He et al. , 2017 ) framework . Our CR-ResNet50 and CR-MobileNetV2 yields 1.3 % and 1.2 % COCO segmentation AP improvement over baseline . To summarize , the contributions of our paper are three-fold : • We propose computation reallocation NAS ( CR-NAS ) to reallocate engaged computation resources . To our knowledge , we are the first to dig inside the computation allocation across different resolution . • We develop a two-level reallocation space and hierarchical search paradigm to cope with the complex search space . Typically in stage reallocation , we exploit a reusable model to reduce stage-level searching cost and adapt different computational requirements . • Our CR-NAS offers significant improvements for various types of networks . The discovered models show great transferablity over other detection neck/head , e.g . NAS-FPN ( Cai & Vasconcelos , 2018 ) , other dataset , e.g . PASCAL VOC ( Everingham et al. , 2015 ) and other vision tasks , e.g . instance segmentation ( He et al. , 2017 ) . 2 RELATED WORK . Neural Architecture Search ( NAS ) Neural architecture search focus on automating the network architecture design which requires great expert knowledge and tremendous trails . Early NAS approaches ( Zoph & Le , 2016 ; Zoph et al. , 2018 ) are computational expensive due to the evaluating of each candidate . Recently , weight sharing strategy ( Pham et al. , 2018 ; Liu et al. , 2018 ; Cai et al. , 2018 ; Guo et al. , 2019 ) is proposed to reduce searing cost . One-shot NAS method ( Brock et al. , 2017 ; Bender et al. , 2018 ; Guo et al. , 2019 ) build a directed acyclic graph G ( a.k.a . supernet ) to subsume all architectures in the search space and decouple the weights training with architectures searching . NAS works only search for operation in the certain layer . our work is different from them by searching for the computation allocation across different resolution . Computation allocation across feature resolutions is an obvious issue that has not been studied by NAS . We carefully design a search space that facilitates the use of existing search for finding good solution . NAS on object detection . There are some work use NAS methods on object detection task ( Chen et al. , 2019 ; Ning Wang & Shen , 2019 ; Ghiasi et al. , 2019 ) . Ghiasi et al . ( 2019 ) search for scalable feature pyramid architectures and Ning Wang & Shen ( 2019 ) search for feature pyramid network and the prediction heads together by fixing the architecture of backbone CNN . These two work both introduce additional computation budget . The search space of Chen et al . ( 2019 ) is directly inherited from the classification task which is suboptimal for object detection task . Peng et al . ( 2019 ) search for dilated rate on channel level in the CNN backbone . These two approaches assume the fixed number of blocks in each resolution , while we search the number of blocks in each stage that is important for object detection and complementary to these approaches . 3 METHOD . 3.1 BASIC SETTINGS . Our search method is based on the Faster RCNN ( Ren et al. , 2015 ) with FPN ( Lin et al. , 2017a ) for its excellent performance . We only reallocate the computation within the backbone , while fix other components for fair comparison . For more efficient search , we adopt the idea of one-shot NAS method ( Brock et al. , 2017 ; Bender et al. , 2018 ; Guo et al. , 2019 ) . In one-shot NAS , a directed acyclic graph G ( a.k.a . supernet ) is built to subsume all architectures in the search space and is trained only once . Each architecture g is a subgraph of G and can inherit weights from the trained supernet . For a specific subgraph g ∈ G , its corresponding network can be denoted as N ( g , w ) with network weights w . 3.2 TWO-LEVEL ARCHITECTURE SEARCH SPACE . We propose Computation Reallocation NAS ( CR-NAS ) to distribute the computation resources in two dimensions : stage allocation in different resolution , convolution allocation in spatial position . 3.2.1 STAGE REALLOCATION SPACE . The backbone aims to generate intermediate-level features C with increasing downsampling rates 4× , 8× , 16× , and 32× , which can be regarded as 4 stages . The blocks in the same stage share the same spatial resolution . Note that the FLOPs of a single block in two adjacent spatial resolutions remain the same because a downsampling/pooling layer doubles the number of channels . So given the number of total blocks of a backbone N , we can reallocate the number of blocks for each stage while keep the total FLOPs the same . Figure 2 shows our stage reallocation space . In this search space , each stage contains several branches , and each branch has certain number of blocks . The numbers of blocks in different branches are different , corresponding to different computational budget for the stage . For example , there are 5 branches for the stage 1 in Figure 2 , the numbers of blocks for these 5 branches are , respectively , 1 , 2 , 3 , 4 , and 5 . We consider the whole network as a supernet T = { T1 , T2 , T3 , T4 } , where Ti at the ith stage hasKi branches , i.e . Ti = { tki |k = 1 ... Ki } . Then an allocation strategy can be represented as τ = [ τ1 , τ2 , τ3 , τ4 ] , where τi denote the number of blocks in the ith branch . All blocks in the same stage have the same structure . ∑4 i=1 τi = N for a network with N blocks . For example , the original ResNet101 has τ = [ 3 , 4 , 23 , 3 ] and N = 33 residual blocks . We make a constraint that each stage at least has one convolutional block . We would like to find the best allocation strategy of ResNet101 is among the ( 32 3 ) possible choices . Since validating a single detection architecture requires hundreds of GPU-hours , it not realist to find the optimal architecture by human trails . On the other hand , we would like to learn stage reallocation strategy for different computation budgets simultaneously . Different applications require CNNs of different numbers of layers for achieving different latency requirements . This is why we have ReseNet18 , ReseNet50 , ReseNet101 , etc . We build a search space to cover all the candidate instances in a certain series , e.g . ResNet series . After considering the trade off between granularity and range , we set the numbers of blocks for T1 and T2 as { 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 } , and set the numbers of blocks for T3 as { 2 , 3 , 5 , 6 , 9 , 11 , 14 , 17 , 20 , 23 } , for T4 as { 2 , 3 , 4 , 6 , 7 , 9 , 11 , 13 , 15 , 17 } for the ResNet series . The stage reallocation space of MobileNetV2 ( Sandler et al. , 2018 ) and ResNeXt ( Xie et al. , 2017 ) can be found in Appendix A.2 . 3.2.2 CONVOLUTION REALLOCATION SPACE . To reallocate the computation across spatial position , we utilize dilated convolution Li et al . ( 2019 ) , Li et al . ( 2018 ) . Dilated convolution effects the ERF by performing convolution at sparsely sampled locations . Another good feature of dilated convolution is that dilation introduce no extra parameter and computation . We define a choice block to be a basic unit which consists of multiple dilations and search for the best computation allocation . For ResNet Bottleneck , we modify the center 3 × 3 convolution . For ResNet BasicBlock , we only modify the second 3 × 3 convolution to reduce search space and searching time . We have three candidates in our operation set O : { dilated convolution 3 × 3 with dilation rate i|i = 1 , 2 , 3 } . Across the entire ResNet50 search space , there are therefore 316 ≈ 4× 107 possible architectures .

Summary: The paper attempts to apply neural architecture search (NAS) to re-arrange, or re-allocate the network backbone blocks and the convolution filters for object detection. The search space is two-fold: 1) the network is allowed to search over allocation of different number of blocks in the backbone (e.g. ResNet, MobileNet); 2) the network is allowed to choose the dilation of each of the block. A one-shot NAS method is adopted for efficient search. After search, the model is shown to have 1) better AP results; and 2) more balanced effective receptive field (ERF).