Multi-scale Network Architecture Search for Object Detection

1 INTRODUCTION . Recognizing and localizing objects at vastly different scales is a fundamental challenge in object detection . Detection performance for objects with different scales is highly related to features with different properties such as feature resolution , receptive fields , and feature fusion ways . The key to solving the multi-scale problem in object detection is how to build a multi-scale network that has proper high-level semantic features for objects with different scales . A recent work in object detection Feature Pyramid Networks ( FPN ) ( Lin et al. , 2017 ) has achieved remarkable success in multi-scale feature design and has been commonly used by many modern object detectors ( He et al. , 2017 ; Lin et al. , 2020 ; Lu et al. , 2019 ) . FPN extracts multi-scale intermediate features from the encoder network and assigns objects grouped by scales to corresponding features according to a heuristic rule . Another prevalent detection framework , SSD ( Liu et al. , 2016 ) , conducts feature generation by a lighter encoder network without upsampling operators . The basic idea to deal with the multi-scale detection problem can be summarized as below . Given the input image , a series of feature maps with the various resolution are generated to detect objects grouped by scale range . We note it as multi-scale feature production . In FPN and its variants , the multi-scale feature production is split into two steps , feature generation and feature utilization . In terms of feature generation , an encoder network composed of blocks provides features with different scales . And the strategy of feature utilization determines the rule of assigning objects to feature maps . These two steps are closely related to each other . Although FPN has achieved promising results on multi-scale object detection tasks , the production of multi-scale features is quite hand-crafted and relies heavily on the experiences of human experts . More specifically , network architectures of FPN are based on a downsample-upsample architecture which may not be effective enough . By changing the downsampling and upsampling operation ’ s positions and numbers , we could obtain many other candidates to generate different multi-scale features . Also , the predefined rule of feature utilization is very empirical and other alternatives may lead to better performance . Therefore we wonder : Can we find network architectures that can build better semantic feature representation for multiple scales ? The answer is yes . Recent advances in neural architecture search have shown promising results compared with handcrafted architecture by human experts ( Zoph et al. , 2018 ; Liu et al. , 2019b ; Cai et al. , 2019 ; Guo et al. , 2019 ) . Several works have also focused on neural architecture search in object detection tasks ( Chen et al. , 2019 ; Ghiasi et al. , 2019 ; Du et al. , 2019 ) , but generating and utilizing multi-scale features are still not well explored . DetNAS ( Chen et al. , 2019 ) adopts the method mainly designed on image classification to search the operations of backbone networks in object detectors . NASFPN ( Ghiasi et al. , 2019 ) focuses on searching for better feature-fusion connections in the neck part of FPN . NAS-FPN doesn ’ t optimize the whole encoder network and still relies on predefined backbone architecture . Recently SpineNet ( Du et al. , 2019 ) proposes a search method with scalepermuted features and cross-scale connections by reinforcement learning , but the search cost is quite large . All these previous works focus on designing better neural network architectures to generate better features given a fixed feature selection strategy . However , they fail to conduct a complete flexible multi-scale feature production strategy . In this paper , we propose a new method to take into account of both aspects and build detection networks with the strong and proper multi-scale feature production strategy by neural architecture search . For feature generation , we put forward a network stride search method to generate multiple feature representations for different scales . Different from the scale-decreasing-increasing architecture of FPN , the scale of our networks can decrease or increase at each block , as illustrated in Figure 1 . By stride search for each block , we could significantly explore a wide range of possible feature generation designs of multi-resolution networks . Most backbones of object detectors are originally designed on image classification without multi-scale problems . However , stride configuration in the encoder network would be optimized in the context of the multi-scale task . Moreover , more complex cross-scale feature fusions might appear according to more complex internal scale changes . For feature utilization , we change the previous one-to-one mapping strategy into a more flexible feature selection . Since each group with objects of the same scale range owns one detection head , feature utilization is implemented by selecting proper features for detection heads . Objects of different scale ranges might be assigned to the same feature map . It is not possible in previous methods , as shown in Figure 1 ( b ) . By jointly optimizing feature generation and utilization of multi-scale features , we search for flexible but complete multi-scale feature production strategies . Extensive experiments demonstrate complete multi-scale feature production search is critical to building strong and proper semantic features for object detection with different scales . On challenging COCO dataset ( Lin et al. , 2014 ) , our method obtains a 2.6 % , 1.5 % , 1.2 % mAP improvement with similar FLOPs as ResNet18-FPN , ResNet34FPN , ResNet50-FPN . 2 RELATED WORK . Neural Architecture Search Neural Architecture Search aims to design better network architectures automatically . RL-based methods ( Zoph et al. , 2018 ; Zoph & Le , 2017 ) have achieved great success despite a huge computation cost . In differentiable algorithms ( Liu et al. , 2019b ; Cai et al. , 2019 ) , architecture parameters are employed and operators in the search space are considered as the weighted sum of candidate operators . There exist difficulties to deal with operators with different strides . Super-nets , acting as the collection of weights shared by all the sub-architectures , and evolutionary search are involved in one-shot NAS ( Guo et al. , 2019 ; Bender et al. , 2018 ) . But it ’ s difficult to ensure strong correlations between one-shot and stand-alone performances of the sub-architectures . Multi-scale Object Detection SSD ( Liu et al. , 2016 ) uses multi-scale features generated by different stages of the backbone network to detect objects of different scales . Feature pyramid architectures are utilized in FPN ( Lin et al. , 2017 ) and RetinaNet ( Lin et al. , 2020 ) to obtain multi-scale features . SNIP ( Singh & Davis , 2018 ) includes the image pyramid architecture to deal with multi-scale detection . Frameworks with multi-scale features are prevalent as objects of different scales appear in one image . Neural Architecture Search on Object Detection DetNAS ( Chen et al. , 2019 ) , NAS-FCOS ( Wang et al. , 2019 ) and Auto-FPN ( Xu et al. , 2019 ) focus on the architecture of the top-down pathway and feature fusion . SM-NAS ( Yao et al. , 2019 ) and CR-NAS ( Liang et al. , 2020 ) try to adjust the computation occupied by different parts of detectors . Also , there are several works aiming to improve FPN using NAS . NAS-FPN ( Ghiasi et al. , 2019 ) improves detection performance by searching for better connections within the feature pyramid network . It is limited without modification to the overall encoder architecture . And it fails to take the multi-scale feature utilization into account . EfficientDet ( Tan et al. , 2019 ) and Auto-FPN ( Xu et al. , 2019 ) search better feature fusion for FPN with differentiable methods . Other relative works like Liu et al . ( 2019a ) conduct similar modifications . Recently , SpineNet ( Du et al. , 2019 ) proposes a backbone search method with scale-permuted features and cross-scale connections by reinforcement learning . Our work has several major differences from it . First , the search space of MSNAS is designed that each operator in the network can downsample or upsample instead of permutation , which builds a much larger search space than SpineNet . Second , the complete multi-scale feature production is considered in our work , while SpineNet only focuses on the architecture of the encoder network . Lastly , our method is based on the one-shot search method instead of reinforcement learning in SpineNet . Our method is much more efficient and requires much less computation cost than SpineNet . 3 METHOD . We start by discussing multi-scale feature production for the object detection network in Section 3.1 . In Section 3.2 we will introduce how to build the search space and search proper stride in detection networks to obtain better multi-scale features . In Section 3.3 , how to search the appropriate strategy of feature utilization is presented . Finally , details about super-net training and search strategy are described in Section 3.4 . 3.1 FEATURE PRODUCTION FOR DETECTION NETWORK . In this section , we ’ ll discuss the production of multi-scale features and define the problem in detail . As noted above , the basic idea of handling the multi-scale detection problem can be summarized as feature production . In feature production , the input image is first encoded into a series of feature maps with various resolutions . Then the objects are detected based on the features according to their scales . One method is to produce the feature for each scale range by one neural network , like the featurized image pyramid discussed in Lin et al . ( 2017 ) . Variants include utilizing the image pyramid , like SNIP ( Singh & Davis , 2018 ) . However , we employ only one neural network and obtain multi-scale features from intermediate features of the network . Because deep neural networks are experts in encoding the image into features . And they are considered to be able to encode information of different scales into features with different resolutions . Then we face the problem of how to utilize these features since there are N features available for K object groups . Therefore , it is reasonable to split feature production into feature generation and feature utilization . To be more specific , the problem of multi-scale feature production can be defined as Equation 1 . When we divide the problem into feature generation and feature utilization , as in Equation 2 and Equation 3 , φ is approximated by g ◦ f . φ : R3×W×H → { RHi×Wi×Hi } K ( 1 ) f : R3×W×H → { RHi×Wi×Hi } N ( 2 ) g : { RHi×Wi×Hi } N → { RHi×Wi×Hi } K ( 3 ) And it is likely that only optimizing feature generation , as previous works do , is not optimal . So instead of optimizing feature generation for all the feature utilization strategies , we jointly optimize feature generation and utilization as a whole . 3.2 FEATURE GENERATION . Resolution of feature maps in one network changes with downsampling or upsampling operators . The network architectures of FPN follow the downsampling-upsampling style , as Figure 1 ( a ) shows . By encouraging a more flexible design of the scale-changing operation ’ s positions and numbers , we could obtain many more promising candidates to generate better multi-scale features . We implement that by searching the stride of each block in the network . Search space By deconstructing and generalizing the prevalent feature pyramid architecture , the basic search space is built as a stride-variable straight structure . A super-net of MSNAS with the depth of N consists of N mixed-blocks , MB1 , MB2 , ... , MBN . For each mixed-block , three possible strides are provided , i.e . 0.5 , 1 , and 2 . The block whose stride equals to 0.5 is implemented as an upsampling block with an interpolation operator followed by a convolution to double the width and height of the feature map . Blocks that don ’ t change the resolution of the feature map are referred to as normal blocks . The resolution of the feature output by one mixed-block could be twice , half or the same as the input feature , as illustrated in Figure 2 ( a ) . Considering operators with different strides within a mixed-block and the variation of sizes output from different operators in one mixed-block , the super-net is designed as a path-wise structure like Guo et al . ( 2019 ) . One path in the super-net is treated as valid if none of the blocks output feature larger than a quarter or smaller than 1/64 of the input image . Invalid paths are removed either in the training process of super-net or the sampling during the evolutionary search . Lateral connections Lateral connections are built according to current sub-architecture , as Figure 1 ( b ) shows . One additional 1×1 lateral convolution attached after every mix-block is available for latter cross-block connections . In scale-decreasing architectures , blocks can be grouped by resolution of output features . Each group is notated as one stage . Similarly , we refer stage to a group of adjacent blocks with the same output resolution , e.g . one downsampling or upsampling block and following normal blocks . The feature map of the last mix-block at one stage is merged with the lateral feature by element-wise addition as Equation 4-Equation 6 shows . xi =MBi ( xi−1 ) + lati ( 4 ) lati = { LateralConvri ( xri ) , if ( stridei+1 6= 1 ) or ( i = N − 1 ) 0 , otherwise ( 5 ) where ri = max ( { k | ( ∏ k < j≤i stridej = 1 ) and ( stridek+1 6= 1 ) } ) ( 6 ) where lati means the lateral connection of block i to combine with . If block i is the last block of a stage or the end of the encoder network as described in Equation5 , lati is generated by lateral convolution LateralConvri . Among blocks with output feature of the same resolution as block i , block ri is the nearest one at a different stage . It can be regarded as an extensive version of lateral connections in the original FPN structure .

Summary: This submission works on the task of architecture search for object detection. The authors focus on two components: how to produce multi-scale features and how to use multi-scale features. The authors formalized a simple search space, and applied an evolution-based search algorithm. Experiments show the proposed searching algorithm is able to outperform the FPN baselines with various (small) backbones.