Multi-scale Network Architecture Search for Object Detection

1 INTRODUCTION . Recognizing and localizing objects at vastly different scales is a fundamental challenge in object detection . Detection performance for objects with different scales is highly related to features with different properties such as feature resolution , receptive fields , and feature fusion ways . The key to solving the multi-scale problem in object detection is how to build a multi-scale network that has proper high-level semantic features for objects with different scales . A recent work in object detection Feature Pyramid Networks ( FPN ) ( Lin et al. , 2017 ) has achieved remarkable success in multi-scale feature design and has been commonly used by many modern object detectors ( He et al. , 2017 ; Lin et al. , 2020 ; Lu et al. , 2019 ) . FPN extracts multi-scale intermediate features from the encoder network and assigns objects grouped by scales to corresponding features according to a heuristic rule . Another prevalent detection framework , SSD ( Liu et al. , 2016 ) , conducts feature generation by a lighter encoder network without upsampling operators . The basic idea to deal with the multi-scale detection problem can be summarized as below . Given the input image , a series of feature maps with the various resolution are generated to detect objects grouped by scale range . We note it as multi-scale feature production . In FPN and its variants , the multi-scale feature production is split into two steps , feature generation and feature utilization . In terms of feature generation , an encoder network composed of blocks provides features with different scales . And the strategy of feature utilization determines the rule of assigning objects to feature maps . These two steps are closely related to each other . Although FPN has achieved promising results on multi-scale object detection tasks , the production of multi-scale features is quite hand-crafted and relies heavily on the experiences of human experts . More specifically , network architectures of FPN are based on a downsample-upsample architecture which may not be effective enough . By changing the downsampling and upsampling operation ’ s positions and numbers , we could obtain many other candidates to generate different multi-scale features . Also , the predefined rule of feature utilization is very empirical and other alternatives may lead to better performance . Therefore we wonder : Can we find network architectures that can build better semantic feature representation for multiple scales ? The answer is yes . Recent advances in neural architecture search have shown promising results compared with handcrafted architecture by human experts ( Zoph et al. , 2018 ; Liu et al. , 2019b ; Cai et al. , 2019 ; Guo et al. , 2019 ) . Several works have also focused on neural architecture search in object detection tasks ( Chen et al. , 2019 ; Ghiasi et al. , 2019 ; Du et al. , 2019 ) , but generating and utilizing multi-scale features are still not well explored . DetNAS ( Chen et al. , 2019 ) adopts the method mainly designed on image classification to search the operations of backbone networks in object detectors . NASFPN ( Ghiasi et al. , 2019 ) focuses on searching for better feature-fusion connections in the neck part of FPN . NAS-FPN doesn ’ t optimize the whole encoder network and still relies on predefined backbone architecture . Recently SpineNet ( Du et al. , 2019 ) proposes a search method with scalepermuted features and cross-scale connections by reinforcement learning , but the search cost is quite large . All these previous works focus on designing better neural network architectures to generate better features given a fixed feature selection strategy . However , they fail to conduct a complete flexible multi-scale feature production strategy . In this paper , we propose a new method to take into account of both aspects and build detection networks with the strong and proper multi-scale feature production strategy by neural architecture search . For feature generation , we put forward a network stride search method to generate multiple feature representations for different scales . Different from the scale-decreasing-increasing architecture of FPN , the scale of our networks can decrease or increase at each block , as illustrated in Figure 1 . By stride search for each block , we could significantly explore a wide range of possible feature generation designs of multi-resolution networks . Most backbones of object detectors are originally designed on image classification without multi-scale problems . However , stride configuration in the encoder network would be optimized in the context of the multi-scale task . Moreover , more complex cross-scale feature fusions might appear according to more complex internal scale changes . For feature utilization , we change the previous one-to-one mapping strategy into a more flexible feature selection . Since each group with objects of the same scale range owns one detection head , feature utilization is implemented by selecting proper features for detection heads . Objects of different scale ranges might be assigned to the same feature map . It is not possible in previous methods , as shown in Figure 1 ( b ) . By jointly optimizing feature generation and utilization of multi-scale features , we search for flexible but complete multi-scale feature production strategies . Extensive experiments demonstrate complete multi-scale feature production search is critical to building strong and proper semantic features for object detection with different scales . On challenging COCO dataset ( Lin et al. , 2014 ) , our method obtains a 2.6 % , 1.5 % , 1.2 % mAP improvement with similar FLOPs as ResNet18-FPN , ResNet34FPN , ResNet50-FPN . 2 RELATED WORK . Neural Architecture Search Neural Architecture Search aims to design better network architectures automatically . RL-based methods ( Zoph et al. , 2018 ; Zoph & Le , 2017 ) have achieved great success despite a huge computation cost . In differentiable algorithms ( Liu et al. , 2019b ; Cai et al. , 2019 ) , architecture parameters are employed and operators in the search space are considered as the weighted sum of candidate operators . There exist difficulties to deal with operators with different strides . Super-nets , acting as the collection of weights shared by all the sub-architectures , and evolutionary search are involved in one-shot NAS ( Guo et al. , 2019 ; Bender et al. , 2018 ) . But it ’ s difficult to ensure strong correlations between one-shot and stand-alone performances of the sub-architectures . Multi-scale Object Detection SSD ( Liu et al. , 2016 ) uses multi-scale features generated by different stages of the backbone network to detect objects of different scales . Feature pyramid architectures are utilized in FPN ( Lin et al. , 2017 ) and RetinaNet ( Lin et al. , 2020 ) to obtain multi-scale features . SNIP ( Singh & Davis , 2018 ) includes the image pyramid architecture to deal with multi-scale detection . Frameworks with multi-scale features are prevalent as objects of different scales appear in one image . Neural Architecture Search on Object Detection DetNAS ( Chen et al. , 2019 ) , NAS-FCOS ( Wang et al. , 2019 ) and Auto-FPN ( Xu et al. , 2019 ) focus on the architecture of the top-down pathway and feature fusion . SM-NAS ( Yao et al. , 2019 ) and CR-NAS ( Liang et al. , 2020 ) try to adjust the computation occupied by different parts of detectors . Also , there are several works aiming to improve FPN using NAS . NAS-FPN ( Ghiasi et al. , 2019 ) improves detection performance by searching for better connections within the feature pyramid network . It is limited without modification to the overall encoder architecture . And it fails to take the multi-scale feature utilization into account . EfficientDet ( Tan et al. , 2019 ) and Auto-FPN ( Xu et al. , 2019 ) search better feature fusion for FPN with differentiable methods . Other relative works like Liu et al . ( 2019a ) conduct similar modifications . Recently , SpineNet ( Du et al. , 2019 ) proposes a backbone search method with scale-permuted features and cross-scale connections by reinforcement learning . Our work has several major differences from it . First , the search space of MSNAS is designed that each operator in the network can downsample or upsample instead of permutation , which builds a much larger search space than SpineNet . Second , the complete multi-scale feature production is considered in our work , while SpineNet only focuses on the architecture of the encoder network . Lastly , our method is based on the one-shot search method instead of reinforcement learning in SpineNet . Our method is much more efficient and requires much less computation cost than SpineNet . 3 METHOD . We start by discussing multi-scale feature production for the object detection network in Section 3.1 . In Section 3.2 we will introduce how to build the search space and search proper stride in detection networks to obtain better multi-scale features . In Section 3.3 , how to search the appropriate strategy of feature utilization is presented . Finally , details about super-net training and search strategy are described in Section 3.4 . 3.1 FEATURE PRODUCTION FOR DETECTION NETWORK . In this section , we ’ ll discuss the production of multi-scale features and define the problem in detail . As noted above , the basic idea of handling the multi-scale detection problem can be summarized as feature production . In feature production , the input image is first encoded into a series of feature maps with various resolutions . Then the objects are detected based on the features according to their scales . One method is to produce the feature for each scale range by one neural network , like the featurized image pyramid discussed in Lin et al . ( 2017 ) . Variants include utilizing the image pyramid , like SNIP ( Singh & Davis , 2018 ) . However , we employ only one neural network and obtain multi-scale features from intermediate features of the network . Because deep neural networks are experts in encoding the image into features . And they are considered to be able to encode information of different scales into features with different resolutions . Then we face the problem of how to utilize these features since there are N features available for K object groups . Therefore , it is reasonable to split feature production into feature generation and feature utilization . To be more specific , the problem of multi-scale feature production can be defined as Equation 1 . When we divide the problem into feature generation and feature utilization , as in Equation 2 and Equation 3 , φ is approximated by g ◦ f . φ : R3×W×H → { RHi×Wi×Hi } K ( 1 ) f : R3×W×H → { RHi×Wi×Hi } N ( 2 ) g : { RHi×Wi×Hi } N → { RHi×Wi×Hi } K ( 3 ) And it is likely that only optimizing feature generation , as previous works do , is not optimal . So instead of optimizing feature generation for all the feature utilization strategies , we jointly optimize feature generation and utilization as a whole . 3.2 FEATURE GENERATION . Resolution of feature maps in one network changes with downsampling or upsampling operators . The network architectures of FPN follow the downsampling-upsampling style , as Figure 1 ( a ) shows . By encouraging a more flexible design of the scale-changing operation ’ s positions and numbers , we could obtain many more promising candidates to generate better multi-scale features . We implement that by searching the stride of each block in the network . Search space By deconstructing and generalizing the prevalent feature pyramid architecture , the basic search space is built as a stride-variable straight structure . A super-net of MSNAS with the depth of N consists of N mixed-blocks , MB1 , MB2 , ... , MBN . For each mixed-block , three possible strides are provided , i.e . 0.5 , 1 , and 2 . The block whose stride equals to 0.5 is implemented as an upsampling block with an interpolation operator followed by a convolution to double the width and height of the feature map . Blocks that don ’ t change the resolution of the feature map are referred to as normal blocks . The resolution of the feature output by one mixed-block could be twice , half or the same as the input feature , as illustrated in Figure 2 ( a ) . Considering operators with different strides within a mixed-block and the variation of sizes output from different operators in one mixed-block , the super-net is designed as a path-wise structure like Guo et al . ( 2019 ) . One path in the super-net is treated as valid if none of the blocks output feature larger than a quarter or smaller than 1/64 of the input image . Invalid paths are removed either in the training process of super-net or the sampling during the evolutionary search . Lateral connections Lateral connections are built according to current sub-architecture , as Figure 1 ( b ) shows . One additional 1×1 lateral convolution attached after every mix-block is available for latter cross-block connections . In scale-decreasing architectures , blocks can be grouped by resolution of output features . Each group is notated as one stage . Similarly , we refer stage to a group of adjacent blocks with the same output resolution , e.g . one downsampling or upsampling block and following normal blocks . The feature map of the last mix-block at one stage is merged with the lateral feature by element-wise addition as Equation 4-Equation 6 shows . xi =MBi ( xi−1 ) + lati ( 4 ) lati = { LateralConvri ( xri ) , if ( stridei+1 6= 1 ) or ( i = N − 1 ) 0 , otherwise ( 5 ) where ri = max ( { k | ( ∏ k < j≤i stridej = 1 ) and ( stridek+1 6= 1 ) } ) ( 6 ) where lati means the lateral connection of block i to combine with . If block i is the last block of a stage or the end of the encoder network as described in Equation5 , lati is generated by lateral convolution LateralConvri . Among blocks with output feature of the same resolution as block i , block ri is the nearest one at a different stage . It can be regarded as an extensive version of lateral connections in the original FPN structure .

Summary: This paper introduces a network architecture search (NAS) suitable for a feature pyramid network (FPN) that provides notable detection accuracy for objects at every scale. Based on the decomposition of FPN structure as (multi-scale) feature generation and feature utilization, the proposed NAS offers a new design strategies for both components. For feature generation, NAS super-net is trained to find the optimal selection of whether to reduce, maintain, or extend feature resolution after each module. For feature utilitization, it defines conditions for efficiently selecting the optimal FPN architecture. The proposed MSNAS yields the better accuracy than its backbone FPN and other NAS methods on the COCO object detection benchmark dataset.