ZenDet: Revisiting Efficient Object Detection Backbones from Zero-Shot Neural Architecture Search

In object detection models , the detection backbone consumes more than half of the overall inference cost . Recent researches attempt to reduce this cost by optimizing the backbone architecture with the help of Neural Architecture Search ( NAS ) . However , existing NAS methods for object detection require hundreds to thousands of GPU hours of searching , making them impractical in fast-paced research and development . In this work , we propose a novel zero-shot NAS method to address this issue . The proposed method , named ZenDet , automatically designs efficient detection backbones without training network parameters , reducing the architecture design cost to nearly zero yet delivering the state-of-the-art ( SOTA ) performance . Under the hood , ZenDet maximizes the differential entropy of detection backbones , leading to a better feature extractor for object detection under the same computational budgets . After merely one GPU day of fully automatic design , ZenDet innovates SOTA detection backbones on multiple detection benchmark datasets with little human intervention . Comparing to ResNet50 backbone , ZenDet is +2.0 % better in mAP when using the same amount of FLOPs/parameters and is 1.54 times faster on NVIDIA V100 at the same mAP . Code and pre-trained models will be released after publication . 1 INTRODUCTION . Seeking better and faster deep models for object detection is never an outdated task in computer vision . The performance of a deep object detection network heavily depends on the feature extraction backbone ( Li et al. , 2018 ; Chen et al. , 2019b ) . Currently , most state-of-the-art ( SOTA ) detection backbones ( He et al. , 2016 ; Xie et al. , 2017 ; Zhu et al. , 2019 ) are designed manually by human experts which could take years to develop . Since the detection backbone consumes more than half of the total inference cost in many detection frameworks , it is critical to optimize the backbone architecture for better speed-accuracy trade-off on different hardware platforms varying from server-side GPUs to mobile-side chipsets . To reduce the manual design , Neural Architecture Search ( NAS ) has emerged to facilitate the architecture design . Various NAS methods have demonstrated their efficacy in designing SOTA image classification models ( Zoph et al. , 2018 ; Liu et al. , 2018 ; Cai et al. , 2019 ; Tan & Le , 2019 ) . These successful stories inspire recent researchers to use NAS to design detection backbones ( Chen et al. , 2019b ; Du et al. , 2020 ; Jiang et al. , 2020 ) in an end-to-end way . The existing NAS methods for detection backbone design are all training-based , meaning that they need to train network parameters to evaluate the performance of candidates on the target dataset , taking a long time and consuming huge hardware resources . Hardware consumption and long time searching make these training-based methods inefficient in modern fast-paced research and development . To reduce the training cost , training-free methods are proposed recently , also known as zero-shot NAS in previous literatures ( Tanaka et al. , 2020 ; Mellor et al. , 2021 ; Chen et al. , 2021b ; Lin et al. , 2021 ) . The zero-shot NAS predicts network performance without training network parameters therefore is way faster than training-based NAS . As a relatively new technique , existing zero-shot NAS methods are mostly validated on image classification datasets . Applying zero-shot NAS to object detection backbone design is still an intact challenge . In this work , we present the first effort of introducing zero-shot NAS technique to design efficient object detection backbones . We show that directly transferring existing zero-shot NAS methods from image classification to detection backbone design will encounter fundamental difficulties . While image classification network only needs to predict the class probability , object detection network needs to additionally predict the bounding boxes of multiple objects , making the direct architecture transfer sub-optimal . To this end , a novel zero-shot NAS method , termed ZenDet , is proposed for searching object detection backbones . The key idea behind ZenDet is inspired by the Principle of Maximum Entropy ( PME ) ( Reza , 1994 ; Kullback , 1997 ; Brillouin , 2013 ) . Informally speaking , when a network is formulated as an information processing system , its capacity is maximized when its differential entropy ( Shannon , 1948 ) achieves maximum under budget constraints , leading to a better feature extractor for object detection . Based on this observation , ZenDet maximizes the differential entropy of detection backbones by searching for the optimal configuration of network depth and width without training network parameters . The above strategy raises two technical challenges . The first challenge is how to estimate the differential entropy of a deep network . The exact computation of differential entropy requires knowing the precise probability distribution of deep features in high dimensional space which is difficult to estimate in practice . To address this issue , ZenDet estimates the Gaussian upper bound of the differential entropy which only requires computing the variance of the feature maps . The second challenge is how to efficiently capture objects of different sizes . In object detection benchmark datasets such as MS COCO Lin et al . ( 2014 ) , the distribution of object size is data-dependent and non-uniform . To bring in this prior knowledge in backbone design , we introduce the Multi-Scale Entropy Prior ( MSEP ) in backbone entropy estimation to capture different-scale objects . We find that the MSEP improves the detection performance significantly . The overall computation of ZenDet only requires one forward inference of the detection backbone at initialization therefore is nearly zero-cost comparing to previous backbone NAS methods . The main contributions of this work are summarized as follows : • Based on the entropy theory , the multi-scale entropy prior is present to rank the expressivity of the backbone instead of training on the target datasets , speeding up searching . • While using less than one GPU day and 2GB GPU memory , ZenDet achieves competitive performance over other NAS methods on COCO with at least 50x times faster . • ZenDet is the first zero-shot NAS method designed for object detection with SOTA performance in multiple benchmark datasets under multiple popular detection frameworks . 2 RELATED WORK . Backbone for Object Detection Recently , object detectors composed of backbone , neck and head have become increasingly popular due to their effectiveness and high performance ( Lin et al. , 2017a ; b ; Tian et al. , 2019 ; Li et al. , 2020 ; 2021 ) . Prevailing detectors directly use the backbone designed for image classification to extract multi-scale features from an image , such as ResNet ( He et al. , 2016 ) , ResNeXt ( Xie et al. , 2017 ) and Deformable Convolutional Network ( DCN ) ( Zhu et al. , 2019 ) . Nevertheless , the backbone migrated from image classification may be suboptimal in object detection ( Ghiasi et al. , 2019 ) . To tackle the gap , many architectures are end-to-end designed for object detection , including Stacked Hourglass Newell et al . ( 2016 ) , FishNet Sun et al . ( 2018 ) , DetNet Li et al . ( 2018 ) , HRNet Wang et al . ( 2020a ) and so on . Albeit with improved performance , these hand-crafted detection architectures heavily rely on human labor and tedious trial-and-error processes . Neural Architecture Search Neural Architecture Search ( NAS ) is initially developed to automatically design network architectures for image classification models ( Zoph et al. , 2018 ; Liu et al. , 2018 ; Real et al. , 2019 ; Cai et al. , 2019 ; Lin et al. , 2020 ; Tan & Le , 2019 ; Lin et al. , 2021 ) . Using NAS to design object detection models has not been well studied . Currently , existing detection NAS methods are all training-based methods . Some methods focus on searching detection backbones , such as DetNAS ( Chen et al. , 2019b ) , SpineNet ( Du et al. , 2020 ) and SP-NAS ( Jiang et al. , 2020 ) , while the others focus on searching FPN neck , such as NAS-FPN ( Ghiasi et al. , 2019 ) , NASFCOS ( Wang et al. , 2020b ) and OPANet ( Liang et al. , 2021 ) . These methods require training and evaluation on the target datasets which is intensive in computation . ZenDet distinguishes itself as being the first zero-shot NAS method for the backbone design of object detection . 3 PRELIMINARY . In this section , we first formulate a deep network as a system with continuous state space . Then we define the differential entropy of this system and show how to estimate this entropy via its Gaussian upper bound . Finally we introduce the basic concept of vanilla network search space for designing our detection backbones . Continuous State Space of Deep Networks A deep network F ( · ) : Rd −→ R maps an input image x ∈ Rd to its label y ∈ R. The topology of a network can be abstracted as a graph G = ( V , E ) where the vertex set V consists of neurons and the edge set E consists of spikes between neurons . For any v ∈ V and e ∈ E , h ( v ) ∈ R and h ( e ) ∈ R present the values endowed with each vertex v and each edge e respectively . The set S = { h ( v ) , h ( e ) : ∀v ∈ V , e ∈ E } defines the continuous state space of the network F . According to the Principle of Maximum Entropy , we want to maximize the differential entropy of network F , under some given computational budgets . The entropy H ( S ) of set S measures the total information contained in the system ( network ) F , including the information contained in the latent features H ( Sv ) = { h ( v ) : v ∈ V } and in the network parameters H ( Se ) = { h ( e ) : e ∈ E } . As for object detection backbone design , we only care about the entropy of latent features H ( Sv ) rather than the entropy of network parameters H ( Se ) . Informally speaking , H ( Sv ) measures the feature representation power of F while H ( Se ) measures the model complexity of F . Therefore , in the remainder of this work , the differential entropy of F refers to the entropy H ( Sv ) by default . Entropy of Gaussian Distribution The differential entropy of Gaussian distribution can be found in many textbooks such as ( Norwich , 1993 ) . Suppose x is sampled from Gaussian distribution N ( µ , σ2 ) . Then the differential entropy of x is given by H∗ ( x ) = 1 2 log ( 2π ) + 1 2 +H ( x ) H ( x ) : = log ( σ ) . ( 1 ) From Eq . 1 , the entropy of Gaussian distribution only depends on the variance . In the following , we will use H ( x ) instead of H∗ ( x ) as constants do not matter in our discussion . Gaussian Entropy Upper Bound Since the probability distribution P ( Sv ) is a high dimensional function , it is difficult to compute the precise value of its entropy directly . Instead , we propose to estimate the upper bound of the entropy , given by the following well-known theorem ( Cover & Thomas , 2012 ) : Theorem 1 . For any continuous distribution P ( x ) of mean µ and variance σ2 , its differential entropy is maximized when P ( x ) is a Gaussian distribution N ( µ , σ2 ) . Theorem 1 says that the differential entropy of a distribution is upper bounded by a Gaussian distribution with the same mean and variance . Combining this with Eq . ( 1 ) , we can easily estimate the network entropy H ( Sv ) by simply computing the feature map variance and then use Eq . ( 1 ) to get the Gaussian entropy upper bound for the network . Vanilla Network Search Space Following previous works , we design our backbones in the vanilla convolutional network space ( Li et al. , 2018 ; Chen et al. , 2019b ; Du et al. , 2020 ; Lin et al. , 2021 ) because this space is widely adopted in detection backbones and is used as a prototype in theoretical literature ( Poole et al. , 2016 ; Serra et al. , 2018 ; Hanin & Rolnick , 2019 ) . A vanilla network is stacked by multiple convolutional layers followed by RELU activations . Auxiliary components such as residual link ( He et al. , 2016 ) , Batch Normalization ( BN ) ( Ioffe & Szegedy , 2015 ) and Squeeze-and-Excitation ( SE ) ( Hu et al. , 2018 ) are all removed during the search and only during the search ( More details in Appendix E ) . These removed auxiliary components are plugged in back after the search . Therefore , the final architecture for training still has these components . Consider a vanilla convolutional network with D layers of weights W1 , ... , WD whose output feature maps are x1 , ... , xD . The input image is x0 . Let φ ( · ) denote the RELU activation function . Then the forward inference is given by xl = φ ( hl ) hl = Wl ∗ xl−1 for l = 1 , ... , D . ( 2 ) For sake of simplicity , we set the bias of the convolutional layer to zero .

Summary: This paper proposes a zero-shot neural architecture search approach for backbone design in object detection. The idea is to compute an entropy-based multi-scale Zen-score and use the score as an objective for evolutionary architecture search. The paper achieves better results comparing with previous zero-shot NAS approaches on object detection and greatly reduces the search time of conventional NAS approaches while maintaining similar accuracy.