ZENDET: REVISITING EFFICIENT OBJECT DETECTION BACKBONES
Year: 2021

Abstract:
In object detection models, the detection backbone consumes more than half of the overall inference cost. Recent researches attempt to reduce this cost by optimizing the backbone architecture with the help of Neural Architecture Search (NAS). However, existing NAS methods for object detection require hundreds to thousands of GPU hours of searching, making them impractical in fast-paced research and development. In this work, we propose a novel zero-shot NAS method to address this issue. The proposed method, named ZenDet, automatically designs efficient detection backbones without training network parameters, reducing the architecture design cost to nearly zero yet delivering the state-of-the-art (SOTA) performance. Under the hood, ZenDet maximizes the differential entropy of detection backbones, leading to a better feature extractor for object detection under the same computational budgets. After merely one GPU day of fully automatic design, ZenDet innovates SOTA detection backbones on multiple detection benchmark datasets with little human intervention. Comparing to ResNet50 backbone, ZenDet is +2.0% better in mAP when using the same amount of FLOPs/parameters and is 1.54 times faster on NVIDIA V100 at the same mAP. Code and pre-trained models will be released after publication.

text: In object detection models, the detection backbone consumes more than half of the overall inference cost. Recent researches attempt to reduce this cost by optimizing the backbone architecture with the help of Neural Architecture Search (NAS). However, existing NAS methods for object detection require hundreds to thousands of GPU hours of searching, making them impractical in fast-paced research and development. In this work, we propose a novel zero-shot NAS method to address this issue. The proposed method, named ZenDet, automatically designs efficient detection backbones without training network parameters, reducing the architecture design cost to nearly zero yet delivering the state-of-the-art (SOTA) performance. Under the hood, ZenDet maximizes the differential entropy of detection backbones, leading to a better feature extractor for object detection under the same computational budgets. After merely one GPU day of fully automatic design, ZenDet innovates SOTA detection backbones on multiple detection benchmark datasets with little human intervention. Comparing to ResNet50 backbone, ZenDet is +2.0% better in mAP when using the same amount of FLOPs/parameters and is 1.54 times faster on NVIDIA V100 at the same mAP. Code and pre-trained models will be released after publication.

heading: 1 INTRODUCTION
text: Seeking better and faster deep models for object detection is never an outdated task in computer vision. The performance of a deep object detection network heavily depends on the feature extraction backbone (Li et al., 2018; Chen et al., 2019b). Currently, most state-of-the-art (SOTA) detection backbones (He et al., 2016; Xie et al., 2017; Zhu et al., 2019) are designed manually by human experts which could take years to develop. Since the detection backbone consumes more than half of the total inference cost in many detection frameworks, it is critical to optimize the backbone architecture for better speed-accuracy trade-off on different hardware platforms varying from server-side GPUs to mobile-side chipsets. To reduce the manual design, Neural Architecture Search (NAS) has emerged to facilitate the architecture design. Various NAS methods have demonstrated their efficacy in designing SOTA image classification models (Zoph et al., 2018; Liu et al., 2018; Cai et al., 2019; Tan & Le, 2019). These successful stories inspire recent researchers to use NAS to design detection backbones (Chen et al., 2019b; Du et al., 2020; Jiang et al., 2020) in an end-to-end way.
The existing NAS methods for detection backbone design are all training-based, meaning that they need to train network parameters to evaluate the performance of candidates on the target dataset, taking a long time and consuming huge hardware resources. Hardware consumption and long time searching make these training-based methods inefficient in modern fast-paced research and development. To reduce the training cost, training-free methods are proposed recently, also known as zero-shot NAS in previous literatures (Tanaka et al., 2020; Mellor et al., 2021; Chen et al., 2021b; Lin et al., 2021). The zero-shot NAS predicts network performance without training network parameters therefore is way faster than training-based NAS. As a relatively new technique, existing zero-shot NAS methods are mostly validated on image classification datasets. Applying zero-shot NAS to object detection backbone design is still an intact challenge.
In this work, we present the first effort of introducing zero-shot NAS technique to design efficient object detection backbones. We show that directly transferring existing zero-shot NAS methods from image classification to detection backbone design will encounter fundamental difficulties. While image classification network only needs to predict the class probability, object detection network needs to additionally predict the bounding boxes of multiple objects, making the direct architecture transfer sub-optimal. To this end, a novel zero-shot NAS method, termed ZenDet, is proposed for searching object detection backbones. The key idea behind ZenDet is inspired by the Principle of Maximum Entropy (PME) (Reza, 1994; Kullback, 1997; Brillouin, 2013). Informally speaking, when a network is formulated as an information processing system, its capacity is maximized when its differential entropy (Shannon, 1948) achieves maximum under budget constraints, leading to a better feature extractor for object detection. Based on this observation, ZenDet maximizes the differential entropy of detection backbones by searching for the optimal configuration of network depth and width without training network parameters.
The above strategy raises two technical challenges. The first challenge is how to estimate the differential entropy of a deep network. The exact computation of differential entropy requires knowing the precise probability distribution of deep features in high dimensional space which is difficult to estimate in practice. To address this issue, ZenDet estimates the Gaussian upper bound of the differential entropy which only requires computing the variance of the feature maps. The second challenge is how to efficiently capture objects of different sizes. In object detection benchmark datasets such as MS COCO Lin et al. (2014), the distribution of object size is data-dependent and non-uniform. To bring in this prior knowledge in backbone design, we introduce the Multi-Scale Entropy Prior (MSEP) in backbone entropy estimation to capture different-scale objects. We find that the MSEP improves the detection performance significantly. The overall computation of ZenDet only requires one forward inference of the detection backbone at initialization therefore is nearly zero-cost comparing to previous backbone NAS methods.
The main contributions of this work are summarized as follows:
• Based on the entropy theory, the multi-scale entropy prior is present to rank the expressivity of the backbone instead of training on the target datasets, speeding up searching. • While using less than one GPU day and 2GB GPU memory, ZenDet achieves competitive performance over other NAS methods on COCO with at least 50x times faster. • ZenDet is the first zero-shot NAS method designed for object detection with SOTA performance in multiple benchmark datasets under multiple popular detection frameworks.

heading: 2 RELATED WORK
text: Backbone for Object Detection Recently, object detectors composed of backbone, neck and head have become increasingly popular due to their effectiveness and high performance (Lin et al., 2017a;b; Tian et al., 2019; Li et al., 2020; 2021). Prevailing detectors directly use the backbone designed for image classification to extract multi-scale features from an image, such as ResNet (He et al., 2016), ResNeXt (Xie et al., 2017) and Deformable Convolutional Network (DCN) (Zhu et al., 2019). Nevertheless, the backbone migrated from image classification may be suboptimal in object detection (Ghiasi et al., 2019). To tackle the gap, many architectures are end-to-end designed for object detection, including Stacked Hourglass Newell et al. (2016), FishNet Sun et al. (2018), DetNet Li et al. (2018), HRNet Wang et al. (2020a) and so on. Albeit with improved performance, these hand-crafted detection architectures heavily rely on human labor and tedious trial-and-error processes.
Neural Architecture Search Neural Architecture Search (NAS) is initially developed to automatically design network architectures for image classification models (Zoph et al., 2018; Liu et al., 2018; Real et al., 2019; Cai et al., 2019; Lin et al., 2020; Tan & Le, 2019; Lin et al., 2021). Using NAS to design object detection models has not been well studied. Currently, existing detection NAS methods are all training-based methods. Some methods focus on searching detection backbones, such as DetNAS (Chen et al., 2019b), SpineNet (Du et al., 2020) and SP-NAS (Jiang et al., 2020), while the others focus on searching FPN neck, such as NAS-FPN (Ghiasi et al., 2019), NASFCOS (Wang et al., 2020b) and OPANet (Liang et al., 2021). These methods require training and evaluation on the target datasets which is intensive in computation. ZenDet distinguishes itself as being the first zero-shot NAS method for the backbone design of object detection.

heading: 3 PRELIMINARY
text: In this section, we first formulate a deep network as a system with continuous state space. Then we define the differential entropy of this system and show how to estimate this entropy via its Gaussian upper bound. Finally we introduce the basic concept of vanilla network search space for designing our detection backbones.
Continuous State Space of Deep Networks A deep network F (·) : Rd −→ R maps an input image x ∈ Rd to its label y ∈ R. The topology of a network can be abstracted as a graph G = (V, E) where the vertex set V consists of neurons and the edge set E consists of spikes between neurons. For any v ∈ V and e ∈ E , h(v) ∈ R and h(e) ∈ R present the values endowed with each vertex v and each edge e respectively. The set S = {h(v), h(e) : ∀v ∈ V, e ∈ E} defines the continuous state space of the network F .
According to the Principle of Maximum Entropy, we want to maximize the differential entropy of network F , under some given computational budgets. The entropy H(S) of set S measures the total information contained in the system (network) F , including the information contained in the latent features H(Sv) = {h(v) : v ∈ V} and in the network parameters H(Se) = {h(e) : e ∈ E}. As for object detection backbone design, we only care about the entropy of latent features H(Sv) rather than the entropy of network parameters H(Se). Informally speaking, H(Sv) measures the feature representation power of F while H(Se) measures the model complexity of F . Therefore, in the remainder of this work, the differential entropy of F refers to the entropy H(Sv) by default. Entropy of Gaussian Distribution The differential entropy of Gaussian distribution can be found in many textbooks such as (Norwich, 1993). Suppose x is sampled from Gaussian distribution N (µ, σ2). Then the differential entropy of x is given by
H∗(x) = 1
2 log(2π) +
1 2 +H(x) H(x) := log(σ) . (1)
From Eq. 1, the entropy of Gaussian distribution only depends on the variance. In the following, we will use H(x) instead of H∗(x) as constants do not matter in our discussion.
Gaussian Entropy Upper Bound Since the probability distribution P(Sv) is a high dimensional function, it is difficult to compute the precise value of its entropy directly. Instead, we propose to estimate the upper bound of the entropy, given by the following well-known theorem (Cover & Thomas, 2012):
Theorem 1. For any continuous distribution P(x) of mean µ and variance σ2, its differential entropy is maximized when P(x) is a Gaussian distribution N (µ, σ2).
Theorem 1 says that the differential entropy of a distribution is upper bounded by a Gaussian distribution with the same mean and variance. Combining this with Eq. (1), we can easily estimate the network entropy H(Sv) by simply computing the feature map variance and then use Eq. (1) to get the Gaussian entropy upper bound for the network.
Vanilla Network Search Space Following previous works, we design our backbones in the vanilla convolutional network space (Li et al., 2018; Chen et al., 2019b; Du et al., 2020; Lin et al., 2021) because this space is widely adopted in detection backbones and is used as a prototype in theoretical literature (Poole et al., 2016; Serra et al., 2018; Hanin & Rolnick, 2019). A vanilla network is stacked by multiple convolutional layers followed by RELU activations. Auxiliary components such as residual link (He et al., 2016), Batch Normalization (BN) (Ioffe & Szegedy, 2015) and Squeeze-and-Excitation (SE) (Hu et al., 2018) are all removed during the search and only during the search (More details in Appendix E). These removed auxiliary components are plugged in back after the search. Therefore, the final architecture for training still has these components.
Consider a vanilla convolutional network with D layers of weights W1, ..., WD whose output feature maps are x1, ..., xD. The input image is x0. Let φ(·) denote the RELU activation function. Then the forward inference is given by
xl = φ(hl) hl = Wl ∗ xl−1 for l = 1, ..., D . (2)
For sake of simplicity, we set the bias of the convolutional layer to zero.

heading: 4 ZENDET-NAS FOR OBJECT DETECTION
text: In this section, we first describe how to compute the differential entropy of the single-scale feature for very deep vanilla networks with the re-scaling trick. Then we introduce the Multi-Scale Entropy Prior (MSEP) to better capture the prior distribution of object size in ZenDet. Finally, we present the complete ZenDet-NAS under the Evolutionary Algorithm (EA) framework.

heading: 4.1 SINGLE-SCALE ENTROPY FOR VERY DEEP BACKBONES
text: To compute the entropy of the detection backbone, we initialize all backbone parameters by the standard Gaussian distribution N (0, 1). Then we randomly generate an image filled with the standard Gaussian noise and perform forward inference. Based on the discussion in Section 3, the (Gaussian upper bound) entropy H(F ) of the network F is then given by
H(F ) = 1
2 log(Var(hD)) . (3)
Please note that the variance is computed on the last pre-activation feature map hD.
For very deep networks, directly using Eq. (3) might cause a numerical overflow. The same problem is also observed in Zen-NAS (Lin et al., 2021). Inspired by the analysis in Zen-NAS where the authors show that the gradient norm is scale-invariant to BN layers after variance compensation, we propose a more straightforward solution. We directly re-scale each feature mapxl by some constants γl during inference, that is xl = φ(hl)/γl, and then compensate the entropy of the network by
H(F ) = 1
2 log(Var(ĥD)) + D∑ l=1 log(γl) . (4)
The values of γl can be arbitrarily given as long as the forward inference does not overflow or underflow. In practice, we find that simply setting γl to the Euclidean norm of the feature map works well. The overall process is illustrated in Figure 1 (a).
Comparison with Zen-NAS Although the definition of vanilla network and solution to overflow are similar, the principles of single-scale entropy and Zen-NAS are fundamentally different. ZenNAS uses the gradient norm of the input image as ranking score and proposes to use two feedforward inferences to approximate the gradient norm for classification. In contrast, ZenDet uses entropy-based score which only requires one feed-forward inference, doubling the search speed.

heading: 4.2 MULTI-SCALE ENTROPY PRIOR FOR OBJECT DETECTION
text: In real-world images, the distribution of object size is not uniform. To bring in this prior knowledge, the detection backbone has 5 stages where each stage downsamples the feature resolution to half. The MSEP collects feature maps from the final output of each stage and weighted-sum the corresponding feature map entropies as a new measurement. We name this new measurement as multi-scale entropy. The overall process is illustrated in Figure 1 (b). In this figure, the backbone
extracts multi-scale featuresC = (C1,C2, ...,C5) at different resolutions. Then the FPN neck fuses C as input features P = (P1,P2, ...,P7) for the detection head. The multi-scale entropy Z(F ) of backbone F is then defined by
Z(F ) := α1H(C1) + α2H(C2) + · · ·+ α5H(C5) (5)
where H(Ci) is the entropy of Ci for i = 1, 2, · · · , 5. The weights α = (α1, α2, · · · , α5) store the multi-scale entropy prior to balance the expressivity of different scale features.
Weights α selection strategy As a concrete example in Fig. 1(b), the parts of P3 and P4 are generated by up-sampling of P5, and P6 and P7 are directly generated by down-sampling of P5 (generated by C5). Meanwhile, based on the fact that C5 carries sufficient context for detecting objects on various scales (Chen et al., 2021a), C5 is important in the backbone search, so it is good to set a larger value for the weight α5. Then, different combinations of α and correlation analysis are explored in Appendix D, indicating thatα = (0, 0, 1, 1, 6) is good enough for the FPN structure.

heading: 4.3 ZENDET-NAS IN EVOLUTIONARY ALGORITHM FRAMEWORK
text: Algorithm 1 ZenDet-NAS with Coarse-to-Fine Evolution Require: Search space S , inference budget B, maximal depth L, total number of iterations T ,
evolutionary population size N , initial structure F0, fine-search flag Flag. Ensure: NAS-designed ZenDet backbone F ∗.
1: Initialize population P = {F0}, Flag=False. 2: for t = 1, 2, · · · , T do 3: if t equals to T/2 then 4: Keep top 10 networks of highest multi-scale entropy in P and remove the others. 5: Set Flag = True. 6: end if 7: Randomly select Ft ∈ P . 8: Mutate F̂t = MUTATE(Ft,S, F lag) 9: if F̂t exceeds inference budget or has more than L layers then
10: Do nothing. 11: else 12: Get multi-scale entropy Z(F̂t). 13: Append F̂t to P . 14: end if 15: Remove networks of the smallest multi-scale entropy if the size of P exceeds B. 16: end for 17: Return F ∗, the network of the highest multi-scale entropy in P .
Algorithm 2 MUTATE Require: Structure Ft, search space S , fine-
search flag Flag. Ensure: Randomly mutated structure F̂t.
1: Uniformly select a block h in Ft. 2: if Flag equals to True then 3: Uniformly alternate the kernel size, width within some range. 4: else 5: Uniformly alternate the block type,
kernel size, width and depth within some range.
6: end if 7: Return the mutated structure F̂t.
Combining all above, we present our ZenDet-NAS in Algorithm 1. The ZenDet-NAS maximizes
the multi-scale differential entropy of detection backbones using Evolutionary Algorithm (EA). To improve the evolution efficiency, a coarse-to-fine strategy is proposed to reduce the search space gradually. First, we randomly generate N seed architectures to fill the population P . As shown in Figure 2, a seed architecture Ft consists of a sequence of building blocks such as ResNet block (He et al., 2016) or MobileNet block (Sandler et al., 2018). Then we randomly select one block and replace it with its mutated version. We use coarse-mutation in the early stage of EA and switch to fine-mutation after T/2 EA iterations. In the coarse-mutation, the block type, kernel size, depth and width are mutated randomly; in the fine-mutation, only kernel size and width are mutated.
After the mutation, if the inference cost of the new structure F̂t does not exceed the budget (e.g., FLOPs, parameters and latency) and its depth is smaller than budget L, F̂t is appended into the population P . The maximal depth L prevents the algorithm from generating over-deep structures. During EA iterations, the population is maintained to a certain size by discarding the worst candidate of the smallest multi-scale entropy. At the end of ZenDet-NAS, the backbone with the highest multiscale entropy is returned.

heading: 5 EXPERIMENTS
text: In this section, we first describe detail settings for searching and training with ZenDet-NAS. Then in subsection 5.2, we apply ZenDet-NAS to design better ResNet-like backbones on COCO dataset (Lin et al., 2014). We align the inference budget with ResNet-50/101. The performance of ZenDet and ResNet are compared under multiple detection frameworks including RetinaNet (Lin et al., 2017b), FCOS (Tian et al., 2019), and GFLV2 (Li et al., 2021). For fairness, we use the same training setting in all experiments for all backbones. In subsection 5.3, we compare the search cost of ZenDet to SOTA NAS methods for object detection. Subsection 5.4 reports the ablation studies of different components in ZenDet-NAS. Finally, subsection 5.5 verifies the transferability of ZenDets on several detection datasets and segmentation tasks.

heading: 5.1 EXPERIMENT SETTINGS
text: Searching Details In ZenDet-NAS, the evolutionary population N is set to 256. The total EA iterations T = 96000. Following the previous designs (Chen et al., 2019b; Jiang et al., 2020; Du et al., 2020), ZenDet is optimized for FLOPs. The resolution for computing entropy is 384× 384. Dataset and Training Details We evaluate detection performance on COCO (Lin et al., 2014) using the official training/testing splits. The mAP is evaluated on val 2017 by default and GFLV2 is additionally evaluated on test-dev 2007 following common practice. All models are trained from scratch (He et al., 2019) for 6X (73 epochs) on COCO. Following the Spinenet (Du et al., 2020), we use multi-scale training and Synchronized Batch Normalization (SyncBN). For VOC dataset, train-val 2007 and train-val 2012 are used for training, and test 2007 for evaluation. For image classification, all models are trained on ImageNet-1k (Deng et al., 2009) with a batch size of 256 for 120 epochs. Other setting details can be found in Appendix A.

heading: 5.2 DESIGN BETTER RESNET-LIKE BACKBONES
text: We search efficient ZenDet backbones for object detection and align with ResNet-50/101 in Table 1. ZenDet-S uses 60% less FLOPs than ResNet-50; ZenDet-M is aligned with ResNet-50 with similar FLOPs and number of parameters as ResNet-50; ZenDet-L is aligned with ResNet-101. The feature dimension in the FPN and heads is set to 256 for ZenDet-M and ZenDet-L but is set to 192 for ZenDet-S. The fine-tuned results of models pre-trained on ImageNet-1k are reported in Appendix C.
In Table 1, ZenDet outperforms ResNet by a large margin. The improvements are consistent across three detection frameworks. Particularly, when using the newest framework GFLV2, ZenDet improves COCO mAP by +2% at the similar FLOPs of ResNet-50, and speeds up the inference by 1.54x times faster at the same accuracy as ResNet-50. Figure 3 visualizes the comparison in Table 1.
Remark Please note that we did not copy the numbers of baseline methods reported in previous works in Table 1 because the mAP not only depends on the architecture but also depends on the training schedule, such as training epochs, learning rate, pre-training and so on. Therefore, for a fair
comparison, all models in Table 1 are trained by the same training schedule. For comparison with numbers reported in previous works, see subsection 5.3.

heading: 5.3 COMPARISON WITH SOTA NAS METHODS
text: †: SpineNet paper did not report the total search cost, only mentioned that 100 TPUv3 was used.
In Table 2, we compare ZenDet with SOTA NAS methods for the backbone design in object detection. We directly use the numbers reported in the original papers. Since each NAS method uses different design spaces and training settings, it is impossible to make an absolutely fair comparison for all methods that everyone agrees with. Nevertheless, we list the total search cost, mAP and
FLOPs of the best models reported in each work. This gives us an overall impression of how each NAS method works in real-world practice. From Table 2, ZenDet is the only zero-shot (trainingfree) method with 48.0% mAP on COCO, using 0.6 GPU days of search. SpineNet (Du et al., 2020) achieves a slightly better mAP with 2x more FLOPs. It uses 100 TPUv3 for searching but the total search cost is not reported in the original paper. ZenDet achieves better mAP than DetNAS (Chen et al., 2019b) and SP-NAS(Jiang et al., 2020) while being 50 ∼ 100 times faster in search. To further fairly compare different backbones under the same training settings, we train backbones designed by ZenDet-NAS and previous backbone NAS methods in Table 3. Because the implementation of SP-NAS is not open-sourced, we re-train ZenDet, DetNAS and SpineNet on COCO from scratch. Table 3 shows that ZenDet requires fewer parameters and has a faster inference speed on V100 when achieving the competitive performance over DetNAS and SpineNet on COCO.

heading: 5.4 ABLATION STUDY AND ANALYSIS
text: Table 4 reports the ZenDet backbones searched by different evolutionary strategies and whether using multi-scale entropy prior. The COCO mAPs of models trained in two detection frameworks (YOLOF and FCOS) are reported in the right big two columns. YOLOF models are trained by 12 epochs with ImageNet pre-trained initialization, while FCOS models are trained with the 3X training epochs. We also compare their image classification ability on ImageNet-1k. All models are constrained by the FLOPs less than 4.4 G while the number of parameters is not constrained. More details about the searching process and architectures could be found in Appendix G, H.
Single-scale score Compared to ResNet-50, the model searched by single-scale entropy score obtains +0.7% accuracy gain on ImageNet, +2% mAP gain with FPN-free YOLOF and +0.8% mAP gain with FPN-based FCOS. Meanwhile, the model searched by Zen-Score achieves +0.9% accuracy gain on ImageNet, +1.1% mAP gain with YOLOF and +0.1% mAP gain with FCOS.
Multi-scale entropy score When using multi-scale entropy, both single-scale model and multiscale model get similar accuracy on ImageNet. The single-scale model uses 2X more parameters than the multi-scale model under the same FLOPs constraint. In terms of mAP, multi-scale model outperforms single-scale model by +0.3% on COCO with YOLOF and 0.6% on COCO with FCOS. From the last row of Table 4, the coarse-to-fine mutation further enhances the performance of multiscale entropy prior, and the overall improvement over ResNet-50 is +1.1% on ImageNet-1k, +2.5% on COCO with YOLOF and +2.0% on COCO with FCOS.
Correlations during the search To further study the correlations between mAP and scores, models during the search are trained and the results are exhibited in Figure 4. The right part of Figure 4 indicates that the mAP positively correlates with the multi-scale entropy score. The left part
of Figure 4 reveals that the single-scale entropy score cannot represent the mAP well, so multi-scale entropy is necessary for detection tasks. By analysing the structures in Appendix G, the computation of single-scale models is concentrated in the last stage C5, ignoring the C3 and C4 stages, and leading the worse multi-scale score. Instead, multi-scale models allocate more computation to the previous stages to enhance the expressivity of C3 and C4, which improve the multi-scale score.

heading: 5.5 TRANSFER LEARNING
text: †: Numbers are cited from SpineNet paper (Du et al., 2020).
VOC and Cityscapes To evaluate the transferability of ZenDet in different datasets, we transfer the FCOS-based ZenDet-M to VOC and Cityscapes dataset, as shown in the upper half of Table 5. The models are fine-tuned after pre-trained on ImageNet. Comparing to ResNet-50, ZenDet-M achieves +4.1% better mAP in VOC, +1.1% better mAP in Cityscape.
Instance Segmentation The lower half of Table 5 reports results of Mask R-CNN (He et al., 2017) and SCNet (Vu et al., 2021) models for the COCO instance segmentation task with 6X training from scratch. Comparing to ResNet-50, ZenDet-M achieves better AP and mask AP with similar model size and FLOPs on Mask RCNN and SCNet.

heading: 6 CONCLUSION
text: In this paper, we propose a zero-shot NAS method termed ZenDet-NAS for designing high performance backbones for object detection. While achieving better or competitive detection accuracy, the search speed of ZenDet-NAS is several orders of magnitude faster than previous training-based NAS methods. Within one GPU day, the ZenDet automatically designed by ZenDet-NAS is significantly more efficient than popular SOTA backbones for object detection in terms of FLOPs and inference latency. Extensive experiments and analyses on various datasets validate its excellent transferability. In the future, we will generalize ZenDet-NAS to search for detection backbone and FPN neck in a unified framework.

heading: A TRAINING DETAILS
text: Searching Details In ZenDet-NAS, the evolutionary population N is set as 256 while total iterations T = 96000. Residual blocks and bottleneck blocks are utilized as searching space when comparing with ResNet series backbone (He et al., 2016). Following the previous designs (Chen et al., 2019b; Jiang et al., 2020; Du et al., 2020), ZenDet is optimized for the budget of FLOPs according to the target networks, i.e., ResNet-50 and ResNet-101. To balance the computational complexity and large resolution demand, the resolution in search is set as 384 × 384 for ZenDet. When starting the search, the initial structure is composed of 5 downsampling stages with small and narrow blocks to meet the reasoning budget. In the mutation, whether the coarse-mutation or finemutation, the width of the selected block is mutated in a given scale {1/1.5, 1/1.25, 1, 1.25, 1.5, 2}, while the depth increases or decreases 1 or 2. The kernel size is searched in set {3, 5}. Note that blocks deeper than 10 will be divided into 2 blocks equally to enhance the diversity.
Dataset and Training Details For object detection, trainval35k with 115K images in the COCO dataset is mainly used for training. With the single-scale testing of resolution 1333 × 800, COCO mAP results are reported on the val 2017 for most experiments and the test-dev 2007 for GFLV2 results in Table 1. When training on the COCO dataset, the initial learning rate is set to 0.02, and decays two times with the ratio of 0.1 during training. SGD is adopted as optimizer with momentum 0.9; weight decay of 10−4; batch size of 16 (on 8 Nvidia V100 GPUs); patch size of 1333× 800. Additionally, multi-scale training and Synchronized Batch Normalization (SyncBN) are adopted to enhance the stability of the scratch training without increasing the complexity of inference. Training from scratch is used to avoid the gap between ImageNet pre-trained model, to ensure a fair comparison with baselines. 3X learning schedule is applied for the ablation study with a multi-scale range between [0.8, 1.0] (36 epochs, decays at 28 and 33 epochs), and 6X learning schedule for the SOTA comparisons with the range between [0.6, 1.2] (73 epochs, decays at 65 and 71 epochs). All object detection training is produced under mmdetection (Chen et al., 2019a) for fair comparisons, and hyperparameters not mentioned in the paper are always set to default values in mmdection.
For image classification, all models are trained on ImageNet-1K with a batch size of 256 for 120 epochs. When training on ImageNet-1K, We use SGD optimizer with momentum 0.9; cosine learning rate decay (Loshchilov & Hutter, 2017); initial learning rate 0.1; weight decay 4× 10−5.

heading: B ZENDET-NAS FOR MOBILE DEVICE
text: For mobile-size object detection, we explore building ZenDet-MB with MobileNetV2 (Sandler et al., 2018) blocks, using the inverted bottleneck block with expansion ratio of 1/3/6. The weight ratio α is still set as 1:1:6 and other searching settings are the same as the Resnet-like searching. In Table. 6, ZenDet-MB use less computation and parameters but outperform MobileNetV2 by 1% AP with similar inference time on Google Pixel 2 phone. Additionally, inserting SE modules to ZenDet-MB-M could improve the mAP by 0.4%.

heading: C OBJECT DETECTION WITH IMAGENET PRE-TRAIN MODELS
text: In the main body of the paper, training from scratch is used to avoid the gap between ImageNet pre-trained model, to ensure a fair comparison with baselines He et al. (2019). Since 6X training
from scratch inevitably consumes 3 times more time than 2X pre-trained training, we use the ImageNet pre-trained model to initialize the ZenDet-M in various heads, including RetinaNet, FCOS and GFLV2. As present in Table. 1, 7, whether using training from scratch or ImageNet pre-training, ZenDet could outperform ResNet-50 in the three popular FPN-based frameworks by large margins.
†: results in this line are reported in the official github (Li et al., 2021).

heading: D WEIGHTS ARRANGEMENT
text: In Table 8, we tune the different arrangements of multi-scale weights in a wide range. Seven multiscale weight ratios are used to search different models, and all models are trained on the COCO dataset with FCOS and 3X learning schedule. Table 8 shows that if the same weights are arranged to C3-C5, the performance of ZenDet on COCO is worse than ResNet-50. Considering the importance of C5 (discussed in Section 4.2), we increase the weight of C5, and ZenDet’s performance continues
to improve. To further explore the correlations between mAP and scores, we use the seven weight ratios to calculate the different scores of each model, along with the single-scale weight ratio of 0:0:1. The correlations between mAP and different scores are plotted in Figure 5. Taking the results in Table 8 and Figure 5, we confirm the ratio of 1:1:6 may be good enough for the current FPN structure.
E DISCUSSION ABOUT AUXILIARY COMPONENTS
In this section, we explain why auxiliary components are removed during the search and plugged in back for training. Common used three auxiliary components are built in Figure 6 to verify effects on feature variance.
BN BN is a widely used method to re-centre and re-scale the feature to make the network converge faster and more stable. In Figure 6 (a), BN normalize entropy adaptively to the network width (which can be related to output variance). When BN is presented, networks of different widths will have the same entropy value. Hence, BN must be removed when calculating the entropy.
Residual link Building the structure like Figure 6 (b), the variances with or without residual link are 767 or 751 respectively. The residual link brings less than 2% difference in entropy score. The results means that residual link has little effect on entropy, so it could be removed during the search.
SE SE module is used to adaptively recalibrate channel-wise feature responses by explicitly modelling interdependencies between channels Hu et al. (2018). Note that the input is a zero-mean distribution in our computation of the entropy. If SE module is used in the network, the output after global pooling is equal to 0. So, the final output of SE module is equal to 0.5, which loses the ability to model interdependencies between channels.
To sum up, removing these auxiliary components makes our method stable and applicable to most single-branch feed-forward networks.

heading: F COMPARISON WITH ZERO-SHOT PROXIES FOR IMAGE CLASSIFICATION
text: We compare ZenDet with architectures designed by zero-shot proxies for image classification in previous works, including SyncFlow (Tanaka et al., 2020), NASWOT (Mellor et al., 2021), ZenNAS (Lin et al., 2021). For a fair comparison, All methods use the same search space, FLOPs budget 91G, searching strategy and training schedule. All searched backbones are trained on COCO with the FCOS head and 3X training from scratch. The results are reported in Table. 9.
Among these methods, SyncFlow and NASWOT perform worse than ResNet-50 on COCO albeit they show competitive performance in image classification tasks. Zen-NAS achieves competitive performance over ResNet-50. The ZenDet outperforms Zen-NAS by +1.3% mAP with slightly fewer FLOPs and nearly one third of parameters.
G VISUALIZATION OF SEARCHING PROCESS
The visualizations of searching process on different entropy scores are shown in Figure 7, 8, 9. We compare the change of layers, FLOPs and parameters during different iterations. Visualizations prove our assumptions in the main body of the paper.

heading: H DETAIL STRUCTURE OF ZENDETS
text: We list detail structure in Table 1, 4.
The ’block’ column is the block type. ’Conv’ is the standard convolution layer followed by BN and RELU. ’ResBlock’ is the residual bottleneck block used in ResNet-50 and is stacked by two Blocks in our design. ’kernel’ is the kernel size of kxk convolution layer in each block. ’in’, ’out’ and ’bottleneck’ are numbers of input channels, output channels and bottleneck channels respectively. ’stride’ is the stride of current block. ’# layers’ is the number of duplication of current block type.

Review summaries:
In this paper, the authors propose a zero-shot NAS to search backbone for detection task. Specifically, this method uses the differential entropy of output features as a metric to measure the performance of architecture on detection task. With the differential entropy , this method does not need training network parameters, reducing the search cost largely. Also, the backbones searched by this method have the state-of-the-art performance on detection task.

This paper proposes a zero-shot neural architecture search approach for backbone design in object detection. The idea is to compute an entropy-based multi-scale Zen-score and use the score as an objective for evolutionary architecture search. The paper achieves better results comparing with previous zero-shot NAS approaches on object detection and greatly reduces the search time of conventional NAS approaches while maintaining similar accuracy.