ZenDet: Revisiting Efficient Object Detection Backbones from Zero-Shot Neural Architecture Search
Authors: Zhenhong Sun
Ming Lin
Zhiyu Tan
Xiuyu Sun
Rong Jin
Venue: ICLR 2022 Submitted (2022)
Keywords: Object Detection
Detection Backbone
Neural Architecture Search
Zero-Shot NAS

Abstract: In object detection models, the detection backbone consumes more than half of the overall inference cost. Recent researches attempt to reduce this cost by optimizing the backbone architecture with the help of Neural Architecture Search (NAS). However, existing NAS methods for object detection require hundreds to thousands of GPU hours of searching, making them impractical in fast-paced research and development. In this work, we propose a novel zero-shot NAS method to address this issue. The proposed method, named ZenDet, automatically designs efficient detection backbones without training network parameters, reducing the architecture design cost to nearly zero yet delivering the state-of-the-art (SOTA) performance. Under the hood, ZenDet maximizes the differential entropy of detection backbones, leading to a better feature extractor for object detection under the same computational budgets. After merely one GPU day of fully automatic design, ZenDet innovates SOTA detection backbones on multiple detection benchmark datasets with little human intervention. Comparing to ResNet-50 backbone,  ZenDet is $+2.0\%$ better in mAP when using the same amount of FLOPs/parameters and is $1.54$ times faster on NVIDIA V100 at the same mAP. Code and pre-trained models will be released after publication.

Under review as a conference paper at ICLR 2022
ZENDET: REVISITING EFFICIENT OBJECT DETECTION BACKBONES FROM ZERO-SHOT NEURAL ARCHITEC- TURE SEARCH
Anonymous authors Paper under double-bling review
ABSTRACT
In object detection models, the detection backbone consumes more than half the overall inference cost. Recent researches attempt reduce this cost by op- timizing the backbone architecture with the help of Neural Architecture Search (NAS). However, existing NAS methods for object detection require hundreds thousands of GPU hours of searching, making them impractical in fast-paced re- search and development. In this work, we propose a novel zero-shot NAS method to address this issue. The proposed method, named ZenDet, automatically de- signs efficient detection backbones without training network parameters, redue- ing the architecture design cost nearly zero yet delivering the state-of-the-ar (SOTA) performance. Under the hood, ZenDet maximizes the differential entropy of detection backbones, leading to better feature extractor for object detection under the same computational budgets. After merely one GPU day of fully au- tomatic design, ZenDet innovates SOTA detection backbones multiple detec- tion benchmark datasets with little human intervention. Comparing to ResNet- 50 backhone, ZenDet is +2.0% better in mAP when using the same amount of PLOOs//arameters and is 1.54 times faster on NVIDIA V100 at the same mAP. Code and pre-trained models will be released after publication.
INTRODUCTION
Seeking better and faster deep models for object detection is never an outdated task in computer vi- sion. The performance of deep object detection network heavily depends on the feature extraction Chen et al., 2019b). Currently, most state-of-the-art (SOTA) detection
backbone (Li et al. backbones (He et al. 2016 Xie et al. 2017 Zhu al. 2019) are designed manually by human ex- perts which could take years develop. Since the detection backbone consumes more than half of the total inference cost in many detection frameworks, it critical to optimize the backbone archi- tecture for better speed-accuracy trade-off on different hardware platforms varying from server-side GPUs to mobile-side chipsets. To reduce the manual design, Neural Architecture Search (NAS) has emerged facilitate the architecture design. Various NAS methods have demonstrated their efficacy in designing SOTA image classification models (Zoph et al. 20188 Liu et al.) 20118) Cai et al. 2019 Tan & Le 2019). These successful stories inspire recent researchers to use NAS design detection
2018
backbones (Chen et al. 2019b Du et al., 2020 Jiang et al. 2020) in an end-to-end way.
The existing NAS methods for detection backbone design are all training-basedd based. meaning that they need to train network parameters to evaluate the performance of candidates on the target dataset, taking a long time and consuming huge hardware resources. Hardware consumption and long time searching make these wrining-basse methods inefficient in modern fast-paced research and devel- opment. reduce the training cost, training-free methods are proposed recently, also known as zero-shot NAS in previous literatures (Tanaka et al.) 2020 Mellor et al. 2021 Chen et al. 202 lb) Lin et al. 2021). The zero-shot NAS predicts network performance without training network pa- rameters therefore is way faster than training- trainingbbased NAS. As relatively new technique, existing zero-shot NAS methods are mostly validated on image classification datasets. Applying zero-shot NAS to object detection backbone design is still intact challenge.

---

Under review as a conference paper at ICLR 2022
In this work, we present the first effort of introducing zero-shot NAS technique to design efficient object detection backbones. We show that directly transferring existing zero-shot NAS methods from image classification to detection backbone design will encounter fundamental difficulties. While image classification network only needs to predict the class probability, object detection network needs to additionally predict the bounding boxes of multiple objects, making the direct architecture transfer sub-optimal. To this end, a novel zero-shot NAS method, termed ZenDet, is proposed for searching object detection backbones. The key idea behind ZenDet is inspired by the Principle of Maximum Entropy (PME) (Reza 1994, Kullback] 1997, Brillouin 2013). Informally speaking, when a network is formulated as an information processing system, capacity maximized when its differential entropy (Shannon] 1948) achieves maximum under budget constraints, leading to a better feature extractor for object detection. Based on this observation, ZenDet maximizes the differential entropy of detection backhones by searching for the optimal configuration of network depth and width without training network parameters.
The above strategy raises two technical challenges. The first challenge is how to estimate the differential entropy of a deep network. The exact computation of differential entropy requires knowing the precise probability distribution of deep features in high dimensional space which is difficult to estimate in practice. To address this issue, ZenDet estimates the Gaussian upper bound of the differential entropy which only requires computing the variance of the feature maps. The second challenge is how to efficiently capture objects of different sizes. In object detection benchmark datasets such as MS COCO Lin et al.) (2014), the distribution of object size is data-dependent and non-uniform. To bring in this prior knowledge in backbone design, we introduce the Multi-Scale Entropy Prior (MSEP) in backbone entropy estimation capture different-scafe objects. We find that the MSEP improves the detection performance significantly. The overall computation of ZenDet only requires one forward inference of the detection backbone at initialization therefore is nearly zero-cost comparing to previous backbone NAS methods.
The main contributions of this work are summarized as follows:
Based on the entropy theory, the multi-scale entropy prior is present to rank the expressivity of the backbone instead of training on the target datasets, speeding up searching.
While using less than one GPU day and 2GB GPU memory, ZenDet achieves competitive performance over other NAS methods on COCO with at least 50x times faster.
ZenDet is the first pero-shot NAS method designed for object detection with SOTA perfor- mance in multiple benchmark datasets under multiple popular detection frameworks.
2 RELATED WORK
Recently, object detectors composed of backbone, neck and head have become increasingly popular due to their effectiveness and high performance (Lin et al.) 2017a by Tian et al. 2019 Li et al. 2020; 202T). Prevailing detectors directly use the backbone designed for image classification to extract multi-scale features from an image, such as ResNet (He cet al. 2016). RtsNeXt (Xie et al. 2017) and Deformable Convolutional Network (DCN) (Zhu et al. 2019 Nevertheless, the backbone migrated from image classification may be suboptimal in ob- ject detection (Ghiasi et al. 2019). To tackle the gap, many architectures are end-to-end designed for object detection, including Stacked Hourglass Newell al.) (2016), FishNet (un et al. (2018), DetNet Li et al. (2018), HRNet Wang et al. (2020a) and so on. Albeit with improved performance, these hand-cralted detection architectures heavily rely on human labor and tedious trial-and-error processes.
Backbone for Object Detection
Neural Architecuure Search Neural Architecture Search (NAS) is initially developed auto- matically design network architectures for image classification models (Zoph et al. 2018) Liu et al.)
2018 Real et al.) 2019, Cai et al.) 2019 Lin et al. 2020 Tan & Le 2019: Lin al.. 2021). Us- ing NAS to design object detection models has not been well studied. Currently, existing detection NAS methods are all training-based methods. Some methods focus on searching detection back- bones, such as DetNAS (Chen et al., 2019b), SpineNet (Du et al. 2020) and SP-NAS (Jiag et al.) 2020). while the others focus on searching FPN neck, such as NAS- FPN (Ghiasi et al., 2019). NAS. FCOS (Wang et al.) 2020b) and OPANet (Liang al., 2021). These methods require training and evaluation on the target datasets which is intensive in computation. ZenDet distinguishes itself as being the first zero-shot NAS method for the backbone design of object detection.
2

---

Under review as a conference paper at ICLR 2022
3 PRELIMINARY
In this section, we first formulate deep network as system with continuous state space. Then define the differential entropy of this system and show how to estimate this entropy via its Gaussian upper bound. Finally we introduce the basic concept of vanilla network search space for designing our detection backbones.
Continuous
State Space of Deep Networks A deep network F(.) Rd R maps input image æ E R2 to its label y E R. The topology of a network can be abstracted as graph g = (V. E) where the vertex set V consists of neurons and the edge set & consists of spikes between neurons. For any E V and and each edge e respectively. The set S (h(v),hle) : Vu E V.e E E} defines the continuous state space of the network F.
E, h(e) E R and h(e) E R present the values endowed with each vertex
According to the Principle of Maximum Entropy, we want to maximize the differential entropy of network F, under some given computational budgets. The entropy H(S) of set S measures the total information contained in the system (network) F, including the information contained in the latent features H (S.) {h(v): E V} and in the network parameters H (S.)) {h(e) E E}. As for object detection backbone design, we only care about the entropy of latent features (SS)) rather than the entropy of network parameters (Se).. Informally speaking, H (S.) measures the feature representation power of F while (Se,) measures the model complexity of F. Therefore, in the remainder of this work, the differential entropy of F refers to the entropy H(S,) by default.
Entropy of Gaussian Distribution The differential entropy of Gaussian distribution can be found in many textbooks such as (Norwich 1993). Suppose is sampled from Gaussian distribution N(,, o²). Then the differential entropy of x is given by
1 2 log 2T) + (x)
11 (x)
H(x) = log(r)
From Eq. [ the entropy of Gaussian distribution only depends on the variance. In the following, we will use (x) instead of HT* (1)) as constants do not matter in our discussion.
Gaussian Entropy Upper Bound Since the probability distribution PPSS.) is a high dimensional function, it is difficult to compute the precise value of its entropy directly. Instead, we propose to estimate the upper bound of the entropy, given by the following well-known theorem (Cover & Thomas 2012):
Theorem 1. For any continuous distribution P(z) of mean μ and variance o², its differential entropy is maximized when (()) is Gaussian distribution N N(,,**
Theorem T says that the differential entropy of a distribution is upper bounded by Gaussian distri- bution with the same mean and variance. Combining this with Eq. { we can easily estimate the network entropy (S,) by simply computing the feature map variance and then use Eq. I to get the Gaussian entropy upper bound for the network.
Vanilla Network Search Space Following previous works, design our backbones in the vanilla
convolutional network space (Li et al.] 20188] Chen et al.] [019b] Du et al. 20200: Lin et al., (202I)
because this space is widely adopted in detection backbones and used a prototype in theoretical
literature (Poole et al.] 2016, [Serra et al. 2018 Hanin & Rolnick 2019). A vanilla network is
stacked by multiple convolutional layers followed by RELU activations. Auxiliary components such as residual link (He et al. 2016). Batch Normalization (BN) (loffe & Szegedy| (2015) and Squeeze-and - (SE) (Hu et al. 2018) are all removed during the search and only during the search (More details in Appendix E. These removed auxiliary components are plugged in back after the search. Therefore, the final architecture for training still has these components.
Consider a vanilla convolutional network with D layers of weights W .... W whose output feature maps are x....., D. The input image is at" Let (b-) denote the RELU activation function. Then the forward inference is given by
a' = (th') h'= * for l = D.
(2)
For sake of simplicity, we set the bias of the convolutional layer zero.
3

---

Under review as a conference paper at ICLR 2022
C7/128
F7
Coew w4-3((01)
Conv -11061))))))
OR/44
A"
CS02 BBC50
(01)
Sollette furuer oup.
04/16 MECI
Exl 25
PC
Conw W4-00200
C388
y' >-|=|]
C24
EPN
Hoad
x2-11(011)
C1/2
ths (N)((")) 1ete)
Inpet
Uppicplle
217) =el
+ -od8055
(a) Single-scale entropy score with rescaling
(b) Multi-scale entropy score for detection
Figure 1: Computational graph of entropy score for detection backbone with multi-scale features.
ZENDET-NAS FOR OBJECT DETECTION
In this section, first describe how compute the differential entropy of the single-scale feature for very deep vanilla networks with the re-scaling trick. Then we introduce the Multi-Scale Entropy Prior (MSEP) better capture the prior distribution of object size in ZenDet. Finally, we present the complete ZenDet-NAS under the Evolutionary Algorithm (EA) framework.
4.1 SINGLE-SCALE ENTROPY FOR VERY DEEP BACKBONES
To compute the entropy of the detection backbone, we initialize all backbone parameters by the stan- dard Gaussian distribution (0, 1). Then randomly generate an image filled with the standard Gaussian noise and perform forward inference. Based on the discussion in Section 3 the (Gaussian upper bound) entropy H F) of the network F is then given by
1
H(F) = log log(Var(h?)))
(3)
Please note that the variance is computed on the last pre-activation feature map h
For very deep networks, directly using Eq. (3) might cause a numerical overflow. The same problem is also observed in Zen-NAS (Lin et al. 2021). Inspired by the analysis in Zen-NAS where the authors show that the gradient norm scale-invaa invariiniiiant to BN layers after variance compensation, we propose a more straightforward solution. We directly re-scale each feature map x' by some constants yd during inference, that x² = (hh')//", and then compensate the entropy of the network by
H(F) log (Va((h")) + Σ log(n').
1
The values of of can be arbitrarily given as long as the forward inference does not overflow or underflow. In practice, we find that simply setting 7¹t to the Fuclidean norm of the feature map works well. The overall process is illustrated in Figure (a).
Comparison with Zen-NAS Although the definition of vanilla network and solution to overflow are similar, the principles of single-scale entropy and Zen-NAS are fundamentally different. Zen- NAS uses the gradient norm of the input image ranking score and proposes to use two feed- forward inferences to approximate the gradient norm for classification. In contrast, ZenDet uses inference, doubling the search speed.
entropy-baad based score which only requires one
4.2 MULTI-SCALE ENTROPY PRIOR FOR OBJECT DETECTION
In real-world images, the distribution of object size is not uniform. To bring in this prior knowl- edge, the detection backbone has 5 stages where each stage downsamples the feature resolution to half. The MSEP collects feature maps from the final output of each stage and weighted-sum the corresponding feature map entropies as a new measurement. We name this new measurement as multi-scale entropy. The overall process is illustrated in Figure (b). In this figure, the backbone
4

---

Under review as a conference paper at ICLR 2022
extracts multi-scale features C (C1, C2, C5) at different resolutions. Then the FPN neck fuses C' as input features P (P1, P2. P2. P7) for the detection head. The multi-scale entropy (F)) of backbone F is then defined by
Z(F) = an (C1) 0. H(()) + +... (C5)
(5)
where (Ci) is the entropy of Ci for i =, 1, ,5. The weights a ,0,, ,05) store the multi-scale entropy prior to balance the expressivity of different scale features.
Weights selection strategy As concrete example Fig. [(b] the parts of P3 and P4 are generated by up-sampling of P5, and P6 and P7 are directly generated by down-sampling of P5 (generated by C5). Meanwhile, based on the fact that C5 carries sufficient context for detecting objects on various scales (Chen et al.) 2021a), C5 is important in the backhone search, so it is good set a larger value for the weight ag. Then, different combinations of and correlation analysis are explored in Appendix D indicating that (0, 0, 1,1,6 6) is good enough for the FPN structure.
4.3 ZENDET-NAS IN EVOLUTIONARY ALGORITIM FRAMEWORK
Algorithm ZenDet-NAS with Coarse-to-Fine Evolution
Require: Search space S, inference budget B, maximal depth L, total number of iterations T, evolutionary population size N, initial structure Fo, fine-search flag Flag.
Ensure: NAS-designed ZenDet backbone F*
1: Initialize population P {Po}, Flag=False.
2: for t 1, . do
3: if tf equals to T/2 then
4: Keep top 10 networks of highest multi-scale entropy in P and remove the others.
5: Set Flag = True. end if
6:
7:
Randomly select Fg E P.
8: Mutate F, = MUTATE(F,, S, Flag)
9: if F, exceeds inference budget has more than L layers then
10: Do nothing.
11:
else
12:
Get multi-scale entropy ((F))
13:
Append F; to P.
end if
15:
Remove networks of the smallest multi-scale entropy if the size of P exceeds B.
end for
17:
Return F* the network of the highest multi-scale entropy in P.
Stnictune
Algorithm MUTATE
C3
Require: Structure F;, search space S, fine- search flag Flag.
g
Uniforlly solor baok
Ensure: Randomly mutated structure f.
1: Uniformly select block h in F..
2: if Flag equals to True then
3: Uniformly alternate the kernel size, width within some range.
Aerrel
wiah
desth
kemal
widha
4: else
J
the montret block
5: Uniformly alternate the block type, kernel size, width and depth within some range.
6: end if
Mulated structure fz Figure 2: Visualization of Algorithm
7: Return the mutated structure Fr.
Combining all above, present our ZenDet-NAS in Algorithm 1
The ZenDettNAS maximizes
5

---

Under review as a conference paper at ICLR 2022
the multi-scale differential entropy of detection backbones using Evolutionary Algorithm (EA). To improve the evolution efficiency, a coarse-to- strategy is proposed to reduce the search space gradually. First, we randomly generate N seed architectures to fill the population P. As shown in Figure 2 a seed architecture F. consists of a sequence of building blocks such as ResNet block (He) et al., 20T6) or MobileNet block (Sandler et al. 2018). Then we randomly select one block and replace it with its mutated version. We use coarse-mutation in the early stage of EA and switch to fine-mutation after 2/2 EA iterations. In the coarse- mutation, the block type, kernel size, depth and width are mutated randomly; in the fine-m mutation, only kernel size and width are mutated.
After the mutation, if the inference cost of the new structure Fe does not exceed the budget (c.g., FLOPs, parameters and latency) and its depth is smaller than budget L, Fe is appended into the population P. The maximal depth prevents the algorithm from generating over-deep structures. During EA iterations, the population is maintained to certain size by discarding the worst candidate of the smallest multi-scale entropy. At the end of ZenDet-NAS, the backbone with the highest multi- scale entropy is returned.
5 EXPERIMENTS
In this section, we first describe detail settings for searching and training with ZenDet-NAS. Then in subsection 5.2 we apply ZenDet-NAS design better ResNet-like backbones on COCO dataset (Lin et al. 2014). We align the inference budget with ResNet-50/101. The performance of ZenDet and ResNet are compared under multiple detection frameworks including RetinaNet (Lin al. 2017b). FCOS (Tian et al. 2019), and GFLV2 (Li al.) (021). For faireess, use the same training setting in all experiments for all backbones. In subsection 5.3 we compare the search cost of ZenDet SOTA NAS methods for object detection. Subsection r.4orts the ablation studies of different components in ZenDet.NAS. Finally, subsection [5.5) verifies the transferability of ZenDets on several detection datasets and segmentation tasks.
5.1 EXPERIMENT SETTINGS
Searching Details In ZenDet-NAS, the evolutionary population N is set to 256. The total EA
iterations T 96000. Following the previous designs (Chen et al.] 2019b Jiang et al., 2020 Du et al.
2020). ZenDet is optimized for FL.PPs. The resolution for computing entropy is 384 x 384.
Dataset and Training Details
We evaluate detection performance COCO (Lin et al. 2014)
using the official training/testing splits. The mAP is evaluated on val 2017 by default and GFLV2
additionally evaluated on test-dev 2007 following common practice. All models are trained from scratch (He et al. (019) for 6X (73 epochs) on COCO. Following the Spinenet (Du et al.) 2020), we use multi-scale training and Synchronized Batch Normalization (SyncBN). For VOC dataset, train-val 2007 and train-val 2012 are used for training, and test 2007 for evaluation. For image classification, all models are trained on ImageNet- Ik (Deng et al., 2009) with a batch size of 256 for 120 epochs. Other setting details can be found in Appendix A
5.2 DESIGN BETTER RESNET-LIKE BACKBONES
We search efficient ZenDet backbones for object detection and align with ResNet-50/101 in Table 1 ZenDet-S uses 60% less FLOPs than ResNet-50; ZenDet-M is aligned with ResNet-50 with similar FLOPs and number of parameters as ResNer-50; ZenDet-L is aligned with ResNet-101. The feature dimension in the FPN and heads is set to 256 for MenDet-M and ZenDet-L but is set to 192 for S.nDet-S. The fine-tuned results of models pre-trained on ImageNet- Ik are reported in Appendix C
In Table D ZenDet outperforms ResNet by large margin. The improvements are consistent across three detection frameworks. Particularly, when using the newest framework GFLV2, ZenDet im- proves COCO mAP by +2% at the similar FLOPs of ResNet-50, and speeds up the inference by 1.54x times faster at the same accuracy as ResNet-50. Figure visualizes the comparison in Table
Remark Please note that did not copy the numbers of baseline methods reported in previous works in Table [ because the mAP not only depends on the architecture but also depends on the training schedule, such as training epochs, learning rate, pre-training and so on. Therefore, for fair

---

Under review as a conference paper at ICLR 2022
Table 1: Zet and ResNet the COCO. All results using the training setting. FPS on V100 is benchmarked on the full model with NVIDIA V100 GPU, pytorch, FP32, batch size 32.
FLOPs Backbone
Params
val2017
test-dev AP APeeet
FPS on V100
Backbone
Backbone
Head
AP
APs s
APM
APr
RetinaNet
40.2
24.3
43.3
52.2
23.2
R50
83.6G
23.5M
FCOS
42.7
28.8
46.2
53.8
27.6
GFLV2
44.7
29.1
48.1
56.6
45.1
24.2
RetinaNet
42.1
25.8
45.7
54.1
18.7
R101
159.5G
42.4M
FCOS
44.4
28.3
47.9
56.9
21.6
GFLV2
46.3
29.9
50.1
58.7
46.5
19.4
RetinaNet
40.0
23.9
43.3
52.7
35.5
7enDet-S
48.7G
21.2M
FOOS
42.5
26.8
46.0
54.6
43.0
GFLV2
44.7
27.6
48.4
58.2
44.8
37.2
RetinaNet
42.0
26.7
45.2
55.1
21.5
ZenDet-M
89.9G
25.8M
FCOS
44.5
28.6
48.1
56.1
24.2
GFLV2
46.8
29.9
50.4
60.0
46.7
22.2
RetinaNet
43.0
27.3
46.5
56.0
17.6
ZenDet-L
152.9G
43.9M
FCOS
45,9
30.2
49.4
58.4
19.2
GFLV2
47.6
30.2
51.8
60.8
48.0
18.1
48
48
GFLV2-ZenDat
GFLV2HZeaDe: GFL.
GFI.V--RoNet
2.00 MAp
FCOS-ResNer
1.54x speed-up
FCOS ReNNet
mAP (%)
mAP (%))
ReinaNee-ZeeDe
44
Reiin
44
RelinNee-Reeaa
40
40
100
150
200
250
300
350
400
20
30
40
50
80
Model FLOPS (G)
Inference time on V100 (ms/img)
(a) mAP vs. FLOPs
(b) mAP Speed
Figure 3: mAP vs. FLOPs and inference speed on COCO val 2017 in Table Note that FLOPs in (a) is the value of the full detector, cotaining backbone, FPN and head.
comparison, all models in Table I are trained by the same training schedule. For comparison with numbers reported in previous works, subsection 5.3
5.3 COMPARISON WITH SOTA NAS METHODS
Table 2: Comparisons with SOTA NAS methods for object detection. FLOPs are counted for full detector.
Training- free
Search Cost GPU Days
FL.OPs All
Pretrain/ Scratch
COCO
Method
Search Part
Epochs
(A (APtest)
DetNAS
X
68
backbone
289G
Pretrain
24
43.4
SP-NAS
X
26
backbone
655G
Pretruin
24
47.4
SpineNet
X
100x TPUv3
backbonc+FPN
524G
Scratch
48.1
ZenDet
0.6
backbone
279G
Scratch
73
48.0
t: SpineNet paper did report the total search cost, only mentioned that 100 TPUv3 was used.
In Table 2 we compare ZenDet with SOTA NAS methods for the backbone design in object detec- tion. We directly the numbers reported in the original papers. Since each NAS method uses different design spaces and training settings, is impossible to make an absolutely fair comparison for all methods that everyone agrees with. Nevertheless, list the total search cost, mAP and
7

---

Under review as a conference paper at ICLR 2022
Table 3: Comparisons between ZenDet, DetNAS (Chen et.
2019b) and SpineNet (Du al.
2020
under the same training settings. All backbones are trained under GFLV2 head with 6X training epochs. FLOPs and parameters are counted for full detector.
FPS
Backbone
Scarch Part
Scarch Space
FLOPs
Params
AP val
AP3
APM
AP2 L
on V100
ShufleNet22
DeINAS-3.8G
backbone
205G
35.5M
46.4
29.3
50.0
59.0
17.6
+Xception
SpineNet-96
backbone+FPN
ResNet Block
216G
41.3M
46.6
29.8
50.2
58.9
19.9
ZenDet-M
backbone
ResNet Block
215G
34.9M
46.8
29.9
50.4
60.0
22.2
FL.OPs of the best models reported in each work. This gives us an overall impression of how each NAS method works in real-world practice. From Table ZenDet is the only zero-shot (training- free) method with 48.0% mAP on COCO, using 0.6 GPU days of search. SpineNet (Du et al. 2020) achieves a slightly better mAP with 2x more FLOPs. It uses 100 TPUv3 for searching but the total search cost is not reported in the original paper. ZenDet achieves better mAP than DetNAS (Chen et al. 2019b) and SP-NAS(Jiang al. 2020 while being 50 ~ 100 times faster in search.
To further fairly compare different backbones under the same training settings, we train backbones designed by ZenDet-NAS and previous backbone NAS methods in Table 3 Because the implemen- tation of SP-NAS is not open-sourced, we re-train ZenDet, DetNAS and SpineNet on COCO from scratch. Table 3 shows that ZenDet requires fewer parameters and has faster inference speed V100 when achieving the competitive performance over DetNAS and SpineNet on COCO.
5.4 ABLATION STUDY AND ANALYSIS
Table Comparison of different evolutionary searching strategies NenDet-NAS. C-to-F: Coarse- to-Fine. Zen-Score is the proxy in Zen-NAS (Lin et al. 2021
ImageNet-II
COCO with YOLOF
COCO with FCOS
Score
Mutation
FLOPs
Params
TOP-1 %
AP vol
APs
APH
AP2
AP
APs
AP
AP2
R50
None
4.1G
23.5M
78.0
37.8
19.1
42.1
53.3
38.0
23.2
40.8
47.6
Zen Score
Coarse
4.4G
67.9M
78.9
38.9
19.0
43,2
56.0
38.1
23.2
40.5
48.1
Single-scale
Coarse
4.4G
60.1M
78.7
39.8
19.9
44.4
56.5
38.8
23.1
41.4
50.1
Multi-scale
Coarse
4.3G
29.4M
78.9
40.1
21.1
44.5
55,9
39.4
23.7
42.3
50.0
Multi scale
C-tn.F
4.4G
25.8M
79.1
40.3
20.88
44,7
56.4
40.0
24.5
42.6
50.6
Table reports the ZenDet backbones searched by different evolutionary strategies and whether using multi-scale entropy prior. The COCO mAPs of models trained in two detection frameworks (YOLOF and FCOS) are reported in the right big two columns. YOLOF models are trained by 12 epochs with ImageNet pre-trained initialization, while FCOS models are trained with the 3X training epochs. We also compare their image classification ability on ImageNet- All models are constrained by the FLOPs less than 4.4 G while the number of parameters is not constrained. More details about the searching process and architectures could be found in Appendix
Single-scale score Compared to ResNet-50, the model searched by single-scale entropy score obtains +0.7% accuracy gain on ImageNet, +2% mAP gain with FPN-free YOLOF and +0.8% mAP gain with FPN-based FCOS. Meanwhile, the model searched by Zen-Score achieves +0.9% accuracy gain on ImageNet, +1.1% mAP gain with YOLOF and +0.1% mAP gain with FCOS.
Multi-scale entropy score When using multi-scale entropy, both single-scale model and multi- scale model get similar accuracy on ImageNet. The single-scale model uses 2X more parameters than the multi-scale model under the same FLOPPs constraint. In terms of mAP, multi-scale model outperforms single-scale model by +0.3% on COCO with YOLOF and 0.6% on COCO with FCOS. From the last row of Table the coarse-to-fine mutation further enhances the performance multi- scale entropy prior, and the overall improvement over Re is +1.1% on ImageNet-1k, +2.5% on COCO with YOLOF and +2.0% on COCO with FCOS.
Correlations during the search To further study the correlations between mAP and scores, models during the search are trained and the results are exhibited in Figure + The right part of Figure T indicates that the mAP positively correlates with the multi-scale entrop score. The left part
8

---

Under review as a conference paper at ICLR 2022
of Figure 4 reveals that the single-scale entropy score cannot represent the mAP well, multi-scale entropy is necessary for detection tasks. By analysing the structures Appendix G the computation of single-scale models is concentrated in the last stage C5, ignoring the C3 and C4 stages, and
leading the worse multi-scale score. Instead, multi-scale models allocate more computation to the previous stages enhance the expressivity of C3 and C4, which improve the multi-scale score.
Iteration:: 4k, 8k, 12k, 30k, 50k, 75k, 96k
Iteratios:: O, 4k, 8k, 12k, 30k, 50k, 75k, 96k
40
40
39
39
mAP (%)
38
38
37
37
Initial
Mulli-CoosF
Initial
Multi- Conne
36
Single Course
36
Single Course
240
250
260
270
280
1700
1720
1740 1760 1780 1800 1820
Scores with ratio 0:0:1
Scores with ratio 1:1:6
Figure 4: mAP (on FCOS) vs. scores during the search with different search strategies. The scores on the left and the right are computed with the ratio of 0:0:1 and 1:1:6 respectively. Starting from the initial point, the dotted line indicates the evolution direction the search process.
5.5 TRANSFER LEARNING
Transferability of ZenDet in multiple object detection and instance segmentation tasks. Table 5:
FLOPs reported are counted for full detector.
Task
Datuset
Head
Backbone
Resolution
Epochs
FL.OPs
AP val
R50
1000 600
12
120G
76.8
VOC
Object Delection
ZenDet-M
1000 600
12
1236
80.9
FCOS
R50
2048 x 1024
64
411G
37.0
Citescapes
ZenDet-M
2048 x 1024
64
426G
38.1
R50
1333 x 800
73
375G
43.2
39.2
MASK R-CNN
ZenDett-M
1333
73
379G
44.5
40.3
Instance Segmentation
R50
640 640
350
228G
42.7
37.8
COCO
S9ineNet-49
640 640
350
216G
42.9
38.1
R50
1333 x 800
73
67TG
46.3
41.6
SCNet
ZenDet-M
1333 800
73
675G
47.1
42,3
t: Numbers are cited from SpineNet paper
(Du et al.
2020).
VOC and Cityscapes To evaluate the transferability of ZenDet in different datasets, we transfer the FCOS-based ZenDet-M to VOC and Cityscapes dataset, as shown in the upper half of Table The models are fine-tuned after pre-trained on ImageNet. Comparing ResNet-50, ZenDet-M achieves +4.1% better mAP in VOC, +1.1% better mAP in Cityscape.
Instance Segmentation The lower half of Table reports results of Mask R-CNN (He et al. 2017) and SCNet (Vu et al.) (021) models for the COCO instance segmentation task with 6X training from scratch. Comparing to ResNet-50, ZenDet-M achieves better AP and mask AP with similar model size and FLOPs on Mask RCNN and SCNet.
CONCLUSION
In this paper, we propose a zero-shot NAS method termed ZenDet-NAS for designing high perfor- mance backbones for object detection. While achieving better competitive detection accuracy, the search speed of ZenDet-NAS methods. Within one GPU day. the ZenDet automatically designed by ZenDet-NAS is significantly more efficient than popular SOTA backbones for object detection in terms of FLOPs and inference latency. Extensive experiments and analyses various datasets validate its excellent transferability. In the future, we will generalize ZenDet-NAS to search for detection backbone and FPN neck in a unified framework.
several orders of magnitude faster than previous training-based NAS

---

Under review as a conference paper at ICLR 2022
REFERENCES
Leon Brillouin. Science and information theory. Courier Corporation, 2013.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv: (908:v99000999// 2019.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: /XXxv/9006.00 9996000055, 2019a.
Qiang Chen, Yingming Wang. Tong Yang. Xiangyu Zhang, Jian Cheng, and Jian Sun. You only look one-level feature. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13039-13048, 2021a.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search imagenet in four gpu hours: A theoretically inspired perspective. In Proceedings of the International Conference on Learning Representations, 2021b.
Yukang Chen, Tong Yang. Xiangyu Zhang, Gaofeng Meng, Xinyu Xiao, and Jian Sun. Detnas: Backbone search for object detection. Advances in Neural Information Processing Systems, 32: 6642-6652, 2019b.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. John Wiley & Sons, 2012.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. 255. leee, 2009.
Xianzhi Du, Tsung-Yi Lin, Pengehong Jin, Golnaz Ghiasi, Mingxing Tan, Yin Cui, Quoc V Le, and Xiaodan Song. Spinenet: Learning scale-permuted backbone for recognition and localization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11592-11601, 2020.
Golnaz Ghiasi, Tsung- Yi Lin, and Quoe Le. Nas-fpn: Learning scalable feature pyramid architec- ture for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7036-7045, 2019.
Boris Hanin and David Rolnick. Complexity of linear regions in deep networks. In International Conference on Machine Learning, pp. 2504-2604. PMLR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4918-4927, 2019.
Jie Hu, Li Shen, and Gang Sun. snneeez--and-excitatiin networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp. 448-456. PMLR, 2015.
Chenhan Jiang, Hang Xu, Wei Zhang, Xiaodan Liang, and Zhenguo Li. Sp-nas: backbone search for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11863-11872, 2020.
Serial-to- torial-t-parralell
10

---

Under review as a conference paper at ICLR 2022
Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.
Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detec- tion. In NeurlPS, 2020.
Xiang Li, Wenhai Wang. Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss v2: Learning reliable localization quality estimation for dense object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp-
2021.
Zeming Li, Chao Peng, Gang Yu, Xiangyu Zhang, Yangdong Deng, and Jian Sun. Detnet: Design backbone for object detection. Proceedings of the European conference on computer vision (ECCV), pp. 334-350, 2018.
Tingting Liang, Yongtao Wang. Zhi Tang, Guosheng Hu, and Haibin Ling. Opanas: One-shot path aggregation network architecture search for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10195-10203, 2021.
Ji Lin, Wei-Ming Chen, John Cohn, Chuang Gan, and Song Han. Mcunet: Tiny deep learning iot devices. In Anmual Conference on Neural Information Processing Systems (NeurIPS), 2020.
Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas: A zero-shot for high-performance deep image recognition. In 2021 IEEE/CVE International Conference on Computer Vision, 2021.
Tsung-V Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017a.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017b.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: 180..0905, 2018
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In Pro- ceedings of the International Conference on Learning Representations, 2017.
Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In International Conference on Machine Learning, pp. 7588-7598. PMLR, 2021.
Alejandro Newell, Kaiyu Yang. and Jia Deng. Stacked hourglass networks for human pose estima- tion. In European conference on computer vision, pp. 483 483-499. Springer, 2016.
Kenneth H Norwich. Information, sensation, and perception. Academic Press San Diego, pp. 81-82, 1993.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponen- tial expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29:3360-3344 2016.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
Fazlollah M Reza. An introduction information theory. Courier Corporation, 1994.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 4510 2018

---

Under review as a conference paper at ICLR 2022
Thiago Serra, Christian Tjandraatmadja, and Srikumar Ramalingam. Bounding and counting linear regions deep neural networks. In International Conference on Machine Learning, pp. 4558- 4566. PMLR. 2018.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423, 1948.
Shuyang Sun, Jiangmiao Pang, Jianping Shi, Shuai Yi, and Wanli Ouyang. Fishnet: A versatile backbone for image, region, and pixel level prediction. Advances Neural Information Pro- cessing Systems, pp. 760-770, 2018.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6105-6114 14. PMLR, 2019.
Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Proceedings of the International Conference on Neural Information Processing Systems, 2020.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEEF/CV international conference on computer vision, pp- 9627-9636, 2019.
Thang Vu, Haeyong Kang. and Chang D Yoo. Scnet: Training inference sample consistency for in- stance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2000-2700999 2021.
Jingdong Wang. Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu. Mingkui Tan, Xinggang Wang, et al. Deep high- high-resolution representation learning for visual recognition. IEFE transactions on pattern analysis and machine intelligence, 2020a.
Ning Wang, Yang Gao, Hao Chen, Peng Wang. Zhi Tian, Chunhua Shen, and Yanning Zhang. Nas-fcos: Fast neural architecture search for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11943-1 11911, 2020b.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9308-9316, 2019.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018
12

---

Under review as a conference paper at ICLR 2022
A TRAINING DETAILS
Searching Details In ZenDet-NAS, the evolutionary population N is set as 256 while total it- erations T 96000. Residual blocks and bottleneck blocks utilized searching space when comparing with ResNet series backbone (He et al. 2016). Following the previous designs (Chen) et al., 2019b; Jiang et al. 2020; Du et al. 2020), ZenDet is optimized for the budget of FLOPs according to the target networks, Lee. ResNet-50 and ResNet-101. To balance the computational complexity and large resolution demand, the resolution in search is set as 384 x 384 for ZenDet. When starting the search, the initial structure is composed of 5 downsampling stages with small and narrow blocks to meet the reasoning budget. In the mutation, whether the coarse-mutation or fine- mutation, the width of the selected block is mutated in given scale {1/1.5, / 1.1.5.. 1, 1.25, 1.5, 2), while the depth increases or decreases or 2. The kernel size is searched in set {3, 5}. Note that blocks deeper than 10 will be divided into 2 blocks equally to enhance the diversity.
Dataset and Training Details For object detection, trainval35k with 115K images in the COCO dataset is mainly used for training. With the single-scale testing of resolution 1333 x 800, COCO mAP results are reported on the val 2017 for most experiments and the test-dev 2007 for GFLV2 results in Table [ When training on the COCO dataset, the initial learning rate is set to 0.02, and decays two times with the ratio of 0.1 during training. SGD is adopted as optimizer with momentum 0.9; weight decay of 10 4; batch size of 16 (on 8 Nvidia V V100 GPUs); patch size of 1333 x 800.
Additionally, multi-scale training and Synchronized Batch Normalization (SyncBN) are adopted to enhance the stability of the scratch training without increasing the complexity of inference. Training from scratch is used to avoid the gap between ImageNet pre-train pre-trained model, to ensure a fair com- parison with baselines. 3X learning schedule is applied for the ablation study with multi-scale range between [0.8, 1.0) (36 epochs, decays at 28 and 33 epochs), and 6X learning schedule for the SOTA comparisons with the range between [0.6, 1.2] (73 epochs, decays 65 and 71 epochs). All object detection training is produced under mmdetection (Chen et al. (019a) for fair comparisons, and hyperparameters not mentioned in the paper are always set default values in mmdection.
For image classification, all models are trained on ImageNet- IK with batch size of 256 for 120 epochs. When training on ImageNet- 1K, We use SGD optimizer with momentum 0.9; cosine learn- ing rate decay (Loshchilov & Hutter 2017); initial learning rate 0.1; weight decay 4 x 10
B ZENDET-NAS FOR MOBILE DEVICE
Table 6: ZenDet-MB and MobileNetV2 on the COCO with the SSDLite head, which are trained from scratch with 600 epochs at resolution 320. FPS on Pixel 2 is benchmarked on the full model with CPU, FP32, batch size 1. ZenDet-MB MB-BettM--MM M--E means inserting SE modules to ZenDet- MB-ttMMMMMMMM-
FLOPPs
Params Backbone
val2017
FPS Pixel 2
Backhone
Backbone
APval
APs
APM
AP1
MobilcNexV2-0.5
217M
0.7M
14.7
0.8
11.0
31.2
13.5
MobileNetV2-1.0
651M
2.2M
21.1
1.7
20.5
39,9
ZenDet-MBBS
201M
0.6M
15.9
0.8
12.2
31.8
13.8
ZenDet-MB-M M
645M
2.0M
22.2
2.1
21.5
42.3
6,3
ZenDet-MB-M-SE
647M
2.3M
22.6
2.3
22.0
42.5
5.6
For mobile-size object detection, we explore building Me with MobileNetV2 (Sandler al. 2018) blocks, using the inverted bottleneck block with expansion ratio of 1/3/6. The weight ratio a is still set as 1:1:6 and other searching settings are the same as the Resnet-like searching. In Table. 6 ZenDet-MB use less computation and parameters but outperform MobileNetV2 by 1% AP with similar inference time on Google Pixel 2 phone. Additionally, inserting SE modules to ZenDet-MB-N MM could improve the mAP by 0.4%.
C OBJECT DETECTION WITH IMAGENET PRE-TRAIN MODELS
In the main body of the paper, training from scratch is used to avoid the gap between ImageNet pre-trained model, to ensure fair comparison with baselines He et al. (2019). Since 6X training
13

---

Under review as a conference paper at ICLR 2022
from scratch inevitably consumes 3 times time than 2X pre- pre-trained training, the Im- ageNet pre-trained model to initialize the ZenDet-M in various heads, including RetinaNet, FCOS and GFLV2. As present in Table. 7 whether using training from scratch or ImageNet pre-traine pre-training, ZenDet could outperform ResNet-50 in the three popular FPN-based frameworks by large margins.
Table 7:
Results between Scratch and Pretrain strategy on the COCO with single-scale testing.
Training strategy on ImageNet is same as Table
FLOPs Backhone
Params Backhone
Backbone
Head
Strategy
Epochs
AP val
APs
AP
AP1
GFLV2
Scratch
73
44.7
29.1
48.1
56.6
R50
83.6G
23.5M
GFLV2
Pretrain
24
44.0
27.1
47.8
56.1
GFLV2
Pretrain
24
43.91
RetinaNet
Scratch
73
42.0
26.7
45.2
55.1
RetinaNet
Pretrain
24
42.3
25.3
46.5
56.0
FCOS
Scratch
73
44.5
28.6
48.1
ZenDet-M
89.9G
25.8M
FCOS
Pretrain
24
44.5
28.8
48.5
56.9
GFLV2
Scratch
73
46.8
29.9
50.4
60.0
GFLV2
Pretrain
24
46.0
29.0
50.0
59.9
t: results in this line reported in the official github (Li al. 2021)
D WEIGHTS ARRANGEMENT
Table 8: Results between different arrangements of weights in MSEP on COCO. R50 represents ResNet-50.
Backbone
FLOPs
Params
AP
AP
AP 73
APs
AP
AP
R50
None
83.6G
23.5M
38.0
55.2
41.0
23.2
40.8
47.6
ZenDet
1:1:1
84.4G
11.5M
37.4
54.6
40.0
23.6
39.8
46.6
ZenDet
1:1:2
84.8G
13.4M
37.8
$4.9
23.2
ZenDet
1:1:4
85.9G
17.2M
38.6
56.0
41.4
23.4
41.3
48.6
ZenDet
1:1:6
88.7G
29.4M
39.4
57.3
42.1
23.7
42.3
50,0
ZenDet
1:1:8
89.9G
31.7M
39.4
57.2
42.0
23.7
42.5
49,5
ZenDet
1:4:1
86.3G
10.9M
35.7
52.6
38.3
22.2
38.1
44.9
ZenDet
4:1:1
86.1G
11.1M
33.9
50.2
36.7
20.4
36.1
mAP (%6
38
IAP (%6)
mAP (%))
mAP (96)
36
36
34
220
240
260
500
550
600
750
800
850
1260
1200
1300
Scores with ratio 0:0:1
Scores with ratio 1:1:1
Scores with ratio 1:1:2
Scores with ratio 1:1:4
mAP (%)
38
mAP (%))
38
mAP((%) (%)
38
mAP (%)
38
36-
36
36
36
34
34
34
34
1700
1730
1800
2200
2300
1100
1200
600
800
1000
1200
Scores with ratio 1:1:6
Scores with ratio 1:1:8
Scores with ratio 1:4:1
Scores with ratio 4:1:1
Figure 5: mAP vs. scores. All models are from Table and the scores are computed with different weight ratios. When the ratio is equal to 1:1:6, the correlation between mAp and score is well fitted.
In Table we tune the different arrangements of multi-scale weights in wide range. Seven multi- scale weight ratios are used to search different models, and all models are trained on the COCO dataset with FCOS and 3X learning schedule. Table shows that if the same weights arranged to C3-C5, the performance of ZenDet on COCO is worse than ResNet-50. Considering the importance of C5 (discussed in Section 4.2 increase the weight of C5, and ZenDet's performance continues
14

---

Under review as a conference paper at ICLR 2022
to improve. To further explore the correlations between mAP and scores, ratios to calculate the different scores of each model, along with the single-scale weight ratio of 0:0:1. The correlations between mAP and different scores are plotted in Figure Taking the results in Table X and Figure 5 we confirm the ratio of 1:1:6 may be good enough for the current FPN structure.
the seven weight
E DISCUSSION ABOUT AUXILIARY COMPONENTS
Fi (0.1)
(0-1)
Xm (0.1)
ConvK1-32
Global pooling
ConvK1-64 W-N((01)
(-2((0.1)
Residual link
FC
ConvK3-32
ReLU
BN-64
FC
1-64
Sigmoid
scale
Yout - (wu)
Xaut N (0, oouc)
(a) BN
(b) Residual link
(c) SE
Figure 6: Build the basic structure of auxiliary components with 64 channel input by standard Gaussian initialization. The resolution of inpput is 192 x 192. ConvK1-64 means the convolution with kernel size and output channel 64.
In this section, we explain why auxiliary components are removed during the search and plugged in back for training. Common used three auxiliary components are built in Figure 6 to verify effects on feature variance.
BN
BN is a widely used method to re-centre and re-scale the feature to make the network con- verge faster and more stable. Figure 6 (a). BN normalize entropy adaptively to the network width (which can be related to output variance). When BN is presented, networks of different widths will have the same entropy value. Hence, BN must be removed when calculating the entropy.
Residual link Building the structure like Figure 6 (b), the variances with or without residual link are 767 or 751 respectively. The residual link brings less than 2% difference in entropy score. The results means that residual link has little effect on entropy, so it could be removed during the search.
SE SE module is used to adaptively recalibrate channel-wise feature responses by explicitly modelling interdependencies between channels Hu et al. (2018). Note that the input is zero-mean distribution in our computation of the entropy. If SE module is used in the network, the output after global pooling is equal to O. So, the final output of SE module is equal to 0.5, which loses the ability to model interdependencies between channels.
To sum up, removing these auxiliary components makes our method stable and applicable to most single-branc branch feed-forward networks.
F COMPARISON WITH ZERO-SHOT PROXIES FOR IMAGE CLASSIFICATION
We compare ZenDet with architectures designed by zero-shot proxies for image classification in previous works, including SyncFlow (Tanaka et al., 2020). NASWOT (Mellor et al. 202T), Zen- NAS (Lin et al. 202II. For fair comparison, All methods use the same search space, FLOPs bud- get 91 G. searching strategy and training schedule. All searched backbones are trained on COCO with the FCOS head and 3X training from seratch. The results are reported in Table. 9
Among these methods, SyncFlow and NASWOT perform worse than 5e on COCO albeit they show competitive performance in image classification tasks. Zen-NAS achieves competitive performance over ResNet-50. The ZenDet outperforms Zen-NAS by +1.3% mAP with slightly fewer FLOPs and nearly one third of parameters.
15

---

Under review as a conference paper at ICLR 2022
Table Different zero-shot proxies COCO with FCOS. All methods the search space, FLOPs budget, searching strategy and training schedule.
FLOPs Backbone
Params Backhone
Proxy
AP
APs
AP
APr
R50
84G
23.5M
38.0
23.2
40.8
47.6
SyncFlow
90G
67.4M
35.6
21.8
38.1
44.8
NASWOT
88G
28.1M
36.7
23.1
38.8
45.9
Zen-NAS
9IG
67.9M
38.1
23.2
40.5
48.1
ZenDet
89G
25.8M
39.4
23.7
42.3
G VISUALIZATION OF SEARCHING PROCESS
(a) Iterations vs. #Layers
(b) Iterations vs. FLOPs
(c) IterationsP vs. Parameters
Figure 7: Visualization of single-scale entropy searching process. #layer is the number of each block of different levels.
45
(a) Iterations vs. #layers
(h) Iterations vs. FL.OPs
(c) IterationsP vs. Parameters
Figure 8: Visualization of multi-scale entropy searching process (Coarse). #layer is the number of each block of different levels.
96c
58c
10
14
12
20
20
30
00
10
20
(a) Iterations vs. #Layers
(b) Iterations vs. FLOPs
(c) IterationsP Parameters
Figure 9: Visualization of single-scale entropy searching process (Coarse-to-Fine). #layer is the number of each block of different levels.
The visualizations of searching process different entropy shown in Figure 7 g We compare the change layers, FLOPs and parameters during different iterations. Visualizations prove assumptions in the main body of the paper.
16

---

Under review as a conference paper at ICLR 2022
H DETAIL STRUCTURE OF ZENDETS
We list detail structure Table
The 'block' column is the block type. 'Conv' is the standard convolution layer followed by BN and RELU. 'ResBlock' is the residual bottleneck block used in ResNet-50 and is stacked by two Blocks in our design. "kernel' is the kernel size of kxk convolution layer in each block. 'in', "out' and 'bottleneck' are numbers of input channels, output channels and bottleneck channels respectively. 'stride'' the stride of current block. '# layers' the number of duplication of current block type.
Table 10: Architecture of single-scale entropy score with coarse mutation in Table,
block
kernel
in
out
stride
bottleneck
# layers
level
Conv 3
96
2
C1
ResBlock 96
208
2
32
C2
ResBlock
208
560
2
56
C3
ResBlock
560
1264
2
112
C4
ResBlock
1264
1712
224
C5
ResBlock
1712
2048
C5
ResBlock
2048
2048
Table 11: Architecture of multi-scale entropy score with coarse mutation in Table.
block
kernel
in
out
stride
bottleneck
layers
level
Conv
3
32
2
Cl
ResBlock
32
128
24
C2
ResBlock
128
72
C3
ResBlock
512
1632
112
C4
ResBlock 1632
2048
2
216
C5
Table 12: Architecture of multi-scale entropy score with coarse-to-fine mutation in Table. ZenDet-M architecture in Table.
block
kernel
in
stride
bottleneck
layers
level
Conv 64
3
64
2
Ci
ResBlock
120
64
C2
ResBlock
120
512
72
C3
ResBlock
512
1632
2
112
C4
ResBlock
1632
2048
2
184
C5
17

---

Under review as a conference paper at ICLR 2022
Table 13: ZenDet-S in Table. D
block
kernel
in
stride
bottleneck
layers
level
Conv
3
32
2
C1
ResBlock
32
48
32
C2
ResBlock
48
272
120
C3
ResBlock
272
1024
80
ResBlock
1024
2048
240
C5
Table 14: Zen in Table.
block kernel
in
out
stride
bottleneck
# layers
level
Conv
3
80
ResBlock
80
144
2
80
C2
ResBlock
144
608
2
88
C3
ResBlock
608
1912
2
136
C4
ResBlock
1912
2400
2
220
C5
Table 15: Initial structure in the search
block kernel
in
out
stride
bottleneck
layers
level
Conv
3
64
Ci
ResBlock
64
64
C2
ResBlock
256
512
128
C3
ResBlock
512
1024
2
256
C4
ResBlock
1024
2048
2
$12
C5
18


Reviews:
[{"review_id": "PDnkUU7lqQV", "reviewer": "Reviewer_34Uv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Saving lots of time and memory for searching architecture for a competitive backbone.\n2. Archiving SOTA performance in effective benchmark datasets, especially good performance on COCO 2017 dataset.\n3. Benefit from Multi-Scale Entropy Prior, ZenDet can adapt different sizes of objects, and results of training from scratch.\n4. In segmentation tasks, ZenDet also outperforms well.\n\nWeakness:\n1. No experiments for comparision of SOTA method in Instance Segmentation and results of training from scratch.\n2. Existing some spelling mistakes，such as retianet->retinanet.\n\n\nWhat are the failure modes of the proposed msep method?", "summary_of_the_paper": "Detection backbones usually cost a lot. With a novel zero-shot NAS method proposed in this work, named ZenDet, detection tasks can free from the heavy models, time, and resources to archive SOTA performance.", "summary_of_the_review": "see main review", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Lm-9UyVxeBd", "reviewer": "Reviewer_KNoQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Here are some strengths and weaknesses for this submission:\n\n**Strengths:**\n\n+ This paper is clearly written and easy to follow.\n+ The proposed method should be fairly easy to implement, and it is appreciated that the authors promise to open source their implementation if the paper is eventually accepted.\n+ The paper focuses on a very important aspect in real-world deployment of NAS algorithms: the search cost, and provides an effective solution.\n+ The experiments are done mainly on large-scale datasets.\n\n**Weaknesses:**\n\nI. Methodology\n\n+ The paper seems to be largely based on Zen-NAS. I feel that only Section 4.2 of this paper is novel and it seems that the effectiveness is not well justified:\n    - How do you choose the weights $\\alpha$? \n    - It seems that such choice impact the final performance a lot. A bad set of $\\alpha$ cannot even outperform manually designed R50. What is the theoretical implication / insight of the current parameter choice?\n+ In general the paper mainly justifies their results on accuracy-based ablation studies and comparisons with previous methods. But for me I'm more interested the effectiveness of multi-scale Zen-Score itself.\n    - I did not see any ablation analysis studying the correlation between Zen-Score and accuracy of different networks within the design space. \n    - I'm also curious whether multi-scale Zen-Score can improve the correlation comparing with single-scale Zen-Score.\n    - It will be great if there are some comparisons between ZenDet and one-shot NAS methods in terms of rank preserving.\n+ It is well-known that searching the architecture of FPN can also potentially improve the efficienty-accuracy tradeoff of object detectors. I'm curious whether the Zen-Det approach can also be utilized to search for complex connections in the FPN architecture. \n+ It'll also be great if the authors can have a small discussion on whether Zen-Det can be applied to recent transformer-based detectors.\n+ Removing auxiliary layers such as SE during search and adding them back during training/inference seem to be ad-hoc and not well justified.\n\nII. Experiments\n\n+ The paper is focused on the ResNet design space, which is shown to be efficient on desktop GPUs. However, it seems that results on edge GPUs or CPUs are missing. This makes the experiment part of this paper rather weak comparing with existing work (ZenNAS).\n+ Some comparisons are not quite fair / clear in the experiment section:\n    - Table 4 claims that the Zen-NAS performance is 38.1 mAP, but in Table 5 it seems that the single-scale version of ZenDet (which I think is exactly Zen-NAS) has 38.8 mAP. This makes me a little bit confused.\n    - The comparison between DetNAS and Zen-Det does not seem to be completely fair. The design space is different. Would it be possible to supplement some results on their design space and justify the effectiveness of ZenDet? \n+ The optimization target for this paper is FLOPs, but I think targeting latency will be better (and I don't think the change of optimization target will significantly influence the current algorithm pipeline).\n    - A natural question related to latency as the optimization target: how well will simpler zero-shot NAS metric work? For example, given a latency budget, if we just randomly select networks whose FLOPs are higher than one certain threshold, will the result be competitive comparing with ZenDet? It will be interesting to present the results on both desktop GPUs and edge devices.", "summary_of_the_paper": "This paper proposes a zero-shot neural architecture search approach for backbone design in object detection. The idea is to compute an entropy-based multi-scale Zen-score and use the score as an objective for evolutionary architecture search. The paper achieves better results comparing with previous zero-shot NAS approaches on object detection and greatly reduces the search time of conventional NAS approaches while maintaining similar accuracy.", "summary_of_the_review": "Overall I think this paper presents promising preliminary results on zero-shot neural architecture search for object detection. However, the novelty seems to be limited (comparing with Zen-NAS) and the experiments don't seem to be thorough enough. These downsides prevent me from recommending acceptance at the current stage. However, I'll be glad to raise my score if my concerns are addressed properly during the author response period.\n\n====\n\nPost-rebuttal update: I decided to raise the score to 6 based on the empirical results presented in this paper. I still think the novelty comparing with Zen-NAS is not that large and the challenges of the current approach (e.g. cannot deal with complex connections in the FPN, basically can only search for relatively simple backbones) prevents me from assigning an even higher score to this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Gfq_WKjSxWN", "reviewer": "Reviewer_YY18", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The proposed method is simple and efficient. This method reduces the search cost largely and speeds up searching. \n- This paper is well written and readable.\n\nWeaknesses\n- In the experiments, the authors valid the effectiveness of the proposed method on the ResNet Block search space. Could you provide some experiments on other search spaces to validate the generalization of this method?\n- During the search stage, residual link, BN and SE are all removed. To my knowledge, these components have some effects on feature variance. Thus, the entropy score based on feature variance can not reflect the performance of networks accurately when removing these components. Further, the network searched by this method may be suboptimal. Could you explain why removing these components?\n- In this paper, the smallest backbones searched by this method is ZenDet-S. Due to the limit of edge devices, the detectors with ZenDet-S can not run fast. This raise a question: can this method search some networks for edge devices?\n- From Table 5, comparing the row1 and row2, the metric with single-scale score does not outperform the metric with multi-scale score largely. This demonstrates the proposed method does not deeply entangle with detection task. Thus, could you provide an experiment on NAS-Bench-201?\n- In Figure 2, there are little details about the search space. Could you provide more details about the search space?", "summary_of_the_paper": "In this paper, the authors propose a zero-shot NAS to search backbone for detection task. Specifically, this method uses the differential entropy of output features as a metric to measure the performance of architecture on detection task. With the differential entropy , this method does not need training network parameters, reducing the search cost largely. Also, the backbones searched by this method have the state-of-the-art performance on detection task.", "summary_of_the_review": "I appreciate the contribution of this paper, but I prefer to reject this paper for the above weaknesses. If the authors solve my concerns, I would like to raise my rate.\n\n==Final Decision==\\\nThe author's feedback has resolved my concerns except the ranking. Though the author did not resolve ranking concern, I decide to raise score to 6. I hope the authors would provide the results about how the multi-scale weight ratio affect the correlation for more heads in the final paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received scores of 6,6,6 after the reviewers succeeded in making two authors raise their scores from 5 to 6. However, even after this, none of the reviewers actively argued for the paper. The only positive point raised in the private discussion was that the results are strong. (However, there is still the question of how much of this was due to the different design space used.)
Negative points raised in the private discussion included that 
- despite the authors clarification on the differences to Zen-NAS, the difference is perceived not to be large.
- there is no theoretical foundation behind the selection of a critical parameter, and this directly limits the applicability of ZenDet in searching for FPN connections. 
- as a paper focused on detection NAS, a limitation to only search for the backbone may not be novel enough for publication at ICLR. 

Overall, I agree with this criticism and weakly recommend rejection.
Decision: Reject