Computation Reallocation for Object Detection
Authors: Feng Liang
Chen Lin
Ronghao Guo
Ming Sun
Wei Wu
Junjie Yan
Wanli Ouyang
Venue:  (2020)
Keywords: Neural Architecture Search
Object Detection

Abstract: The allocation of computation resources in the backbone is a crucial issue in object detection. However, classification allocation pattern is usually adopted directly to object detector, which is proved to be sub-optimal. In order to reallocate the engaged computation resources in a more efficient way, we present CR-NAS (Computation Reallocation Neural Architecture Search) that can learn computation reallocation strategies across different feature resolution and spatial position diectly on the target detection dataset. A two-level reallocation space is proposed for both stage and spatial reallocation. A novel hierarchical search procedure is adopted to cope with the complex search space. We apply CR-NAS to multiple backbones and achieve consistent improvements. Our CR-ResNet50 and CR-MobileNetV2 outperforms the baseline by 1.9% and 1.7% COCO AP respectively without any additional computation budget. The models discovered by CR-NAS can be equiped to other powerful detection neck/head and be easily transferred to other dataset, e.g. PASCAL VOC, and other vision tasks, e.g. instance segmentation. Our CR-NAS can be used as a plugin to improve the performance of various networks, which is demanding.

Published as a conference paper at ICLR 2020
COMPUTATION REALLOCATION FOR OBJECT DETECTION
Feng Liang' ,Chen Lin', Ronghao Guo', Ming Sun', Wei Wu', Junjie Yan', Wanli Ouyang² "Sensetime Research Group
(liangfeng, linchen, guoronghao, sunmingl, wuwei, haspes/eaaaaiieesssssssttteeee. "hhe University of Sydney wanli. ouy ang sygeeydneyyyn.oooomooooooannynne edu. au
ABSTRACT
The allocation of computation resources in the backbone is crucial issue in ob- ject detection. However, classification allocation pattern is usually adopted di- rectly object detector, which is proved be sub-optimal. In order reallocate the engaged computation resources in a more efficient way, we present CR-NAS (Computation Reallocation Neural Architecture Search) that can learn computa- tion reallocation strategies across different feature resolution and spatial position diectly on the target detection dataset. A two-level reallocation space is proposed for both stage and spatial reallocation. A novel hierarchical search procedure is adopted cope with the complex search space. We apply CR-NAS multi- ple backbones and achieve consistent improvements. Our CR-ResNe150 and CR- MobileNetV2 outperforms the baseline by 1.9% and 1.7% COCO AP respectively without any additional computation budget. The models discovered by CR-NAS can be equiped to other powerful detection neck/head and be easily transferred to other dataset, e.g. PASCAL VOC, and other vision tasks, e.g. instance segmenta- tion. Our CR-NAS can be used as a plugin to improve the performance of various networks, which is demanding.
INTRODUCTION
Object detection one of the fundamental tasks in computer vision. The backbone feature extractor is usually taken directly from classification literature (Girshick) 2015 Ren et al. 2013 Lin et al.) 2017a; [u et al.) 2019). However, comparing with classification, object detection aims to know not only what but also where the object is. Directly taking the backbone of classification network for object detectors is sub-optimal, which has been observed in Li et al. (2018). To address this issue, there many approaches either manually automatically modify the backbone network. Chen et al. (2019) proposes a neural architecture search (NAS) framework for detection backbone avoid expert efforts and design trails. However, previous works rely the prior knowledge for classification task, either inheriting the backbone for classification, or designing search space similar NAS on classification. This raises a natural question: How to design an effective backbone dedicated to detection tasks?
To answer this question, first draw link between the Effective Receptive Field (ERF) and the computation allocation of backbone. The ERF is only small Gaussian-like factor of theoretical receptive field (TRF), but dominates the output (Luo al. 2016). The ERF of image classification task can be easily fulfilled, e.g. the input size is 224x224 for the ImageNet data, while the ERF of object detection task need more capacities to handle scale variance across the instances, e.g. the input size is 800x 1333 and the sizes of objects vary from 32 to 800 for the COCO dataset. [in jet al. (2017a) allocates objects of different seales into different feature resolutions to capture the appropriate ERF in each stage. Here we conduct experiment to study the differences between the ERF of several FPN features. As shown in Figure П] we notice the allocation of computation across different resolutions has great impact the ERF. Furthermore, appropriate computation allocation across spacial position (Dai et al. 2017) boost the performance of detector by affecting the ERF.

---

Published as a conference paper at ICLR 2020
P3
P4
P5
ResNet0
FPN:
P3
P4
SCR-ResNat0
Figure Following the instructions in Luo et (016).. we draw the ERF of FPN in different resolution features. The size of base plate is 512x512 12, with respective anchor boxes ((64, 128, 256} for (Ps. P4. Ps3) drawn The classification CNNs ResNet50 tends have redundant ERF for high resolution features and limited ERF for low resolution features ps- After stage reallocation, our JCR-ResNethh has more balanced ERF across all resolutions which leads high performance.
Based on the above observation, in this paper, we aim to automatically design the computation allocation backbone for object detectors. Different from existing detection NAS works (Ghiasi et al. 2019 Ning Wang & Shen 2019) which achieve accuracy improvement by introducing higher computation complexity. we reallocate the engaged computation cost in a more efficient way. We propose computation reallocation NAS (CR-NAS) to search the allocation strategy directly the detection task. A two-level reallocation space is conducted to reallocate the computation across different resolution and spatial position. In stage level, search for the best strategy distribute the computation among different resolution. In operation level, we reallocate the computation by introducing powerful search space designed specially for object detection. The details about search space can be found in Sec. 3.2 We propose a hicrarchical search algorithm to cope with the complex search space. Typically in stage reallocation, we exploit reusable search space reduce stage-level searching cost and adapt different computational requirements.
Extensive experiments show the effectiveness of our approach. Our CR-NAS offers improvements for both fast mobile model and accurate model, such as ResNet (He al. 2016), MobileNetV2 (Sandler et al.) 2018). ResNeXt (Xie et al., 2017). On the COCO dataset, our CR-ResNet50 and CR-MobileNetV2 can achieve 38.3% and 33.9% 9% AP, outperforming the baseline by 1.9% and 1.7% respectively without any additional computation budget. Furthermore, transfer our CR-ResNet and CR-MobileNerr2 into the another ERF- sensitive task, instance segmentation, by using the Mask RCNN (He et al., 2017) framework. Our CR ResNet50 and CR-MobileNetV2 yields 1.3 3% and 1.2% COCO segmentation AP improvement over baseline.
To summarize, the contributions of our paper three-fold:
We propose computation reallocation NAS(CR-NAS) reallocate engaged computation resources. To our knowledge, we are the first to dig inside the computation allocation across different resolution.
We develop a two-level reallocation space and hierarchical search paradigm to cope with the complex search space. Typically in stage reallocation, we exploit reusable model to reduce stage-level searching cost and adapt different computational requirements.
Our CR-NAS offers significant improvements for various types of networks. The discov- ered models show great transferablity over other detection neck/head, e.g. NAS-FPN (Cai & Vasconcelos) 2018), other dataset, e.g. PASCAL VOC (Everingham et al. 2015 and other vision tasks, e.g. instance segmentation (He et al. 2017).
2

---

Published as a conference paper at ICLR 2020
2 RELATED WORK
Neural Architecture Sea)ch(NAS) Neural architecture search focus on automating the network architecture design which requires great expert knowledge and tremendous trails. Early NAS ap- proaches (Zoph & Le. 2016; Zoph et al. 2018) are computational expensive due to the evaluating of each candidate. Recently, weight sharing strategy (Pham al., 2018 Liu et al. 2018 Cai al, 2018 Guo et al.] 22019) is proposed to reduce searing cost. One-shot NAS method (Brock et al., 2017 Bender et 2018 Guo et al. 2019) build a directed acyelic graph G (a.k.a. supernet) to subsume all architectures in the search space and decouple the weights training with architectures searching. NAS works only search for operation in the certain layer. our work is different from them by searching for the computation allocation across different resolution. Computation alloca- tion across feature resolutions is an obvious issue that has not been studied by NAS. We carefully design search space that facilitates the use of existing search for finding good solution.
NAS on object detection. There are some work use NAS methods on object detection task (Chen et al. 2019: Ning Wang & Shen 2019: Ghiasi et al. 2019). Ghiasi et al. (2019) search for scalable feature pyramid architectures and Ning Wang & Shen (2019) search for feature pyramid network and the prediction heads together by fixing the architecture of backbone CNN. These two work both
introduce additional computation budget. The search space ofCnn et al. (2019) is directly inherited from the classification task which is suboptimal for object detection task. Peng al. (2019) search for dilated rate on channel level in the CNN backbone. These two approaches assume the fixed number of blocks in each resolution, while we search the number of blocks in each stage that is important for object detection and complementary to these approaches.
3 METHOD
3.1
BASIC SETTINGS
Our search method is based on the Faster RCNN (Ren et al. 2015) with FPN (Lin et al. 2017a) for its excellent performance. We only reallocate the computation within the backbone, while fix other components for fair comparison.
For more efficient search, we adopt the idea of one-shot NAS method (Brock et al. 2017 Bender et al. 2018) Guo et al. (019). In one-shot NAS, a directed acyclic graph G (a.k.a. supernet) is built subsume all architectures in the search space and is trained only once. Each architecture g is subgraph of G and can inherit weights from the trained supernet. For a specific subgraph E G, its corresponding network can be denoted N(g, w) with network weights w.
3.2
TWO-LEVEL ARCHITECTURE SEARCH SPACE
We propose Computation Reallocation NAS (CR-NAS) distribute the computation resources in two dimensions: stage allocation in different resolution, convolution allocation in spatial position.
3.2. STAGE REALLOCATION SPACE
The backbone aims to generate intermediate-leved features C with increasing downsampling rates 4x, 8x, 16x, and 32x, which can be regarded as 4 stages. The blocks in the same stage share the same spatial resolution. Note that the FLOPs of a single block in two adjacent spatial resolutions remain the same because downsampling/pooliig layer doubles the number of channels. So given the number of total blocks of a backbone N. we can reallocate the number of blocks for each stage while keep the total FLOPs the same. Figure 2 shows our stage reallocation space. In this search space, each stage contains several branches, and each branch has certain number of blocks. The numbers of blocks in different branches are different. corresponding to different computational budget for the stage. For example, there are 5 branches for the stage 1 in Figure 2 the numbers of blocks for these 5 branches are, respectively. 2, 3. 4, and 5. We consider the whole network as supernet 7 (Ti, T2, T3, T4 where T; at the ith stage has K; branches, i.e. T; [B = 1...K,). Then an allocation strategy can be represented as of blocks in the ith branch. All blocks in the same stage have the same structure. Σi= T2 = N for network with N blocks. For example, the original ResNet101 has 7 [3, 4, 23, 3 and N 33
[TI, T2,72, T3, 74], where т; denote the number
3

---

Published as a conference paper at ICLR 2020
stage
stage
Legend
Edge
Edge off
Block on
Block off
Figure 2:
Stage reallocation in different resolution. In supernet training, random sample choice each stage and optimize corresponding weights. reallocation searching, eligible strategies are evaluated according to the computation budget.
residual blocks. We make constraint that each stage at least has one convolutional block. We would like to find the best allocation strategy of ResNer101 is among the (+) possible choices. Since validating a single detection architecture requires hundreds of GPU-hours, it not realist to find the optimal architecture by human trails.
On the other hand, we would like to learn stage reallocation strategy for different computa- tion budgets simultaneously. Different applications require CNNs of different numbers of layers for achieving different latency requirements. This is why we have ReseNet18, ReseNet50, Re- seNet101, etc. We build search space cover all the candidate instances in certain series, e.g. ResNet series. After considering the trade off between granularity and range, we set the numbers of blocks for Tj and T2 21, 3, 4, 5, 6, 7, 87,88 9, 10), and set the numbers of blocks for T3 as 3,, 5, 6, 9, 11, 14, 17, 20, 23). for T3 as 32 4, 6, 7,9, 11, 13, 15, 17] for the ResNet series. The stage reallocation space MobileNetV2 (Sandler al. 2018 and ResNeXt (Xie al. 2017 can be found in Appendix A.2
3.2.2 CONVOLUTION REALLOCATION SPACE
To reallocate the computation across spatial position, we utilize dilated convolution Li et al. (2019), Li al. (2018). Dilated convolution effects the ERF by performing convolution sparsely sam- pled locations. Another good feature of dilated convolution is that dilation introduce no extra pa- rameter and computation. We define choice block to be basic unit which consists of multiple dilations and search for the best computation allocation. For ResNet Bottleneck, we modify the center convolution. For ResNet BasicBlock, we only modify the second 3 3 convolu- tion to reduce search space and searching time. We have three candidates in our operation set O: (dilated convolution x 3 with dilation rate ili = 1,2.3). Across the entire ResNet50 search space, there therefore 316 Rs 4 x 107 possible architectures.
3.3 HIERARCHICAL SEARCH FOR OBJECT DETECTION
We propose a hierarchical search procedure to cope with the complex reallocation space. Firstly, the stage space explored find the best computation allocation for different resolution. Then, the operation space explored to further improve the architecture with better spatial allocation.
3.3. STAGE REALLOCATION SEARCH
To reduce the side effect of weights coupling, we adopt the uniform sampling in supernet train- ing(a.k.a single-path one-shot) (Guo et al. 2019). After the supernet training, we can validate the allocation strategies 7 E directly on the task detection task. Model accuracy(COCO AP) is de- fined as (N (T) We set the block number constraint N. We find the best allocation strategy the following equation:
T" = arg max (r,
(1)
Ei T.=N
4

---

Published as a conference paper at ICLR 2020
Legend
Block
Block 2
Block
Block
Block
Block
Block on
J
Block off Block evalsattnng Block be sampled
Partial Architecture
Operations to be Sampled
Figure 3: Evaluation of choice in block operation search approach. As shown in figure, have partial architecture of block and block 2, and now we need to evaluate the performance of convolu- tion with dilated rate in the third block. We uniform sample the operation of rest blocks generate a temporary architecture and then evaluate the choice through several temporary architectures.
3.3.2 BLOCK OPERATION SEARCH
Algorithm 1:
Greedy operation search algorithm
Input:
Number of blocks B: Possible operations set of each blocks O = Supernet with trained weights (O, W*); Dataset for validation Deat; Evaluation metric APrai.
B);
Output: Best architecture
Initialize top K partial architecture p Ø
for i= 2,
do
Pextend = p x O;
> denotes Cartesian product
resull = (urch, AP) arch Pentend; AP evaluale(areh))
p choose lopK (result)
end
Output: Best architecture o choose tpp.
By introducing the operation allocation space in Sec. 3.2.2 we can reallocate the computation across spatial position. Same as stage reallocation search, we train an operation supernet adopting random sampling in each choice block (Guo al. 2019). For architecture search process, previous one-shot works use random search (Brock et al. 2017 Bender et al.) 2018) or evolutionary search (Guo et al. 2019). In our approach. We propose greedy algorithm make sequential decisions to obtain the final result. We decode network architecture In each choice step, the top K partial architectures are maintained shrink the search space. We evaluate each candidate operation from the first choice block to the last. The greedy operation search algorithm is shown in Algorithm
sequential of choices [01, o2. oB).
The syper-parameter K' is set equal to 3 in our experiment. We first extend the partial architecture in the first block choice which contains three partial architectures in Pextend- Then we expand the top partial architectures into the whole length B, which means that there are 3 x architectures in other block choice. For a specific partial architecture arch, we sample the operation of the unselected blocks uniformly for architectures where denotes mini batch number of D We validate each architecture on a mini batch and combine the results to generate iinluate(arch), We finally choose the best architecture obtain o*.
9 partial
EXPERIMENTS AND RESULTS
4.1 DATASET AND IMPLEMENTATION DETAILS
Dataset We evaluate method the challenging MS COCO benchmark (Lin et al. 2014). We split the 135K training images trainval135 into 130K images archtrain and 5K images archyal. First, train the supernet using archtrain and evaluate the architecture using archval. After the architec- ture is obtained, we follow other standard detectors (Ren et al.) 2013) Lin et al., (217a)) on using Im- ageNet (Russakovsky al. 2015) for pre-training the weights of this architecture. The final model
5

---

Published as a conference paper at ICLR 2020
Table 1:
Faster RCNN FPN detection performance on COCO minival for different backbones using our computation reallocation (denoted by 'CR-x'). FLOPs are measured on the whole detec- tor(w/o ROIAlign layer) using the input size 800 1088, which is the median of the input size on COCO.
Backbone
FLOPs(G)
AP
AP
AP75
AP,
APm
APt
MobileNetV2
121.1
32.2
54.0
33.6
18.1
34.9
42.1
CR-MobileNerv2
121.4
33.9
56.2
35.6
19.7
36.8
44.8
ResNet18
147.7
32.1
53.5
33.7
17.4
34.6
41.9
CR-ResNet18
147.6
33.8
55.8
35.4
18.2
36.2
45.8
ResNet50
192.5
36,4
58.6
38.7
21.8
39.7
47.2
CR-ResNet50
192.7
38.3
61.1
40.9
21.8
41.6
50.7
ResNet101
257.3
38.6
60.7
41.7
22.8
42.8
49.6
CR-ResNet101
257.5
40.2
62.7
43.0
22.7
43.9
54.2
Figure 4: Architecture sketches. From top bottom, they are baseline ResNet50, stage reallocation 4CR-ResNet50 and final CR-ResNet50.
fine-tuned on the whole COCO trainval135 and validated on COCO minival. Another detection dataset VOC (Everingham et al. 2015) is also used. We use VOC traimmail20077trriimoz2222 as vaildation dataset.
training dataset and VOC tes7
Implementation details The supernet training setting details can be found in Appendix A. For the training of our searched models, the input images are resized have short side of 800 pixels or a long side of 1333 pixels. We use stochastic gradient descent (SGD) as optimizer with 0.9 momentum and 0.0001 weight decay. For fair comparison, all our models are trained for epochs, known as 1 xx schedule [Girshick et al. (2018). We use multi-GPU training over 8 1080TI GPUs with total batch size 16. The initial learning rate is 0.00125 per image and is divided by at 8 and epochs. Warm-up and synchronized BatchNorm (SyncBN) (Peng et al. 2018) are adopted for both baselines and our searched models.
4.2
MAIN RESULTS
4.2.
COMPUTATION REALLOCATION PERFORMANCE
We denote the architecture using our computation reallocation by prefix 'CR-', e.g. CR-ResNer50. Our final architectures have the almost the same FLOPs the original network(the negligible dif- ference FLOPs is from the BatchNorm layer and activation layer). As shown in Table [ our CR-ResNet50 and CR-ResNet101 outperforms the baseline by 1.9% and 1.6% respectively. It is worth mentioning that many mile-stone backbone improvements also only has around 1.5% gain. For example, the gain is 1.5% from ResNet50 to ResNeXt50-32x4d addition, we run the baselines and searched models under longer 2x setting (results shown in Ap- pendix A.4). It can be concluded that the improvement from approach consistent.
indicated in Table 4 In

---

Published as a conference paper at ICLR 2020
Table 2: Faster RCNN FPN detection performance VOC les/2007. Our computation realloca- tion models are denoted by 'CR-x'
ResNet50
CR-ResNet50
ResNet101
CR-ResNet101
AP50
84.1
85.1
85.8
86.5
Table 3: Mask RCNN detection and instance segmentation performance on COCO minival for dif- ferent backhones using our computation reallocation (denoted by 'CR-x'). Box and Seg are the AP (%) the bounding box and segmentation results respectively.
Backbone
FLOPs
Seg
Seg.
Segm
Seg:
Box
Box,
Box,
Box
MobileNetV2
189.5
30.6
15.3
33.2
42.2
33.1
18.8
35.8
43.3
CR-MobileNetV2
189.8
31.8
16.3
34.3
44.1
34.6
19.9
37.3
45.7
ResNet50
261.2
33,9
17.4
37.3
46.6
37.6
21.8
41.2
48.9
CR-ResNet50
261.0
35.2
17.6
38.5
49.4
39.1
22.2
42.3
52.3
ResNet101
325.9
35.6
18.6
39.2
49.5
39.7
23.4
43.9
51.7
CR-ResNet101
325.8
36.7
19.4
40.0
52.0
41.5
24.2
45.2
55.7
Our CR-ResNet50 and CR-ResNet101 are especially effective for large objects(3.5%, 4.8% im- provement for APr). To understand these improvements, we depict the architecture sketches in Figure We can find in stage-level, our Stage CR-ResNer50 reallocate more capacity in deep stage. It reveals the fact that the budget in shallow stage is redundant while the resources in deep stage is limited. This pattern consistent with ERF as in Figure I In operation-level, dilated con- volution with large rates tends to appear in the deep stage. We explain the shallow stage needs more dense sampling to gather exact information while deep stage aims to recognize large object by more sparse sampling. The dilated convolutions in deep stage further explore the network potential to detect large objects, it is an adaptive way to balance the ERE For light backbone, our CR-ResNet18 and CR-MobileNetV2 both improves 1.7% AP over the baselines with all-round AP, AP; im- provements. For light network, it is more efficient way to allocate the limited capacity in the deep stage for the discriminative feature captured in the deep stage can benefit the shallow small object by the FPN top-down pathway.
4.2.2 TRANSFERABILIT VERIFICATION
Different dataset We transfer our searched model another object detection dataset VOC (Ev- eringham et al. (015). Training details can be found in Appendix (A.3] We denote the VOC metric mAP@0.5 as APso for consistency. As shown Table 2 our CR-ResNet50 and CR-ResNet101 achieves AP5o improvement 1.0% and 0.7% comparing with the already high baseline.
Different task Segmentation is another task that is highly sensitive to the ERF (Hamaguchi al. 2018 Wang et al. 2018). Therefore, we transfer our computation reallocation network into the instance segmentation task by using the Mask RCNN (He al. 2017) framework. The experimental results on COCO are shown in Table 3 The instance segmentation AP of our CR-MobileNetV2. CR-ResNet50 and CR-ResNet101 outperform the baseline respectively by 1.2%. 1.3% and 1.1% absolute AP. We also achieve bounding box AP improvement by 1.5%, 1.5% and 1.8% respectively.
Different head/neck Our work is orthogonal to other improvements on object detection. We exploit the SOTA detector Cascade Mask RCNN (Cai & Vasconcelos 2018) for further verification. The detector equipped with our 101 can achieve 44.5% AP, better than the regular Res101 43.3% baseline by significant 1.2% gain. Additionally, we evaluate replacing the original FPN with searched NAS-FPN (Ghiasi et al. 2019) neck to strength results. The Res50 with NAS- FPN neck can achieve 39.6% AP while our CR-Res50 with NAS-FPN can achieve 41.0% AP using the same x setting. More detailed results can be found in Appendix A.4

---

Published as a conference paper at ICLR 2020
Table 4: COCO minival AP (%) evaluating stage reallocation performance for different networks.
Res50 denotes ResNet50, similarly for Res ReX50 denotes ResNeXi50, similarly for ReXt101.
MbileNetV2
Res18
Res50
Res101
RcX50-32x4d
ReX101-32x4d
Baseline AP
32.2
32.1
36.4
38.6
37.9
40.6
Stage-CR AP
33.5
33.4
37.4
39.5
38.9
41.5
42.0
41.0
(77.3, 40.2)
40.0
38.0
39.0
36.0
SCR Backbonejstage
(76.5, 38.3)
34.0
SCR ResNet ResNext
FLOPs equivalent
R50 FLOPs
37.0
32.0
Mobile/ett2
R101 FLOPs
SCR MobileNeWe
R101 FLOPs (best)
30.0
36.9
100
120
140
160
180
200
220
240
260 280
75.0
75.5
76,0
76.5
77.0
77.5
78.0
FLOPs(G)
Topl Acc.
Figure 5: Detector FLOPs(G) versus AP COCO minival. The bold lines and dotted lines are the baselines and our stage computation real- location models(SCR-) respectively.
Figure 6: Topl accuracy ImageNet validation set versus AP on COCO ménival. Each dot is model which has equivalent FLOPs the base- line.
4.3
ANALYSIS
4.3.1
EFFECT OF STAGE REALLOCATION
Our design includes two parts, stage reallocation search and block operation search. In this section, analyse the effectiveness of stage reallocation search alone. Table 4 shows the performance comparison between the baseline and the baseline with our stage reallocation search. From light MobileNetV2 model heavy ResNeXt101, our stage reallocation brings a solid average 1.0% AP improvement. Figure 5 shows that our Stage-CR network series yield overall improvements over baselines with negligible difference in computation. The stage reallocation results for more models are shown in Appendix A.2 There is trend to reallocate the computation from shallow stage to deep stage. The intuitive explanation that reallocating more capacity in deep stage results balanced ERF as Figure I shows and can enhance the ability to detect medium and large object.
4.3.2 CORRELATIONS BETWEEN CLS. AND DET. PERFORMANCE
Often, a large AP increase could be obtained by simply replacing backbone with stronger network, e.g. from ResNet50 to ResNet101 and then to ResNeXt101. The assumption is that strong network can perform well on both classification and detection tasks. We further explore the performance correlation between these two tasks by a lot of experiments. We draw ImageNet top1 accuracy versus COCO AP correlation in 6 different architectures of the same FLOPS. Each dot single network architecture. We can easily find that although the performance correlation between these two tasks is basically positive, better classification accuracy may not always lead to better detection accuracy. This study further shows the gap between these two tasks.
CONCLUSION
In this paper, we present CR-NAS (Computation Reallocation Neural Architecture Search) that can learn computation reallocation strategies across different resolution and spatial position. We design
8

---

Published as a conference paper at ICLR 2020
two-level reallocation space and novel hierarchical search procedure cope with the complex search space. Extensive experiments show the effectiveness of our approach. The discovered model has great transfer-ability to other detection neck/head, other dataset and other vision tasks. Our CR- NAS can be used as plugin to other detection backbones to further booster the performance under certain computation resources.
REFERENCES
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understand- ing and simplifying one-shot architecture search. In International Conference on Machine Learn- ing. pp- 549-5444 558, 2018.
Andrew Brock, Theodore Lim, James Ritchie, and Nick Weston. Smash: one-shot model archi- tecture search through hypernetworks. arXiv preprint
2017.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search target task and hardware. arXiv preprint
2018.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn:
Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6154-6144 6162, 2018.
Yukang Chen, Tong Yang, Xiangyu Zhang, Gaofeng Meng, Chunhong Pan, and Jian Sun. Detnas: Neural architecture search on object detection. arXiv preprint arXiv: 10979, 2019.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 764-773, 2017.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1):8--13,, 2015.
Golnaz Ghiasi, Tsung- Yi Lin, and Quoc V Le. Nas-fpn: Learning scalable feature pyramid architec- ture for object detection. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 7036-7045, 2019.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 1440-1448, 2015.
Ross Girshick, Ilija Radosavovic, Georgia Gkioxari, Piotr Dollár, and Kaiming He. Detectron, 2018.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint arXiv: 100::1904.0022 2019.
Ryuhei Hamaguchi, Aito Fujita, Keisuke Nemoto, Tomoyuki Imaizumi, and Shuhei Hikosaka. Ef- fective use of dilated convolutions for segmenting small object instances in remote sensing im- agery. In 2018 JEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1442- 1450. IEEE, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference computer vision, pp. 2961-29969 2017.
Yanghao Li, Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Scale-aware trident networks for object detection. arXiv preprint arXiv:190..0189. 2019.
Zeming Li, Chao Peng, Gang Yu, Xiangyu Zhang, Yangdong Deng, and Jian Sun. Detnet: A backbone network for object detection. arXiv preprint 1 1804.06215, 2018.

---

Published as a conference paper at ICLR 2020
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017a.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017b.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: (801v 880.00955, 2018.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint a60iv:1608.mm (0t.::6600..0983,, 2016.
the Xin Lu, Buyu Li, Yuxin Yue, Quanquan Li, and Junjie Yan. Grid r-cnn. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7363-7372, 2019.
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. In Advances in neural information processing sys- tems, pp. 4898-4906, 2016.
Hao Chen Peng Wang Zhi Tian Ning Wang. Yang Gao and Chunbua Shen. Nas-fcos: Fast neural architecture search for object detection. arXiv preprint 1rXiv:1906.0440 2019.
Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, and Jian Sun. Megdet: A large mini-batch object detector. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6181-6189, 2018.
Junran Peng, Ming Sun, Zhaoxiang Zhang, Tieniu Tan, and Junjie Yan. Efficient neural architecture transformation searchin channel-leve for object detection. arXiv preprint arXiv: 1909:.99000000 1909.0299, 2019.
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv: 032608 2018
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3)211-255) 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510 2018
Panqu Wang, Pengfei Chen, Ye Yuan, Ding Liu, Zehua Huang, Xiaodi Hou, and Garrison Cot- trell. Understanding convolution for semantic segmentation. In 2018 IEEE winter conference on applications of computer vision (WACV), pp. 1451-1460, IEEE, 2018.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv: / 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018
10

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.I SUPERNET TRAINING
Both stage and operation supernets use exactly the same setting. The supernet training process adopt the pre-traae pre-trrinnng and fine-tuning' paradigm. For ResNet and ResNeXt, the supernet channel distribution is [32, 64, 1 128, 256].
Supernet pre-training. We use ImageNet-Ik for supernet pre-training. We use stochastic gradient descent (SGD) as optimizer with 0.9 momentum and 0.0001 weight decay. The supnet are trained for 150 epochs with the batch size 1024. To smooth the jittering in the training process, we adopt the cosine learning rate decay (Loshchilov & Hutter 2016) with the initial learning rate 0.4. Warming up and synchronized BN (Peng et al. 2018) are adopted to help convergence.
Supernet fine-tuning. We fine tune the pretrained supernet archtrain. The input images resized to have short side of 800 pixels or a long side 1333 pixels. We use stochastic gradient descent (SGD) optimizer with 0.9 momentum and 0.0001 weight decay. Supernet is trained for 25 epochs (known as 2x schedule (Girshick et al. 2018)). We use multi-GPU training over 8 1080TI GPUs with total batch size 16. The initial learning rate is 0.00125 per image and is divided by 10 at 16 and 22 epochs. Warm-up and synchronized BatchNorm (SyncBN) (Peng et al. 2018) are adopted help convergence.
A.2 REALLOCATION SETTINGS AND RESULTS
stage allocation space For ResNeXt, the stage allocation space exactly the same as ResNet series. For MobileNetV2, original block numbers in [Sandler et al. (2018) is defined by n=[1, 1, 2, 4, 3, 1, 1, 1]. We build our allocation space on the the bottleneck operator by fixing stem and tail components. architecture is represented as m 1,, m1, m2. m3. m4, ma, 1, The allocation space is M
[M, M2, M3, M4, Ms). M1 = {1, 2, 3, 4, 4,5 Ms = {3, 4,5, 6, 7]. M4, Mr = {2, 3, 4, 5, 6]. It's worth to mention the computation cost in different stage of is not exactly the same because of the abnormal channels. We format the weight as [1.5, 1, 0.75, 1.25) for [m1, m2 my, ma ms].
computation reallocation results We propose our CR-NAS in sequential way. At first we real- locate the computation across different resolution. The Stage CR results is shown in Table A.2
Table 5: Stage reallocation strategies of different networks. MV2 denotes MobileNetV2. Res18 denotes ResNet18, similarly for Res50, Res101. ReX50 denotes ResNeXt50-32x4d. similarly for ReXt101.
MV2
Res18
Res50
Res101
ReX50
ReX101
Baseline
[1,.
[2,2,2,2]
[3,4,6,3
[3,4,23,3]
[3,4,6,3]
[3,4,23.3]
Stage CR
11,
[1,1,2,4]
[1,3,5,7]
[2,3,17,11]
[2,2,6.6]
[3,4.15,11]
Then we search for the spatial allocation by adopting the dilated convolution with different rates. the operation code denote final model
Table 6: Final network architectures.
stage code
operation code
CR-MobileNat22
[1..222233.44.......
[0, 1,
0,
CR-ResNer18
[1,1,2,4]
[0. 0, 1, 0, 1, 0, 1]
CR-ResNet50
[1,3,5.7]
[0. 0, 1, 0, 0, 0, 2, 0. 011.000..1..111110000000000..0.0
1,,
CR-ReeNe1101
[2,3,17,11]
1. 0.1.0.1.......11111111.11
11

---

Published as a conference paper at ICLR 2020
[0] dilated conv with rate I(normal conv) [1] dilated conv with rate 2 [ dilated conv with rate
Our final model can be represnted as a series of allocation codes.
A.3 IMPLEMENTATION DETAILS OF VOC
We the VOC 1rainee/20077+rriii results on the VOC les/2007. The pretrained model is apoted. The input images are resized to have a short side of 600 pixels or long side of 1000 pixels. We use stochastic gradient descent (SGD) as optimizer with 0.9 momentum and 0.0001 weight decay. We train for 18 whole epochs for all models. We use multi-GPU training over 1080TI GPUs with total batch size 16. The initial learning rate is 0.00125 per image and is divided by 10 at 15 and 17 epochs. Warm-up and synchronized BatchNorm (SyneBN) (Peng al. 2018 are adopted help convergence.
server whole training We conduct our
A.4 MORE EXPERIMENTS
longer schedule 2x schedule means training totally 25 epochs indicated in Girshick et al. (2018. The initial learning rate is 0.00125 per image and divided by 10 at 16 and 22 epochs. Other training settings is exactly the same in 1x.
Table 7: Longer 2x Faster RCNN FPN detection performance COCO minival for different backbones using our computation reallocation (denoted by 'CR-x').
Backhone
FLOPs (G)
AP
AP50
AP75
AP,
APm m
AP;
ResNe150
192.5
37.6
59.5
40.6
22.4
40.8
48.5
CR-ResNet50
192.7
39.3
60.8
42.1
22.0
42.4
52.5
ResNer101
257.3
39.8
61.5
43.2
23.2
44.0
51.4
CR-ResNet101
257.5
41.2
62.5
44.6
23.7
44.8
54.6
Powerful detector The Cascade Mask RCNN (Cai Vasconcelos 2018 is a SOTA multi-stage object detector. The detector is trained for 20 epochs. The initial learning rate is 0.00125 per image and is divided by 10 at 16 and 19 epochs. Warming up and synchronized-BN (Peng et al. 2018) are adopted to help convergence.
Table 8: SOTA Cascade Mask RCNN detection performance COCO minival for ResNet101 and our 101
Detector
Bakebone
AP
AP50
AP75
APs
APm
AP1
Cascade Mask
Res101
43.3
61.5
47.3
24.7
46.6
57.6
Cascade Mask
CR-Res101
44.5
62.6
48.0
25.6
47.7
60.2
Powerful searched neck NAS-FPN GGiasi al. 2019) is powerful scalable feature pyramid architecture searched for object detection. We reimplement NAS-FPN (7 @ 384) Faster RCNN (The original paper is implemented in RetinaNet (Lin al. 20176)). The detector is training under 1x setting as described in
Table 9: Faster RCNN + NAS-FPN detection performance on COCO minival for ResNet50 and our CR-ResNet50.
Bakebone
Neck
AP
AP50
AP75
APs
APm
AP1
Res50
NAS-FPN (7 @ 384)
39.6
60.4
43.3
22.8
42.8
51.5
CR-Res50
NAS-FPN (7 @ 384)
41.0
61.2
44.2
22.7
44.9
54.2
12


Reviews:
[{"review_id": "SygeNOnwqB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper works on neural architecture search for object detection. Two search directions are proposed: 1) searching the number of conv blocks at each resolution (or \"stage\"). 2) searching the dilations for each conv block. A greedy neighbor-based search algorithm is adopted. The results show healthy improvements among different network architectures. And the searched architecture also performs well on other tasks or datasets. \n\nOverall it is a valid paper with reasonable ideas and decent results. I like the conclusion that the searched architecture also works well on other tasks. This can be a universal replacement of the regular Resnets if people are willing to switch. However, the results are not exciting enough. The baseline models are old and it is not surprising doing an architecture search can improve. It seems that the major improvements are from re-arranging the convolutional blocks (comparing Table. 4 and Table. 1), which is one of the most straightforward directions for architecture search. The improvements of adding dilation on earlier layers are not exciting. Also, the authors do not compare to any other neural architecture search methods, which makes the improvements less convincing. \n\nI vote for a weak rejection for now, mainly based on the limited novelty. A more interesting improvement will be (manually) comparing the searched architecture for different tasks. E.g., will all tasks prefer more layers in deeper stages or does classification prefer more layers in the middle, and segmentation prefers more layers in the beginning. I will be happy to alter my rating if the authors show more exciting observations (not limited to the above direction).\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkeuiUTw9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a neural architecture search method for computation resources allocation across feature resolutions in object detection. A two level reallocation space is proposed for both stage and spatial reallocation. The experiment results have quite nice improvements on several standard data sets.\n\nThis is a great, well written paper overall. The design and experiment settings are well described with details. In short, this is a perfect paper that I enjoy reading.\n\nI only have very small questions and suggestions to this paper.\n\nThe paper claimed the approach is able to reallocate the engaged computation resources in a more efficient way. If I did not missing anything, the paper only shows related experiments in figure 5 and figure 6 with corresponding descriptions in 4.3.2. I hope the author could have more details in these two figures with more analysis. Personally, I think more analysis on computational effectiveness may make the paper more attractive.\n\nWe all know that neural network training may not be very stable in some settings. One thing I am curious about in this paper is whether the output network architectures from different training are always the same. If they are not the same, can you compare the differences?\n\nI am also curious if the author could give some intuition of the network architecture of the final best network. In other words, we want to know why the final network is better than other networks. I read the Figure 4, Table 5 and Table 6, but I really cannot understand why those networks are that 'good'. Maybe, we can find some clues by answering the last paragraph.\n \nFollowing the last question, we also find out that the same NAS algorithm produces different networks on different data sets. Is it because of the data set settings, or because of the content of the data sets or because of network randomness? What are your intuitions?\n\nA detailed question in 3.2.2. Why you only modify the second 3x3 conv in ResNest BasicBlock and only modify the center 3x3 conv in ResNet Bottleneck.\n\nDoes the hyperparameter K in 3.3.2 mater a lot (like 4 or 5)?\n\nThe 4.2.2 \"transfer-ability verification\" is a very nice section. Do you  train NAS on VOC or only a fixed network architecture on VOC? If you did both, what is the performance difference?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJgu9zzXtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "<Strengths> \n+ This paper performs architecture search for object detection, especially for the computation allocation across different resolutions. It is a new application for NAS research. \n+ The proposed approach shows some marginal improvement of object detection accuracy across multiple backbones and datasets. \n+ This work proposes a new formulation to apply NAS approach to object detectors, including linking between the ERF and the computation allocation of backbone and two-level hierarchical search for stages and convolution operations.\n\n<Weakness>\n1. This paper can be regarded as an engineering work for a new domain of problem with little technical novelty. \n- From the perspective of NAS research, the proposed approach has little technical novelty; instead, it seems like an application of existing techniques (or even simpler ones) (e.g.  to a new domain of problem - object detection. \n- Given that the NAS is only applied to CNN backbones in this paper, the novelty (of proposing a new task) may be further weakened.\n- The novelty of this work over existing works of “NAS on detection” (Chen et al 2019, Wang et al 2019 and Ghiasi et al 2019) is not justified.\n\n2. Experimental results are rather weak. \n- This paper only focuses on showing that the proposed method marginally improves the performance of basic backbones (MobileNet V2 and ResNet 18/50/101). However, the improvement gaps are rather marginal (about 1.0% in average as shown in Fig.5).\n- This paper does not compare its performance with other SOTA detection methods but only compare with some baselines instead. However, the performances of baselines are too low; for example, in Table 1, the mAP of the ResNet101 baseline is 38.6, which is lower by about 10 than the SOTA detector with the same ResNet101+FRCNN+FPN. It may not be convincing to use the baselines that have more than 20% lower accuracy compared to SOTA and show only ~1.x improvement over it.\n- Given that the proposed approach is applicable to any backbones and detectors, more experiments should be done using recent stronger baselines (including many recent tricks like RoIAlign and DCN).\n- Another experimental weakness is lack of comparison with existing NAS methods on object detections such as (Chen et al 2019, Wang et al 2019 and Ghiasi et al 2019). Even though experiment settings here may be different from those of these papers,  they should be compared in any reasonable ways. Moreover, the detection accuracies reported in this paper are not as good as the numbers of these papers. This paper argues that the proposed approach is complementary to these methods, so it would be good to report the results of the combined model with them.\n\n3. The paper is written poorly.\n- There are many grammatically wrong and awkward expressions. The draft should be thoroughly proofread. \n\n<Conclusion>\nAlthough this work shows a new application of NAS for object detection, my initial decision is ‘weak reject’ mainly due to lack of technical novelty, limited experiments and poor writing. \n\n<Post-rebuttal comments>\nAuthors' responses partly resolved my concerns on the experiments. I have no object to accept this paper.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1x_OVc85S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper attempts to apply neural architecture search (NAS) to re-arrange, or re-allocate the network backbone blocks and the convolution filters for object detection. The search space is two-fold: 1) the network is allowed to search over allocation of different number of blocks in the backbone (e.g. ResNet, MobileNet); 2) the network is allowed to choose the dilation of each of the block. A one-shot NAS method is adopted for efficient search. After search, the model is shown to have 1) better AP results; and 2) more balanced effective receptive field (ERF). \n\n+ I am not aware of any work that performs search on backbone architectures for object detection yet. So the idea itself is novel;\n+ The visualization of the ERF is interesting -- it reals that ERF is more balanced after searching. \n\n- My biggest concern is in results. It seems for Faster R-CNN with FPN, the detection results should be higher in general (e.g. R-50-FPN should at least give ~37 Ap with 1x training, and can reach 38 if it trains longer --  the same as CR-R-50-FPN in Table 1). Therefore I am not fully convinced that the searched results are obtaining meaningful gains -- maybe a result that trains longer can help here.\n- Related -- I think while the idea is interesting, the limited improvement is hurting the significance of the work. In fact, to me the most important result would be on Fig 5, where it compares the speed/accuracy trade-off (directly comparing accuracy is meaning less unless the paper reaches state-of-the-art -- which is around 50 now); however, again no significant gains. Here it is because many \"improvements\" have been proposed after Faster/Mask R-CNN as baseline.\n- (Minor) I am not sure the computation of possible choices 33^3 is accurate for the search space, because some of these 33^3 blocks are identical and therefore redundant.\n- 4.3.2 is a bit misleading. At least improving backbone helps improves object detection performance (as far as I know), the plot also shows quite a bit of correlation between classification and detection performance -- please report at least the correlation for the points on the figure.\n\nQuestion: \n* From Fig 4, it seems for baseline R-50, it would be best to allocate more computation on the later stages (e.g. 1st one only has 3, and last has 7), Is it true for other search results? Is it true for other head (instead of FPN)? Are there intuitive explanations for that?\n* Also from Fig 4, it seems the network tend to favor more dilated convolutions toward the end? Does it have something to do with the ERF balancing?\n\nDespite the concerns, I am still in favor of accepting the paper, as the paper concerns both accuracy and speed for detection, and applying NAS to such kind of search reals some interesting patterns (please answer the questions above).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission applies architecture search to object detection architectures. The work is fairly incremental but the results are reasonable. After revision, the scores are 8, 6, 6, 3. The reviewer who gave "3" wrote after the authors' responses and revision that "Authors' responses partly resolved my concerns on the experiments. I have no object to accept this paper. [sic]". The AC recommends adopting the majority recommendation and accepting the paper.
Decision: Accept (Poster)