Multi-scale Network Architecture Search for Object Detection
Authors: Yuxin Yue
Quanquan Li
Yujie Wang
Venue:  (2021)
Keywords: Object Detection
Neural Architecture Search

Abstract: Many commonly-used detection frameworks aim to handle the multi-scale object detection problem. The input image is always encoded to multi-scale features and objects grouped by scale range are assigned to the corresponding features. However, the design of multi-scale feature production is quite hand-crafted or partially automatic. In this paper, we show that more possible architectures of encoder network and different strategies of feature utilization can lead to superior performance. Specifically, we propose an efficient and effective multi-scale network architecture search method (MSNAS) to improve multi-scale object detection by jointly optimizing network stride search of the encoder and appropriate feature selection for detection heads. We demonstrate the effectiveness of the method on COCO dataset and obtain a remarkable performance gain with respect to the original Feature Pyramid Networks.

Under review as a conference paper at ICLR 2021
MULTI-SCALE NETWORK ARCHITECTURE SEARCH FOR OBJЕCT DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Many commonly-used detection frameworks aim handle the multi-scale object detection problem. The input image is always encoded to multi-scale features and objects grouped by scale range are assigned to the corresponding features. How- ever, the design of multi-scale feature production is quite hand-crafted or partially automatic. In this paper, we show that more possible architectures of encoder network and different strategies of feature utilization can lead to superior perfor- mance. Specifically, we propose an efficient and effective multi-scale network architecture search method (MSNAS) to improve multi-scale object detection by jointly optimizing network stride search of the encoder and appropriate feature selection for detection heads. We demonstrate the effectiveness of the method on COCO dataset and obtain a remarkable performance gain with respect to the original Feature Pyramid Networks.
INTRODUCTION
Recognizing and localizing objects at vastly different scales fundamental challenge in object detection. Detection performance for objects with different scales highly related features with different properties such as feature resolution, receptive fields, and feature fusion ways. The key to solving the multi-scale problem in object detection is how to build multi-scale network that has proper high-level semantic features for objects with different scales.
recent work in object detection Feature Pyramid (etNorks(FPN) (Lin al. 2017) has achieved markable success in multi-scale feature design and has been commonly used by many modern object detectors (He et 2017 Lin et al. 2020 Lu al.) 2019). FPN extracts multi-scale intermediate features from the encoder network and assigns objects grouped by scales to corresponding features according to a heuristic rule. Another prevalent detection framework, SSD (Liu et al. 2016), con- ducts feature generation by lighter encoder network without upsampling operators. The basic idea to deal with the multi-scale detection problem can be summarized as below. Given the input image, series of feature maps with the various resolution generated to detect objects grouped by scale range. We note it as multi-scale feature production. In FPN and its variants, the multi-scale feature production is split into steps, feature generation and feature utilization. In terms of feature gen- eration, an encoder network composed of blocks provides features with different scales. And the strategy of feature utilization determines the rule of assigning objects feature maps. These two steps are closely related to each other.
Although FPN has achieved promising results on multi-scale object detection tasks, the production of multi-scale features is quite hand-crafted and relies heavily on the experiences of human experts. architecture
More specifically, network architectures FPN are based on a which may not be effective enough. By changing the downsampling and upsampling operation's 's positions and numbers, we could obtain many other candidates to generate different multi-scale features. Also, the predefined rule of feature utilization is very empirical and other alternatives may lead to better performance. Therefore we wonder: Can we find network architectures that can build better semantic feature representation for multiple scales? The answer is yes.
Recent advances in neural architecture search have shown promising results compared with hand- crafted architecture by human experts (Zoph et al.] 2018 Liu et al. 2019b; Cai et al.) [019 Guo jet al., 2019). Several works have also focused on neura architecture search in object detection tasks (Chen al. Ghiasi
Ghiasi et al. 2019 Du al. 2019), but generating and utilizing multi-scale

---

Under review as a conference paper at ICLR 2021
Encoder Network
Detection
Heads
(a) FPN
(b) MSNAS
Figure I: Architecture ResNetl8-FPP and the searched network MSNAS-RI8. MSNAS-RIE has different stride values of blocks in the encoding network and a more flexible feature utilization strategy. For simplification, P6 of FPN is not included the figure and only three detection heads presented.
features are still not well explored. DetNAS (Chen et al. 2019) adopts the method mainly designed on image classification to search the operations of backbone networks in object detectors. NAS- FPN (Ghiasi et al.) 2019) focuses on searching for better feature-fusion connections in the neck part of FPN. NAS-FPN doesn't optimize the whole encoder network and still relies on predefined backbone architecture. Recently SpineNet (Du et al. 2019) proposes a search method with scale- permuted features and cross-scale connections by reinforcement learning, but the search cost is quite large. All these previous works focus on designing better neural network architectures to generate better features given fixed feature selection strategy. However, they fail to conduct complete flexible multi-scale feature production strategy.
In this paper, we propose networks with the strong and proper multi-scale feature production strategy by neural architecture search. For feature generation, we put forward a network stride search method to generate multiple feature representations for different scales. Different from the seale-decressinn-incrasss tecture of FPN, the scale of our networks can decrease or increase at each block, illustrated in Figure | By stride search for each block, we could significantly explore a wide range of possible feature generation designs
method to take into account of both aspects and build detection
archi-
reltiireeooltion networks. Most backbones of object detectors are originally designed on image classification without multi-scale problems. However, stride configu- ration in the encoder network would be optimized in the context of the multi-scale task. Moreover, more complex cross-scale feature fusions might appear according to changes. For feature utilization, we change the previous one-to-one mapping strategy into a more flexible feature selection. Since each group with objects of the same scale range head, feature utilization implemented by selecting proper features for detection heads. Objects of different scale ranges might be assigned to the same feature map. It is not possible in previous methods, as shown in Figure ((b)
complex internal scale
detection
By jointly optimizing feature generation and utilization of multi-scale features, we search for flexible but complete multi-scale feature production strategies. Extensive experiments demonstrate complete multi-scale feature production search critical building strong and proper semantic features for object detection with different scales. On challenging COCO dataset (Lin et al. 2014), our method obtains 2.6%, 1.5%, 1.2% mAP improvement with similar FLOPs as ResNet 8-NN, ResNet34. FPN, ResNet50-FPN.
2 RELATED WORK
Neural Architecture Search Neural Architecture Search aims to design better network architec- tures automatically. RL-based methods (Zoph al. 2018 Zoph & Le. (017) have achieved great success despite a huge computation cost. In differentiable algorithms (Liu et al.) [0196]] Caa et al.] 2019), architecture parameters employed and operators the search space considered as
2

---

Under review as a conference paper at ICLR 2021
[log:(viw//bsee_ccrm
N
stride
Head1
k<2
Head2
2 <
16
Head3
35k<
32
Head4
4
64
Head5
k>
(a) Super-net and one of the paths of feature generation
(h) One example of feature utiliza- tion
Figure 2: Figure (a) shows the basic architecture of stride selection in the super-net for feature generation. The path represents one of its sub-architectures. The colored nodes are corresponding output features of the encoder network. Figure (b) illustrates the feature utilization search. The formula in Figure (b) represents the rule to assign Rols multi-scale features in FPN. We use different ranges of distinguish different object groups as well as their detection heads. The solid lines show example of feature utilization strategies. The dotted lines imply each detection head can select any the output features.
the weighted sum of candidate operators. There exist difficulties to deal with operators with dif- ferent strides. Super-nets, acting the collection of weights shared by all the sub-architectures, and evolutionary search are involved in one-shot NAS (Guo et al. 2019); Bender et al., 2018). But it's difficult to ensure strong correlations between one-shot and stand-alone performances of the sub-architectures.
Multi-scale Object Detection SSD (Liu al. al. 2016) uses multi-scale features generated by different stages of the backbone network to detect objects of different scales. Feature pyramid architectures utilized in FPN (Lin al. 2017) and RetinaNet (Lin et al. 2020) obtain multi-scale features. SNIP (Singh & Davis 2018) includes the image pyramid architecture to deal with multi-scale de- tection. Frameworks with multi-scal features are prevalent as objects of different scales appear in one image.
Neural Architecture Search on Object Detection DetNAS (Cen et al. 2019). NAS-FCOS (Wang et al. (2019) and Auto-FPN (Xu et al. 2019) focus on the architecture of the top-down pathway and feature fusion. SM-NAS Yao et al, 2019) and CR-NAS (Liang al.. 2020) try to adjust the compu- tation occupied by differentt parts of detectors. Also, there are several works aiming to improve FPN using NAS. NAS-FPN (Ghiasi et al. 2019) improves detection performance by searching for better connections within the feature pyramid network. It is limited without modification to the overall encoder architecture. And it fails to take the multi-scale feature utilization into account. Efficient- Det (Tan al. 2019) and Auto-FPN (Xu et al. 2019) search better feature fusion for FPN with differentiable methods. Other relative works like Liu et al. (2019a) conduct similar modifications. Recently, SpineNet (Du et al. 2019) proposes backbone search method with scale-permuted fea- tures and cross-scale connections by reinforcement learning. Our work has several major differences from it. First, the search space of MSNAS is designed that each operator in the network can down- sample upsample instead of permutation, which builds a much larger search space than Setne Second, the complete multi-scale feature production is considered in our work, while SpineNet only focuses on the architecture of the encoder network. Lastly, our method is based the one-shot search method instead of reinforcement learning in SpineNet. Our method is much more efficient and requires much less computation cost than SpineNet.
3 METHOD
We start by discussing multi-scale feature production for the object detection network in Section 3.1 In Section[3.2] we will introduce how to build the search space and search proper stride detection networks to obtain better multi-scale features. In Section 3.3 how to search the appropriate strategy
3

---

Under review as a conference paper at ICLR 2021
of feature utilization is presented. Finally, details about super-net training and search strategy described in Section 3.4
3.1 FEATURE PRODUCTION FOR DETECTION NETWORK
In this section, we'll discuss the production of multi-scale features and define the problem in detail. As noted above, the basic idea of handling the multi-scale detection problem can be summarized as feature production. In feature production, the imput image is first encoded into a series of feature maps with various resolutions. 'hen the objects are detected based on the features according to their scales. One method is to produce the feature for each scale range by one neural network, like the featurized image pyramid discussed in Lin et al. (2017). Variants include utilizing the image pyramid. like SNIP (Singh & Davis 2018). However, we employ only one neural network and obtain multi-scale features from intermediate features of the network. Because deep neural networks are experts in encoding the image into features. And they are considered to be able to encode information of different scales into features with different resolutions. Then we face the problem of how to utilize these features since there are N features available for K object groups. Therefore, it is reasonable split feature production into feature generation and feature utilization. To be more specific, the problem of multi-scale feature production can be defined as Equation I When we divide the problem into feature generation and feature utilization, as in Equation and Equation 3 & is approximated by g f.
á R 3X + {R HiH. }K
(1)
f 3XWW UIR H.XWixH IN 9: (RR IR= IN H..... 1x
(2)
(3)
And it is likely that only optimizing feature generation, as previous works do, is not optimal. So instead of optimizing feature generation for all the feature utilization strategies, we jointly optimize feature generation and utilization as a whole.
3.2 FEATURE GENERATION
Resolution of feature maps in one network changes with downsampling or upsampling operators. The network architectures of FPN follow the downsampling-upsaee uupammpiigsagni style, Figure ] shows. By encouraging a more flexible design of the scale-changing operation's positions and numbers, we could obtain many more promising candidates to generate better multi-scale features. We implement that by searching the stride of each block in the network.
Search space By deconstructing and generalizing the prevalent feature pyramid architecture, the basic search space is built as a varideevariable straight structure. A super-net of MSNAS with the depth of N consists of N mixed-blocks MB, M B2, ..., Bx. . For each mixed-blie blockk, three possi- ble strides are provided, i.e. 0.5, 1, and 2. The block whose stride equals to 0.5 is implemented as an upsampling block with an interpolation operator followed by a convolution to double the width and height of the feature map. Blocks that don't 't change the resolution of the feature map are referred to as normal blocks. The resolution of the feature output by one mixed-block could be twice, half or the same the input feature, as illustrated in Figure ((a) Considering operators with different strides within a mixed-block and the variation of sizes output from different operators in one mixed-block, the super-net is designed as a path-wise structure like Guo al. (2019). One path in the super-net is treated as valid if none of the blocks output feature larger than a quarter or smaller than 1/64 of the input image. Invalid paths are removed either in the training process of super-net or the sampling during the evolutionary search.
Lateral connections Lateral connections are built according current sub-architecture as Fig- ure [hows. One additional 1 Xx lateral convolution attached after every mix-block is available for latter cross-block connections. In scale-decreasing architectures, blocks can be grouped by res- olution of output features. Each group is notated as one stage. Similarly, we refer stage to group of adjacent blocks with the same output resolution, e-g. one downsampling or upsampling block and following normal blocks. The feature map of the last mix-block at one stage is merged with the lateral feature by element-wise addition as Equation Equation 6 shows.
34 M M, lal.
4

---

Under review as a conference paper at ICLR 2021
lat; Lot,ralConu,,((rl)) (stride:41 # 1) or (i N -]
(5)
0,
otherwise
where ri max (fk
II R<
stride, = 1) and (stridex+1 + 1)})
(6)
where lat; means the lateral connection of block i to combine with. If block i is the last block of a stage or the end of the encoder network as described Equation lat, is generated by lateral convolution LateralConori. Among blocks with output feature of the same resolution as block i, block ri is the nearest one at different stage. It can be regarded as an extensive version of lateral connections in the original FPN structure.
3.3 FEATURE UTILIZATION
In most multi-scale detection frameworks, objects are assigned to feature maps according to their scales given predefined strategy. In this section, we will discuss how to build the search space of feature utilization with respect to the feature generation network. Basically, objects are split into G groups and there exists one detection head for each group. feature utilization strategy could be simplified by selecting the resolution of feature maps from generated multi-scale features for each detection head. Figure 2 shows one example of feature utilization. Three feature maps of different resolutions are available. In this case, objects in various scale ranges might be assigned to the same feature map. This very different from previous predefined strategies. The dotted lines represent possibilityy of connecting to features of any resolution provided by the encoder network.
When searching for feature utilization, the exploration to obtain better features object groups is implemented within a lessened search space. For convenience, several constraints designed for more efficient search, as Equation 7 shows.
8 S:11 0A <i<
mins S < 8
maxx $
mins A max
Let s be an array with length of G. G equals to the number of object groups as described above. The ith item in s, noted as s;, represents the selected size with respect the input image of the feature for the corresponding object group. For example, s (4, 8, 16, 32, 64) is the configuration counterpart of FPN. s is assumed as a monotonic sequential based on insights to assign multi-scale objects. That is to say, smaller objects are considered to be assigned to finertresolution features, while larger ones are more compatible with coarser ones. Besides, the degraded pyramid structures are excluded in the super-net. We expect to focus more on hierarchical architectures and avoid extreme memory consumption of some special architectures.
3.4 SUPER-NET TRAINING AND SEARCH STRATEGY
It's difficult to combine features with different resolutions by element-wise addition, one-shot based search strategies show great compatibility with our search space. During training the super- net, one valid path, fulfilling all the requirements in Section 3.2 and Section 3.3 is randomly sam- pled to optimize weights in the super-net. Inspired by ZZhng et al. (2020). we treat the super-net as good pre-trained model. A better rank could be obtained within few iterations of individual fine-tuning, although the primitive weights in the super-net don't 't perform well in terms of ranking random samples. Fine-tuning for each architecture individually for few iterations not only mod- ulates the globally-optimized shared weights towards more personalized weights but also modifies the statisties of batch normalization operators. And the additional computation cost marginal in the entire pipeline.
Evolutionary search is adopted after the super-net training as Algorithm shows. The function GetV Meid&anddom.Smmppeeii
returns valid random samples described in the last paragraph. The evolution process starts from population with size P. Variation operations are performed on both the encoder and stride of heads' selected features. In Algorithm CrossouerEncodee means doing crossover concerning the stride values in the encoder network and Crossover Feature Stride means doing crossover concerning the selected stride values of utilization. Mulation Encoder
5

---

Under review as a conference paper at ICLR 2021
and Mutation FeatureStuurree have similar meanings about doing mutation. Given that the set of valid children within computation constraints not continuous with respect to crossover and mutation, not enough children could be generated in some iterations and the search process would terminate at a local optimum of the search space. Like Liu et al. (2020), a random set of new children from R attempts of valid samples with various computations appended as the proposals of population. In this way, both exploitation and exploration in the search space are encouraged to be conducted.
Algorithm 1: Evolution Process
Input: population size P. total evolution iteration T, max variation attempts M, attempts of random children R, Constraints C, return top samples k
Output: the top architectures with the best one-shot performances that meet both the validity requirements and computation constraints pope == GetV alidRanndmSaampe(())
for i 1: I do
pop; := V:
// Generate children by crossover and mutation
j == 0:
while j < M and poppl S do
children == Crossover (popt- U ErwcaaiionEnooeerreeerrrre U
Crossover wtrdeaarrredepppppp..... ) U M SetaattonFeauueeSSrriieeee
children == Select Select(children, (children, C);
pop: = pop: U children;
j= + 1;
end
10
// Add random children random children random_childrenn:: == GetV SanidaaandooSappllee R); random sardlommchildren:-e == Select trandonnchilliie C);; pop: = pop4 U random randomn_hiiddren children;
11
12
13
14
pop; = Topk(pop, pop4-1; P):
15 end
16 return Tpppy
EXPERIMENTS
Experiments presented in the following sections. Section 4.1 describes the implementation de- tails. Section 4.2 shows the main results of MSNAS along with their FPN baselines. Ablation experiments are conducted and discussed in Section 4.3 Finally, the performance of MSNAS and comparison with other methods are included in Section 4.4
4.1 IMPLEMENTATION DETAILS
Dataset COCO (Lin et al. 2014) is one of the commonly used dataset for object detection and instance segmentation. It contains training set with around 1 118K images, a validation set with around 5K images, and test-dev set with about 20k images. The annotations cover 80 categories of common objects.
Super-net training details We train our super-net and retrain the best architectures in the same settings. An input image is resized so that the shorter side is no more than 800 and the longer side less than 1333, then both sides will be fulfilled by padding to be divided by 64. The models are trained from scratch for 4x-long time with batch size of 32 images 16 GPUs. The learning rate initialized as 0.00125 and increases to 0.04 after a warm-up epoch. Then is divided by 10 the 42nd and 47th epoch. The weight decay is set to le-4 and the momentum is 0.9. Each architecture sample is fine-tuned for several iterations( 100 iterations) with a batch size of 32 at a learning rate of 0.004 before testing and evaluation. The evolutionary search process is repeated for 20 iterations. The population size is 50 and 50 children are generated to update the population in each iteration. Only those with computation within 1% gap of the target FLOPss will be considered valid

---

Under review as a conference paper at ICLR 2021
Table 1: Experimental results with respect their FPN baselines.
Baseline
Baseline
FLOPs
MSNAS
Mean
Var
Max
Architectures
mAP
(Eneoder+RPN)
(Ours)
mAP
mAP
mAP
R18-FPN
34.6
145.45
MSNAS-RI8
36,94
0.03
37.2
R34-FPN
37.7
182.20
MSNAS-R34
38,86
0.0784
39.2
R50-FPN
38.3
197.33
MSNAS-R50
39.3
0.028
39.5
children. As we focus on the design of the encoder network and multi-scale feature utilization, only the computations of encoder and RPN head are involved during the search. So in [ the FLOPs of the first stage of the network are used. For better comparison, we report our results with FLOPs of the entire network, as shown in Since the real image imput size differs from image to image as indicated in the previous part, fxxe approximate input size is used when computing FLOPs of the architectures. In both [6 and [ the approximate value for architectures with 800 1333 input is set to x32 1280 and that for networks with 600 1000 input is 576x 1024. About 10 individuals will be appended to the population randomly in every iteration. Finally, five of the top samples after the search procedure are retrained compute the statistics.
Detection network details Following He et al. (2019), the b-box head at the second stage originally composed of two fully-connected layers is replaced by a structure with two convolutions and one fully-connected layer. We adopt synchronized batch normalization to both the encoder and the b- box head. Blocks with different strides share the same number of channels, which we adjust to get proper distribution of computation in the search space. An ideal search space includes a large proportion of architectures with similar FLOPs as the target FL.OPs. Following the principle above, the numbers of channels for MSNAS-RU R18, MSNAS-R3A and MSNAS-R50 are set to 180, 160, 144 respectively.
4.2 RESULTS
Table shows the main results of MSNAS comparing with FPN counterparts. As we can see in the table, best searched architectures of MSNAS achieve mAP at 37.2%, 39.2%, 39.5% the compara- ble computation with MAANNet1--FPN(34..O
ResNet34-FPN(37.25) 77)) and DesNNe55-FFN(38..3 with a remarkable improvement of 2.6%. 1.5%, 1.2% mAP gain. Moreover, the best samples in all the experiments outperform the sanually-designed baseline networks average with relatively small variance. In particular, the performance of the best sample of MSNAS-R is comparable with ResNet34-FPN, while the computation of the former one 20% less than that of the latter one. Also, the maximum of mAP of top samples in the search space of MSNAS-R34 is superior over ResNe150-FPN.
Computation cost The super-nets are trained with 4x-schedule for around 30 hours on 16 GPUs. And the evolutionary search stage costs around 3 hours per iteration and about 60hours in total. Then around 90 hours are spent to search the optimal architectures. It could be further improved if better schedules and strategies for training detectors from scratch are proposed.
4.3 ABLATION STUDY
Effectiveness of feature utilization and feature generation search To verify the effectiveness of feature utilization. we conduct experiments compare fixed predefined feature selection and our proposed search-based feature selection. For fixed predefined FPN-style feature utilization, all sub- networks in the search space extract feature maps following the same strategies as FPN. Results are shown in Table 2 Feature utilization search in MSNAS shows large improvement compared with the pre-defined feature selection way in the original FPN. By comparing performances of ResNetl8. -0.%%MMP performance gain is obtained by search- ing stride for encoder network. We find that the correlation between the one-shot performances and stand-alone samples is weaker for the experiment in FPN-style feature utilization. according to Kendall's 's tau listed Table 2 We infer that the reason is the discontinuity among paths inside the super-net intensifies.
FPN and FPN-style searched architectures,
7

---

Under review as a conference paper at ICLR 2021
Table 2: Ablation experiments of pre-defined FPN-style and searched feature utilization.
Encoder
Feature Utilization
Mean mAP
Var mAP
Max mAP
Kendall's 's tau
ResNet18
FPN
34.6
Searched
FPN-style
34.98
0.061
35.3
-0.2247
Searched
Searched
36.94
0.03
37.2
0.4495
Table 3: Ablation experiments of stride range constraints Stride Constraints yes Mcan mAP Var mAP
Max mAP
36.94
0.03
37.2
no
36.98
0.107
37.4
Feature utilization search space constraints Several constraints of selected features' resolution are
applied when searching feature utilization, described in Section 3.3 Table 3 shows experiment results with or without stride range constraints. It can be found that although adding constraints cannot achieve much gain of performance but can reduce the variations of performance. Besides, the evolutionary process converges faster with constraints and the variance of one-shot performances the population is reduced if constraints are added, as shown Figure
Fine-tuning strategy In Table 4 Kendall's taus are computed by the one-shot performances from super-net after fine-tuning and the stand-alone performances of ten random samples with the same computation as the target FLOPs. It can be easily observed that the ranks have better performance after fine-tuning several iterations in both MSNAS-RI8 and MSNAS-R34. MSNAS-R18 R88 with fine- tuning achieves +0.3 mAP average performance and MSNAS-R34 obtains +0.6 mAP gain at both the maximal and average performance of top-5 samples.
Random children search strategy As noted in Section 3.4 several random children added to
improvement in MSNAS-RA can be observed by including random children. At the same time, the average one-shot performances of top-5 samples increase by more than 0.1 mAP, which is relatively remarkable among low values of one-shot performances.
the population for better exploration. According to Table 5
4.4 COMPARING WITH OTHER METHODS
In Table 6 comparison with other algorithms is conducted. An outstanding performance of 40.7%mAP is achieved by MSNAS-RS5 R50. The network of ASNNA--55O-MaskRCCN R50 are trained for 6x-long order get comparable performance with that of 2x-schedule with pre-trained models. R50-FPN-Faster R- R-CNN(heavy head) is trained for 6x-long with SyncBN and the bounding-box head at the second stage follows the 4conv-1fc format as noted in He et al. 2019). In order to perform a more fair comparison, reproduced NAS-FPN. It is trained with
and MSNAS-
0.25
0.1
0.22
0.2
0.20
0.18
0.15
0.24
0.12
12.0
12.5
15.0
175
20.2
23.0 35.0
12.5
15.0
17.5
euccution berat
evclution beration
(a) Evolutionary process with constraints
(b) Evolutionary process without constraints
Figure 3: Comparison of the evolutionary process in the ablation of constraints. It's easy to observe that the evolutionary process converges faster with constraints.
8

---

Under review as a conference paper at ICLR 2021
Table 4: Ablation experiments of fine-tuning strategy
Network
Fine-tuning
Kendall's s tau Mean mAP
Var mAP
Max mAP
MSNAS-RI8
yes 0.4495
37.1
0.008
37.2
MSNAS-RI8
no -0.0899
36.84
0.1024
37.2
MSNAS-RAS
yes
0.5683
38.86
0.0784
39.2
MSNAS-R34
no 0.2501
38,28
0.0936
38.6
Table 5: Ablation experiments of random children strategy
Network
Random Children One-shot mAP
Mean mAP
Var mAP
Max mAP
MSNAS-RI8
yes
27.12
37.1
0.008
37.2
MSNAS-RIE
27.31
36.94
0.03
37.2
MSNAS-R50
yes
25.99
39.3
0.028
39.5
MSNAS-RE no
25.85
38.84
0.0064
38.9
weights pre-trained on ImageNet for 2x-schedule. We can see that MSNAS-R50 has an advantage over 155-NAS-FPN((7222 at comparable computation.
CONCLUSION
By analyzing the commonly-used detection framework FPN, we find critical generate better multi-scale features and select proper features for detection heads. Considering the fact that multi- scale feature production plays an important role object detection, we propose a one-shot-based method to efficiently search a complete multi-scale feature generation strategy in the generalized detection architecture. Instead of only modifying network architecture for feature generation, we jointly optimize feature generation and feature utilization. The searched architectures achieve outstanding performance compared with the state-of-the- algorithms. More exploration and im- provement could be carried out by further works.
REFERENCES
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Under- standing and simplifying one-shot architecture search. In ICML, volume 80 of Proceedings of Machine Learning Research, pp. 549-558. PMLR, 2018
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search target task and hardware. In ICLR (Poster). OpenReview.net, 2019.
Yukang Chen, Tong Yang. Xiangyu Zhang, Gaofeng Meng, Xinyu Xiao, and Jian Sun. Det- nas: Backbone search for object detection. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Buc, Emily B. Fox, and Roman Garnett (eds.),
Table 6: Comparison with state-of-the the-arrt art algorithms. For details, please refer to the appendix.
Model
Encoder
Input Size
FLOPs
mAP
Faster R-CNN
ResNet50-FPN
800x 1333
207.6G
36.8
Faster R-CNN
ResNet101-FPN
500x1333
290.4G
39.1
Faster R- R-CNN(heavy head)
ResNet50-EPN
800x 1333
322.8G
39.1
DetNAS (Chen et al. 2019
800x 1333
42.0
NAS-FPN R50(7 256)
ResNet50-NAS-FI
800x800
562.5G
41.2
NAS-FPN R50(7 @ 256)
GesNet50-NAS-
600x 1000
317.9G
39.7
CR-NAS (Liang et al. 2020
CR-ResNetl01
1333
40.2
MSNAS MAAAS--Msk-RCCA
MSNAS-R50'
800x 1333
308.4G
40.7
MSNAS-R50
800x 1333
376.8G
41. 4box(box)
37.5(mask)

---

Under review as a conference paper at ICLR 2021
Advances in Neural Information Processing Systems 32: Annual Conference Neu- ral Information Processing Systems 2019. NeurIPS 2019. 8-14 December 2019, Vancou- wwwpaaperssiipscooooo.om.m nww..cccpppc...c paper/ detecttont
ver, BC, Canada, pp. 6638 6648. 2019. URL http://
search-
Xianzhi Du, Tsung-Yi, Lin, Pengchong Jin, Golnaz Ghiasi, Mingxing Tan, Yin Cui, Quoc Le, and Xiaodan Song. Spinenet: Learning scale-permutee backbone for recognition and localization. CoRR, 0bss/912.000027 2019. htw.h/pp://axx..oooor////////22.
Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid ar- chitecture for object detection. In CVPR, pp. 7036-7045. Computer Vision Foundation / IEEE, 2019.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. CoRR, abs/ abs/1904.00420, 2019.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, pp- 2980-2988. IEEE Computer Society, 2017.
Kaiming He. Ross B. Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In ICCV, pp- 4917-4923 IEEE, 2019.
Feng Liang, Chen Lin, Ronghao Guo, Ming Sun, Wei Wu, Junjie Yan, and Wanli Ouyang. Compu- tation reallocation for object detection. In ICLR. OpenReview.net, 2020.
Tsung-Yt Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects context. In ECCV (5), volume 8693 of Lecture Notes in Computer Science, pp. 740-755. Springer, 2014.
Tsung-Yi Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge Be- longie. Feature pyramid networks for object detection. In CVPR, pp. 936-944. IEEE Computer Society, 2017.
Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. IEEE Trans. Paltern Anal. Mach. Intell., 42(2):318-333 2020.
Chenxi Liu, Liang- Chich Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L. Yuille, and Fei-Fei Li. Auto-decplab: Hicrarchical neural architecture search for semantic image segmentation. nition, CVPR 2019, Long Beach, CA, USA, June 16-20, 20/9, pp. 82-92. Computer Vision Foundation IEEE, 2019a. doi: 10.1109/CVPR.2099.00011 //openaccess. cow.ses....ccccccccocooooooo com wttpe/ccco//ooteeeecccee.. Cconteent_VPR__211//aaaaa Auto DeepLab Hierarchical l fo for-rmanttor__Geatiiii Lec_manttc_mmae_______e___
In IEEE Conference on Computer Vision and Pattern Recog-
URL http:
Secmentation Cirieentttioo__ccRRRRR _019__aperrr______p__p_____p____ html
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019b. URL http:ss//ppeneeeeee.etttteeeeeee
S1 S1eYHoC5FX
Hanxiao Liu, Andrew Brock, Karen Simonyan, and Quoc V. Le. Evolving normalization-activaii layers. CoRR, obs/2000.00267,, 2020.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng- Yang Fu, and Alexander C. Berg. SSD: single shot multibox detector. In ECCV(1), volume 9905 of Lecture Notes in Computer Science, pp. 21-37. Springer, 2016.
Xin Lu, Buyu Li, Yuxin Yue, Quanquan Li, and Junjie Yan. Grid R-CNN. In CVPR, pp. 7363-7372 Computer Vision Foundation IEEE, 2019.
10

---

Under review as a conference paper at ICLR 2021
Bharat Singh and Larry S. Davis. An analysis of scale invariance in object detection SNIP. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 3578-3587. IEEE Computer Society, 2018. doi: 10.1 109//0R/ 2018.00377. URL ht http:///ppppccccsscc
/011//thmm/ thecvf..c.cccc com/ cttpt/nnnnttttreeaaaeeeee 8/htm1/
Singh_An An An____nn_____aa__ Ana
ys of CVPR 201 2a__pp__211__pperrro_m ht.m
Mingxing Tan, Ruoming Pang, and Quoc V. Le. Efficientdet: Scalable and efficient object detection. CoRR, abs/1911.09070, 2019.
Ning Wang, Yang Gao, Hao Chen, Peng Wang, Zhi Tian, and Chunhua Shen. neural architecture search for object detection. CoRR, 19111006644233 2019.
NAS-FCOS: fast
Hang Xu, Lewei Yao, Zhenguo Li, Xiaodan Liang, and Wei Zhang. Auto-fpn: Automatic network architecture adaptation for object detection beyond classification. In ICCV, pp. 6648 6657. IEEE, 2019.
Lewei Yao, Hang Xu, Wei Zhang, Xiaodan Liang, and Zhenguo Li. SM-NAS: structural-to-modular neural architecture search for object detection. CoRR, abs/1911 htn//991.00999, 09929, 2019.
Yuge Zhang, Zejun Lin, Junyang Jiang, Quanlu Zhang, Yujing Wang, Hui Xue, Chen Zhang, and Yaming Yang. Deeper insights into weight sharing in neural architecture search. CoRR, abs/2001.01431, 2020.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR. OpenReview.net, 2017.
Barret Zoph. Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition. In CVPR, pp. 8697-8710. IEEE Computer Society, 2018.
A APPENDIX
A.1 SELECTED STRIDE OF ENCODER NETWORK AND FEATURE STRIDE OF DETECTION HEADS
We present the architecture and performance of top-5 samples of MSNAS-R18, MSNAS-R34, MSNAS-R50 in Table 7 Table g and Table 9 The stride of blocks in the encoder network could be (.Spupaamlle). 1, and 2. The searched architectures do not follow the downsample-upsan style of FPN. Upsampling blocks are more possible to be observed around the middle parts of the networks.
Also, different features of detection heads are selected using MSNAS. Features with absolute stride of 4 and 64 are not utilized for detection heads in most of the best samples.
Table 7: Architecture of top5 samples of MSNAS- R18
Stride Blocks the Encoder
Selected Feature Stride
Performance
2, 2, 0.5, 1, 0.5,, I, 0.5 0.5, 0.5, 0,, 0,5,,
8, 8, 16, 16, 32
36.7
8, 8, 16, 16, 32
36.8
1,
8,8. 16, 16, 16
37.0
8, 8, 16, 16, 32
37.0
8, 8, 16, 16, 32
37.2
11

---

Under review as a conference paper at ICLR 2021
Table 8: Architecture of top5 samples of MSNAS-R34
Stride of Blocks in the Encoder
Selected Feature Stride
Performance
1,,, 0.5, 1, 1, 0,5, 0.5, 0.5, 1,,.
4, 8,8, 16, 32, 32
39.0
4, 16, 32, 64
38.7
12212212211110011111111111110111 12212211222211111111111201111111 12212222111000031111111111122121
4, 16, 16, 64
38.4
4, 16, 32, 32
39.2
4, 32,
39.0
Table 9: Architecture of top5 samples of MSNAS-R50
Stride of Blocks in the Encoder
Selected Feature Stride
Performance
1,2, 1,2, 1, I, 0.5, 0.5, 0.5, 0.5, 2, 1,2
8, 8, 16, 32, 32
39.4
112.1221111111111211000000.0 0,5,, 0.5,
1,2,2
8, 16, 32, 32, 32
39.1
1,2, 12222211111111111000011111111111
8, 3,,, 32, 64
39.5
12222211111111122000101111111113 1,
8, 16, 32, 64
39.4
122121112211130011122222220.0001
32, 32, 32
39.1
12


Reviews:
[{"review_id": "jVcwNQp7MCw", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper argues searching for both encoder and anchor assignment (feature utilization) in a unified NAS framework would lead to better detector performance. Experiments have been conducted on COCO dataset to show proposed method outperform baseline FPN by a significant margin, and both searching for better encoder and feature utilization benefit object detection task.\n\nPros:\n\n- dealing with objects with different scale is a fundamental problem in detection, and research in this direction would benefit vision community.\n\n- Most NAS work on detection focus on the backbone feature extractor (encoder). This paper brings new perspective for NAS on object detection task.\n\n- The experimental results back authors' claim that searching for feature utilization brings more performance gains than searching for encoder alone.\n\nCons:\n\n- The proposed method for searching a path throw super-net across multiple stride is not new. E.g.  \"Auto-DeepLab:\nHierarchical Neural Architecture Search for Semantic Image Segmentation\" has adopted similar techniques.\n\n- While searching for feature utilization looks novel, from Figure 2 b) it is effectively searching for how to select and fuse features with different resolution to each detector head. Assigning objects with different scales has been explored by works such as \"Scale-Aware Trident Networks for Object Detection\" / \"Feature Selective Anchor-Free Module for Single-Shot Object Detection\" in a non-NAS setting, which might be interesting to compare the proposed method with. Also the proposed approach is essentially similar to works that search for FPN connections, e.g. \"NAS-FCOS: Fast Neural Architecture Search for Object Detection\".\n\n- [minor] anchor-free object detection has growing more popularity in detection tasks, and thus might limit the importance of searching for feature utilization. \n\n- This paper is a bit challenging to follow and would benefit from careful proofreading. See comments below for more details.\n\nOther comments:\n\n- Abstract: \"we show that more possible architectures of encoder network and different strategies of feature utilization\". by \"more possible architectures\" does it mean \"larger search space\"?\n\n- Intro, paragraph 1 \"the key to solving\" -> \"the key to solve\"\n\n- Intro, paragraph 2 after introducing FPN / SSD and how they deal with multiple scales, \"The basic idea to deal with the multi-scale detection problem can be summarized as below\" and re-evaluate FPN architecture. this feels a bit redundant and doesn't read smoothly.\n\n- Intro, paragraph 3 \"Also, the predefined rule of feature utilization is very empirical and other alternatives may lead to better performance\". Agree with the statement and there has been quite a few work that tries to better align object scales with feature maps (like the papers mentioned above). Please cite and compare.\n\n- Related work. The method proposed in this paper is based on one-shot search and should be simpler than SpineNet. It would be great to have SpineNet in comparison (specially when SpineNet seem to show better performance)?\n\n- Sec 3.1: This section reiterate the motivation of this paper, which have been stated in Intro section and feels a bit redundant.\n\n- Sec 3.2, paragraph 2. \"Considering operators with different strides within a mixed-block and the variation of sizes output from different operators in one mixed-block\" this is very confusing. Does it indicate we have additional upscaling / downscaling operations within a mixed-block? How this related to searching through a path in a super-net?\n\n- Sec 3.2, Eq 3. Should it be $stride_{j+1} \\ne 1$?\n\n- Sec 3.3 last paragraph \"descrbied\" -> \"described\"\n\n- Sec 3.3. $s_i$ here indicates the selected feature size w.r.t. to image size (actually should be the inverse of it? otherwise it would be 1/4, ..., 1/64). It is also unclear how this setup would facilitate using multiple feature maps for one head.\n\n- Sec 3.4 \"It’s difficult to combine features with different resolutions by element-wise addition, so one-shot based search strategies show great compatibility with our search space.\" It is not straightforward to understand the connection between combining feature maps at different resolution, proposed scheme and one-shot search strategy. Please elabrate.\n\n- Sec 3.4 \" although the primitive weights in the super-net don’t perform well in terms of ranking random samples\". This is very confusing. Could authors clarify?\n\n- Sec 3.4 \"statics of batch norm...\" it should be \"statistics\"? \n\n- Sec 4: first paragraph. the experiment setup (dataset, splits) is mentioned in appendix but would be better to place it here.\n\n- Sec 4.1 first paragraph \"m=in\" what does this mean here?\n\n- Table 1: the table is a bit confusing. It compares ResNet-FPN with MSNAS-Resnet. It might be more clear to indicate the backbone network on each row and set the first column to be Resnet-FPN and others for MSNAS-Resnet\n\n- Table 1: Both baseline and proposed method share the same FLOPs. Is this the case? please clarify.\n \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "c2w17Glo78Z", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n\nThis submission works on the task of architecture search for object detection. The authors focus on two components: how to produce multi-scale features and how to use multi-scale features. The authors formalized a simple search space, and applied an evolution-based search algorithm. Experiments show the proposed searching algorithm is able to outperform the FPN baselines with various (small) backbones.\n\n#### Strengths\n\n- The task of multi-scale feature modeling in object detection is indeed important. \n\n- The proposed searching method is straightforward and easy to understand, and the performance compared to the baseline is OK. \n\n- Figure 1 is good. It clearly summarized the contribution.\n\n#### Weaknesses\n\n- A clear limitation is the authors did not compare to other NAS-based FPN as discussed in the related work section. Only NAS-FPN is listed in Table. 6, and it outperforms the proposed method (despite heavier computation). In my opinion, the most important comparison is to BiFPN, which also has similar FLOPS as FPN and performs 2-4 mAP higher according to EfficientDet paper. \n\n- The second major complaint is the scalability. The authors only show backbones up to Res 50, but not stronger ones. For example, ResNeXt101-DCN is used for reporting the state-of-the-art number in many detection paper. Ideally, the author should show their method works on this large backbone to really push the state-of-the-art. If the computation is a problem (The author already used 16 GPUs, which is more than the \"standard budget\" in object detection frameworks (8 GPUs)), then it's a limitation to the method.\n\n- The authors claimed changing the backbone encoder as an advantage of the proposed method. I would argue that this also slightly makes the proposed method less practical: using a fixed backbone can better utilize the pretrained weights, and can better use the community progress in designing backbone networks. \n\n- While the overall idea is straightforward, it also makes the paper lack excitement. While I am not working on NAS, the paper makes me feel it just applies a NAS recipe to multi-scale feature production (I can see designing the search space is a contribution). Please correct me if I am wrong.\n\n- (Minor) The authors only evaluate on two-stage Faster/ Mask RCNN, where the object level assign is less critical (e.g., PointRend assigns all objects to stride 4 level). I would suggest the author also try one-stage detectors, where the effectiveness of FPN is more pronounced (e.g., FCOS shows subtle differences in FPN level assign yield 1-2 mAP difference). Note that this is not a requirement for rebuttal. \n\n#### Summary\n\nThis paper applies a reasonable method to an important problem. The results are OK (decently outperforming the baseline) but not exciting (lack comparison to the state-of-the-art). Also, the computation requirement may limit further applying the proposed method on large backbones. In its current status, I suggest a rejection. In the rebuttal, if the authors clearly show the searched structure outperforms BiFPN and works on large backbones, I will improve my rating. \n\n\n#### After rebuttal\n\nQ1: Thanks for running the additional experiments. Unfortunately, the results are not strong enough to convince me to use the searched architecture instead of BiFPN. It will also be interesting to see how BiFPN works under similar FLOPs with the searched model (e.g., the structure of EfficientD4 or D3).\n\nQ2: Please do run the ResNeXt101-DCN experiments for the revision or next submission. These are critical to make people use the proposed method.\n\nQ3: True. However training from scratch requires 6x longer training time according to [1], and is considered as a drawback.\n\nQ4: Thank you for the clarification, this makes the contribution clearer. However my concerns on changing the backbone remains (Q3).\n\nQ5: Thank you for considering. I agree the ranks in the leaderboard is a main factor for design choice. This also highlight the importance of Q2. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "aftOPvWyd8K", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a network architecture search (NAS) suitable for a feature pyramid network (FPN) that provides notable detection accuracy for objects at every scale. Based on the decomposition of FPN structure as (multi-scale) feature generation and feature utilization, the proposed NAS offers a new design strategies for both components. For feature generation, NAS super-net is trained to find the optimal selection of whether to reduce, maintain, or extend feature resolution after each module. For feature utilitization, it defines conditions for efficiently selecting the optimal FPN architecture. The proposed MSNAS yields the better accuracy than its backbone FPN and other NAS methods on the COCO object detection benchmark dataset.\n\nI have a concern in technical novelty.\nThe only novel element considered in the proposed NAS method is to learn the stride value that controls the feature resolution at each module. The most contribution of this paper is very similar to FPN-NAS in that it re-designs FPN by choosing the optimal connections between all the modules. Furthermore, this search method does not retain the main claim of FPN, in which features at every level can be trained to have the same level of semantics.\n\nThe method used for Feature utilization is a very simple rule not relying on training to select the final feature layer to be connected to object detection heads. This simple rule can speed up training at the expense of marginal accuracy. This can be seen as part of engineering rather than technical contribution.\n\nI also have a concern in its presentation. \nAlgorithm 1 is very important for understanding how to implement the proposed method. However, it is very difficult to understand this algorithm due to missing definition of some terms such as CrossoverEncoder, MutationEncoder, CrossoverFeatureStride, MutationFeatureStride etc.", "extra_scores": {}, "extra_text": {}}, {"review_id": "VFp_2BEPAt", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a one-shot based multi-scale features ﻿generation and utilization framework for object detection. This method searches the ﻿network stride for features generation and detection heads location for features utilization.\n\n Pros:\nStride and detection heads location are important factors, and automated learning of them is desirable.\nThis method improves the performance based on several baselines. \nSearching the detection heads location is a new idea, to my knowledge.\n\nCons:\nHowever, there are some concerns about this paper.\n1.     This paper seems like an improved version of the SpineNet. Searching for ﻿strides instead of permutation and utilizing a one-shot method instead of reinforcement learning is not very novel.\n2.     If we treat this paper as the improved version of SpineNet, the improvement of the performance is strange. SpineNet improves the performance from 37 to 42.7 for ResNet50. Why jointly searching the backbone features generation and FPN features utilization only improves 1.2 mAP in this paper?\n3.     The comparison with other state-of-the-art methods is unfair. To my knowledge, the results for DetNas (42.0) and CR-NAS (40.2) are based on 1x schedule while this method is trained from scratch for 6x schedule.\n4. It would be good to evaluate on another dataset.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission got 3 rejection and 1 marginally below the threshold. In the original reviews, most of the concerns lie in the limited novelty, the inferior performance to some existing similar works and the limited scalability of the proposed method. Though authors provide some additional experiments, the reviewers still feel the experiments are not convincing and keep their ratings. AC agrees with the reviewers comments on this paper. Though achieving SOTA performance is not necessary for every submission, NAS-alike method is purely pursuing better performance (either higher accuracy or better efficiency). Thus, the performance is also important for evaluating a NAS paper. From the reviewers, the proposed method does not show better performance than some existing works, like BiFPN. This makes the value of the paper is not clear, in particular considering the method novelty is limited. The authors could consider to improve the submission in the experiments to better justify the proposed method, either achieving better performance or higher efficiency than existing works. At its current status, AC cannot make accept recommendation. 
Decision: Reject