Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper describes a neural architecture search method for computation resources allocation across feature resolutions in object detection. A two level reallocation space is proposed for both stage and spatial reallocation. The experiment results have quite nice improvements on several standard data sets.

This is a great, well written paper overall. The design and experiment settings are well described with details. In short, this is a perfect paper that I enjoy reading.

I only have very small questions and suggestions to this paper.

The paper claimed the approach is able to reallocate the engaged computation resources in a more efficient way. If I did not missing anything, the paper only shows related experiments in figure 5 and figure 6 with corresponding descriptions in 4.3.2. I hope the author could have more details in these two figures with more analysis. Personally, I think more analysis on computational effectiveness may make the paper more attractive.

We all know that neural network training may not be very stable in some settings. One thing I am curious about in this paper is whether the output network architectures from different training are always the same. If they are not the same, can you compare the differences?

I am also curious if the author could give some intuition of the network architecture of the final best network. In other words, we want to know why the final network is better than other networks. I read the Figure 4, Table 5 and Table 6, but I really cannot understand why those networks are that 'good'. Maybe, we can find some clues by answering the last paragraph.
 
Following the last question, we also find out that the same NAS algorithm produces different networks on different data sets. Is it because of the data set settings, or because of the content of the data sets or because of network randomness? What are your intuitions?

A detailed question in 3.2.2. Why you only modify the second 3x3 conv in ResNest BasicBlock and only modify the center 3x3 conv in ResNet Bottleneck.

Does the hyperparameter K in 3.3.2 mater a lot (like 4 or 5)?

The 4.2.2 "transfer-ability verification" is a very nice section. Do you  train NAS on VOC or only a fixed network architecture on VOC? If you did both, what is the performance difference?
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper works on neural architecture search for object detection. Two search directions are proposed: 1) searching the number of conv blocks at each resolution (or "stage"). 2) searching the dilations for each conv block. A greedy neighbor-based search algorithm is adopted. The results show healthy improvements among different network architectures. And the searched architecture also performs well on other tasks or datasets. 

Overall it is a valid paper with reasonable ideas and decent results. I like the conclusion that the searched architecture also works well on other tasks. This can be a universal replacement of the regular Resnets if people are willing to switch. However, the results are not exciting enough. The baseline models are old and it is not surprising doing an architecture search can improve. It seems that the major improvements are from re-arranging the convolutional blocks (comparing Table. 4 and Table. 1), which is one of the most straightforward directions for architecture search. The improvements of adding dilation on earlier layers are not exciting. Also, the authors do not compare to any other neural architecture search methods, which makes the improvements less convincing. 

I vote for a weak rejection for now, mainly based on the limited novelty. A more interesting improvement will be (manually) comparing the searched architecture for different tasks. E.g., will all tasks prefer more layers in deeper stages or does classification prefer more layers in the middle, and segmentation prefers more layers in the beginning. I will be happy to alter my rating if the authors show more exciting observations (not limited to the above direction).

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The paper attempts to apply neural architecture search (NAS) to re-arrange, or re-allocate the network backbone blocks and the convolution filters for object detection. The search space is two-fold: 1) the network is allowed to search over allocation of different number of blocks in the backbone (e.g. ResNet, MobileNet); 2) the network is allowed to choose the dilation of each of the block. A one-shot NAS method is adopted for efficient search. After search, the model is shown to have 1) better AP results; and 2) more balanced effective receptive field (ERF). 

+ I am not aware of any work that performs search on backbone architectures for object detection yet. So the idea itself is novel;
+ The visualization of the ERF is interesting -- it reals that ERF is more balanced after searching. 

- My biggest concern is in results. It seems for Faster R-CNN with FPN, the detection results should be higher in general (e.g. R-50-FPN should at least give ~37 Ap with 1x training, and can reach 38 if it trains longer --  the same as CR-R-50-FPN in Table 1). Therefore I am not fully convinced that the searched results are obtaining meaningful gains -- maybe a result that trains longer can help here.
- Related -- I think while the idea is interesting, the limited improvement is hurting the significance of the work. In fact, to me the most important result would be on Fig 5, where it compares the speed/accuracy trade-off (directly comparing accuracy is meaning less unless the paper reaches state-of-the-art -- which is around 50 now); however, again no significant gains. Here it is because many "improvements" have been proposed after Faster/Mask R-CNN as baseline.
- (Minor) I am not sure the computation of possible choices 33^3 is accurate for the search space, because some of these 33^3 blocks are identical and therefore redundant.
- 4.3.2 is a bit misleading. At least improving backbone helps improves object detection performance (as far as I know), the plot also shows quite a bit of correlation between classification and detection performance -- please report at least the correlation for the points on the figure.

Question: 
* From Fig 4, it seems for baseline R-50, it would be best to allocate more computation on the later stages (e.g. 1st one only has 3, and last has 7), Is it true for other search results? Is it true for other head (instead of FPN)? Are there intuitive explanations for that?
* Also from Fig 4, it seems the network tend to favor more dilated convolutions toward the end? Does it have something to do with the ERF balancing?

Despite the concerns, I am still in favor of accepting the paper, as the paper concerns both accuracy and speed for detection, and applying NAS to such kind of search reals some interesting patterns (please answer the questions above).
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: <Strengths> 
+ This paper performs architecture search for object detection, especially for the computation allocation across different resolutions. It is a new application for NAS research. 
+ The proposed approach shows some marginal improvement of object detection accuracy across multiple backbones and datasets. 
+ This work proposes a new formulation to apply NAS approach to object detectors, including linking between the ERF and the computation allocation of backbone and two-level hierarchical search for stages and convolution operations.

<Weakness>
1. This paper can be regarded as an engineering work for a new domain of problem with little technical novelty. 
- From the perspective of NAS research, the proposed approach has little technical novelty; instead, it seems like an application of existing techniques (or even simpler ones) (e.g.  to a new domain of problem - object detection. 
- Given that the NAS is only applied to CNN backbones in this paper, the novelty (of proposing a new task) may be further weakened.
- The novelty of this work over existing works of “NAS on detection” (Chen et al 2019, Wang et al 2019 and Ghiasi et al 2019) is not justified.

2. Experimental results are rather weak. 
- This paper only focuses on showing that the proposed method marginally improves the performance of basic backbones (MobileNet V2 and ResNet 18/50/101). However, the improvement gaps are rather marginal (about 1.0% in average as shown in Fig.5).
- This paper does not compare its performance with other SOTA detection methods but only compare with some baselines instead. However, the performances of baselines are too low; for example, in Table 1, the mAP of the ResNet101 baseline is 38.6, which is lower by about 10 than the SOTA detector with the same ResNet101+FRCNN+FPN. It may not be convincing to use the baselines that have more than 20% lower accuracy compared to SOTA and show only ~1.x improvement over it.
- Given that the proposed approach is applicable to any backbones and detectors, more experiments should be done using recent stronger baselines (including many recent tricks like RoIAlign and DCN).
- Another experimental weakness is lack of comparison with existing NAS methods on object detections such as (Chen et al 2019, Wang et al 2019 and Ghiasi et al 2019). Even though experiment settings here may be different from those of these papers,  they should be compared in any reasonable ways. Moreover, the detection accuracies reported in this paper are not as good as the numbers of these papers. This paper argues that the proposed approach is complementary to these methods, so it would be good to report the results of the combined model with them.

3. The paper is written poorly.
- There are many grammatically wrong and awkward expressions. The draft should be thoroughly proofread. 

<Conclusion>
Although this work shows a new application of NAS for object detection, my initial decision is ‘weak reject’ mainly due to lack of technical novelty, limited experiments and poor writing. 

<Post-rebuttal comments>
Authors' responses partly resolved my concerns on the experiments. I have no object to accept this paper.
review_title: Official Blind Review #671

Meta-review: The submission applies architecture search to object detection architectures. The work is fairly incremental but the results are reasonable. After revision, the scores are 8, 6, 6, 3. The reviewer who gave "3" wrote after the authors' responses and revision that "Authors' responses partly resolved my concerns on the experiments. I have no object to accept this paper. [sic]". The AC recommends adopting the majority recommendation and accepting the paper.
Decision: Accept (Poster)