Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper introduces a network architecture search (NAS) suitable for a feature pyramid network (FPN) that provides notable detection accuracy for objects at every scale. Based on the decomposition of FPN structure as (multi-scale) feature generation and feature utilization, the proposed NAS offers a new design strategies for both components. For feature generation, NAS super-net is trained to find the optimal selection of whether to reduce, maintain, or extend feature resolution after each module. For feature utilitization, it defines conditions for efficiently selecting the optimal FPN architecture. The proposed MSNAS yields the better accuracy than its backbone FPN and other NAS methods on the COCO object detection benchmark dataset.

I have a concern in technical novelty.
The only novel element considered in the proposed NAS method is to learn the stride value that controls the feature resolution at each module. The most contribution of this paper is very similar to FPN-NAS in that it re-designs FPN by choosing the optimal connections between all the modules. Furthermore, this search method does not retain the main claim of FPN, in which features at every level can be trained to have the same level of semantics.

The method used for Feature utilization is a very simple rule not relying on training to select the final feature layer to be connected to object detection heads. This simple rule can speed up training at the expense of marginal accuracy. This can be seen as part of engineering rather than technical contribution.

I also have a concern in its presentation. 
Algorithm 1 is very important for understanding how to implement the proposed method. However, it is very difficult to understand this algorithm due to missing definition of some terms such as CrossoverEncoder, MutationEncoder, CrossoverFeatureStride, MutationFeatureStride etc.
review_title: Incremental contribution in FPN design element considered in the proposed NAS method

final_score: rating: 4: Ok but not good enough - rejection
review_content: #### Summary

This submission works on the task of architecture search for object detection. The authors focus on two components: how to produce multi-scale features and how to use multi-scale features. The authors formalized a simple search space, and applied an evolution-based search algorithm. Experiments show the proposed searching algorithm is able to outperform the FPN baselines with various (small) backbones.

#### Strengths

- The task of multi-scale feature modeling in object detection is indeed important. 

- The proposed searching method is straightforward and easy to understand, and the performance compared to the baseline is OK. 

- Figure 1 is good. It clearly summarized the contribution.

#### Weaknesses

- A clear limitation is the authors did not compare to other NAS-based FPN as discussed in the related work section. Only NAS-FPN is listed in Table. 6, and it outperforms the proposed method (despite heavier computation). In my opinion, the most important comparison is to BiFPN, which also has similar FLOPS as FPN and performs 2-4 mAP higher according to EfficientDet paper. 

- The second major complaint is the scalability. The authors only show backbones up to Res 50, but not stronger ones. For example, ResNeXt101-DCN is used for reporting the state-of-the-art number in many detection paper. Ideally, the author should show their method works on this large backbone to really push the state-of-the-art. If the computation is a problem (The author already used 16 GPUs, which is more than the "standard budget" in object detection frameworks (8 GPUs)), then it's a limitation to the method.

- The authors claimed changing the backbone encoder as an advantage of the proposed method. I would argue that this also slightly makes the proposed method less practical: using a fixed backbone can better utilize the pretrained weights, and can better use the community progress in designing backbone networks. 

- While the overall idea is straightforward, it also makes the paper lack excitement. While I am not working on NAS, the paper makes me feel it just applies a NAS recipe to multi-scale feature production (I can see designing the search space is a contribution). Please correct me if I am wrong.

- (Minor) The authors only evaluate on two-stage Faster/ Mask RCNN, where the object level assign is less critical (e.g., PointRend assigns all objects to stride 4 level). I would suggest the author also try one-stage detectors, where the effectiveness of FPN is more pronounced (e.g., FCOS shows subtle differences in FPN level assign yield 1-2 mAP difference). Note that this is not a requirement for rebuttal. 

#### Summary

This paper applies a reasonable method to an important problem. The results are OK (decently outperforming the baseline) but not exciting (lack comparison to the state-of-the-art). Also, the computation requirement may limit further applying the proposed method on large backbones. In its current status, I suggest a rejection. In the rebuttal, if the authors clearly show the searched structure outperforms BiFPN and works on large backbones, I will improve my rating. 


#### After rebuttal

Q1: Thanks for running the additional experiments. Unfortunately, the results are not strong enough to convince me to use the searched architecture instead of BiFPN. It will also be interesting to see how BiFPN works under similar FLOPs with the searched model (e.g., the structure of EfficientD4 or D3).

Q2: Please do run the ResNeXt101-DCN experiments for the revision or next submission. These are critical to make people use the proposed method.

Q3: True. However training from scratch requires 6x longer training time according to [1], and is considered as a drawback.

Q4: Thank you for the clarification, this makes the contribution clearer. However my concerns on changing the backbone remains (Q3).

Q5: Thank you for considering. I agree the ranks in the leaderboard is a main factor for design choice. This also highlight the importance of Q2. 


review_title: Ok but not good enough - rejection

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper argues searching for both encoder and anchor assignment (feature utilization) in a unified NAS framework would lead to better detector performance. Experiments have been conducted on COCO dataset to show proposed method outperform baseline FPN by a significant margin, and both searching for better encoder and feature utilization benefit object detection task.

Pros:

- dealing with objects with different scale is a fundamental problem in detection, and research in this direction would benefit vision community.

- Most NAS work on detection focus on the backbone feature extractor (encoder). This paper brings new perspective for NAS on object detection task.

- The experimental results back authors' claim that searching for feature utilization brings more performance gains than searching for encoder alone.

Cons:

- The proposed method for searching a path throw super-net across multiple stride is not new. E.g.  "Auto-DeepLab:
Hierarchical Neural Architecture Search for Semantic Image Segmentation" has adopted similar techniques.

- While searching for feature utilization looks novel, from Figure 2 b) it is effectively searching for how to select and fuse features with different resolution to each detector head. Assigning objects with different scales has been explored by works such as "Scale-Aware Trident Networks for Object Detection" / "Feature Selective Anchor-Free Module for Single-Shot Object Detection" in a non-NAS setting, which might be interesting to compare the proposed method with. Also the proposed approach is essentially similar to works that search for FPN connections, e.g. "NAS-FCOS: Fast Neural Architecture Search for Object Detection".

- [minor] anchor-free object detection has growing more popularity in detection tasks, and thus might limit the importance of searching for feature utilization. 

- This paper is a bit challenging to follow and would benefit from careful proofreading. See comments below for more details.

Other comments:

- Abstract: "we show that more possible architectures of encoder network and different strategies of feature utilization". by "more possible architectures" does it mean "larger search space"?

- Intro, paragraph 1 "the key to solving" -> "the key to solve"

- Intro, paragraph 2 after introducing FPN / SSD and how they deal with multiple scales, "The basic idea to deal with the multi-scale detection problem can be summarized as below" and re-evaluate FPN architecture. this feels a bit redundant and doesn't read smoothly.

- Intro, paragraph 3 "Also, the predefined rule of feature utilization is very empirical and other alternatives may lead to better performance". Agree with the statement and there has been quite a few work that tries to better align object scales with feature maps (like the papers mentioned above). Please cite and compare.

- Related work. The method proposed in this paper is based on one-shot search and should be simpler than SpineNet. It would be great to have SpineNet in comparison (specially when SpineNet seem to show better performance)?

- Sec 3.1: This section reiterate the motivation of this paper, which have been stated in Intro section and feels a bit redundant.

- Sec 3.2, paragraph 2. "Considering operators with different strides within a mixed-block and the variation of sizes output from different operators in one mixed-block" this is very confusing. Does it indicate we have additional upscaling / downscaling operations within a mixed-block? How this related to searching through a path in a super-net?

- Sec 3.2, Eq 3. Should it be $stride_{j+1} \ne 1$?

- Sec 3.3 last paragraph "descrbied" -> "described"

- Sec 3.3. $s_i$ here indicates the selected feature size w.r.t. to image size (actually should be the inverse of it? otherwise it would be 1/4, ..., 1/64). It is also unclear how this setup would facilitate using multiple feature maps for one head.

- Sec 3.4 "It’s difficult to combine features with different resolutions by element-wise addition, so one-shot based search strategies show great compatibility with our search space." It is not straightforward to understand the connection between combining feature maps at different resolution, proposed scheme and one-shot search strategy. Please elabrate.

- Sec 3.4 " although the primitive weights in the super-net don’t perform well in terms of ranking random samples". This is very confusing. Could authors clarify?

- Sec 3.4 "statics of batch norm..." it should be "statistics"? 

- Sec 4: first paragraph. the experiment setup (dataset, splits) is mentioned in appendix but would be better to place it here.

- Sec 4.1 first paragraph "m=in" what does this mean here?

- Table 1: the table is a bit confusing. It compares ResNet-FPN with MSNAS-Resnet. It might be more clear to indicate the backbone network on each row and set the first column to be Resnet-FPN and others for MSNAS-Resnet

- Table 1: Both baseline and proposed method share the same FLOPs. Is this the case? please clarify.
 


review_title: Good results with limited novelty

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a one-shot based multi-scale features ﻿generation and utilization framework for object detection. This method searches the ﻿network stride for features generation and detection heads location for features utilization.

 Pros:
Stride and detection heads location are important factors, and automated learning of them is desirable.
This method improves the performance based on several baselines. 
Searching the detection heads location is a new idea, to my knowledge.

Cons:
However, there are some concerns about this paper.
1.     This paper seems like an improved version of the SpineNet. Searching for ﻿strides instead of permutation and utilizing a one-shot method instead of reinforcement learning is not very novel.
2.     If we treat this paper as the improved version of SpineNet, the improvement of the performance is strange. SpineNet improves the performance from 37 to 42.7 for ResNet50. Why jointly searching the backbone features generation and FPN features utilization only improves 1.2 mAP in this paper?
3.     The comparison with other state-of-the-art methods is unfair. To my knowledge, the results for DetNas (42.0) and CR-NAS (40.2) are based on 1x schedule while this method is trained from scratch for 6x schedule.
4. It would be good to evaluate on another dataset.

review_title: Major concern is about the experimental results

Meta-review: This submission got 3 rejection and 1 marginally below the threshold. In the original reviews, most of the concerns lie in the limited novelty, the inferior performance to some existing similar works and the limited scalability of the proposed method. Though authors provide some additional experiments, the reviewers still feel the experiments are not convincing and keep their ratings. AC agrees with the reviewers comments on this paper. Though achieving SOTA performance is not necessary for every submission, NAS-alike method is purely pursuing better performance (either higher accuracy or better efficiency). Thus, the performance is also important for evaluating a NAS paper. From the reviewers, the proposed method does not show better performance than some existing works, like BiFPN. This makes the value of the paper is not clear, in particular considering the method novelty is limited. The authors could consider to improve the submission in the experiments to better justify the proposed method, either achieving better performance or higher efficiency than existing works. At its current status, AC cannot make accept recommendation. 
Decision: Reject