Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: A fully transformer-based object detector (transformer-based backbone and neck) is an interesting topic. This paper proposes the method of the Fine-grained and Coarse-grained crossing representations for building efficient transformer-based object detectors. In the backbone, this paper maintains both the fine-grained and coarse-grained features and introduce a lightweight local-global cross fusion module. In the neck, the proposed module allows the detection tokens to make attention-based interaction with fine-grained representations firstly, and then perform further interaction with coarse-grained representations. The results on the COCO dataset demonstrate the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths
1. This paper is technically sound and easy to understand.

2. The proposed method is interesting and has a clear motivation. Other transformer-based detectors do not make good use of the fine-grained information, resulting in unsatisfactory performance. The proposed local-global cross fusion module, fine-coarse aware neck, and efficient multi-scale feature integration are novel.

3. The experiments are extensive and ablation studies are comprehensive to understand the proposed method.

Weaknesses

1. The writing needs to be improved slightly.

2. The results in Tab.1 are impressive, yet the authors are advised to make more discussions on the results to make the paper stronger.

questions: 1. Can the authors further explain the calculation process of Eq. (9)?

2. Where does the proposed efficient multi-scale feature integration demonstrate efficiency compared to other FPNs?

3. In which stage is the proposed multiscale feature integration used?

limitations: Yes, the authors have discussed the limitations and potential negative social impact of their work in the Appendix.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces the Fine-grained and Coarse-grained crossing representations for building a Detection Transformer by using a local-global cross fusion module and Fine-Coarse Aware Neck.
strengths_and_weaknesses: Strengths:
+ The paper is well organized.
+ The performance boost seems good with a slight increase in Flops.

Weaknesses:
- The whole pipeline seems to be fine, but lacks deep insight. Most of the operations are feature aggregation between different levels, which has been deeply explored in object detection.
- Some of the statements in this paper are confusing. For example, how to understand “combine the MSDA(...) to Qdet and interact with Fo” in Line192.
- Writing errors. For example, “the lightweight bottom-up feature integration algorithm Efficient Multi-scale Feature Integration” should be “the lightweight bottom-up feature integration algorithm, i.e., Efficient Multi-scale Feature Integration” in Line138. The authors are advised to double-check for similar errors.

Overall, the simple and lightweight structure brings a huge performance boost is the reason I agree to accept this article.

questions: - Why are features of outer and features of inner different sizes in Line148?
- What happens when det tokens interacts with coarse-grained outer patches first and then with fine-grained inner patches in section3.2?
- Based on Eq. 8, my understanding is that Qdet and MSDA(Qdet, Fi) have a shortcut, but why Qdet and MSDA(Qdet, Fo) do not have shortcut?

limitations: The authors have discussed the limitations and potential negative societal impact of their work in A.4.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the topic of 'Fully Transformer-based Object Detector' (except the stem layer, do not use CNN layers for operations). Existing Transformers mainly rely on global interaction between outer patches, but this work proposes a way of integrating attention of inner patches, capturing local & fine-grained information; (1) LGCF: mixing fine & coarse patches with interacting them, (2) FCAN: alternating cross-attention btw <det x fine-grained patch> and <det x coarse-grained patches>, and (3) EMFI: using the first block patches efficiently. The proposed architecture shows a better AP and Speed trade-off compared with other compared detectors. To support their justification, the authors provide some ablation studies on the proposed three components.
strengths_and_weaknesses: There are both strengths and weaknesses I found while reading the paper.

### Strength
1) Integrating the idea of 'local + global information' into Transformer-based object detection is reasonable because improving AP on varying-size objects (especially, small-size) is a major challenge in this domain. 
2) The proposed three components (blocks) look simple and effective when combined. Therefore, they did not harm the inference speed of the detector.
3) Experimental results are not promising but reasonable improvements.

### Weakness
1) ***Performance drop for small-size objects***. According to the experiments, $AP_s$ drops from 30.4 to 28.1 compared to ViDT, although more fine-grained information was integrated. I can't understand why this happens. Intuitively, the proposed ideas of fusing more fine-grained information should improve $AP_s$.
2) ***Incomplete Analysis on Backbones***. In Section 4.3, there is a comparison between PyramidTNT and Swin-Transformer. I think this analysis is not complete to show that the improvement did not come from the replacement. First, ViDT is not designed to be with PyramidTNT. Second, only accuracy and AP measures are reported without 'FLOPS or FPS'. To sum up these, I think a more suitable comparison is combining Swin-Transformer with your three components. The performance trends with ViDT could not tell us their impact on your architecture.
3) ***Weak Novelty***. The proposed approach mostly relies on existing works. For local- fine-grained information, PyramidTNT is adopted as the backbone, and the proposed three components are very simple and straightforward; this could be an advantage in practice, but academically looks like an incremental paper.

questions: 1) Could you explain why $AP_s$ drops compared with ViDT (Swin-small)?
2) I feel hard to know what components make the architecture faster than others. I understand your design is simple and effective but there is no explanation why it is faster than another efficient mechanism (e.g., local attention in Swin - ViDT).
3) Your architecture seems to be coupled with PyramidTNT. This could be a limitation (low extendability). Do you think your three components can be combined with other backbones as well? If possible, doing and comparing them would be good to prove that the performance improvement is not from the backbone.


limitations: Refer to the strengths and weaknesses above.
review_title: null

Meta-review: metareview: This work proposes a new object detector architector that is based on a CNN stem, combined with a mostly transformer-based architecture, with the addition of a cross-fusion module that allows for reconciling coarse and high-grained features for more precise object detection.

Thee paper is well-written, novel and presents a significant gain over the state of the art, using reasonable amount of compute.

Object detection is a central area of interest and this method shows how to leverage the power of transformers to push the envelope in this domain. Therefore, I propose this paper be accepted at NeurIPS 2022. 
confidence: Certain
Decision: Accept