Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper builds upon the recent advances in transformer based image classification methods (ViT variants) and detection methods (DETR variants). It argues that naively replacing the conv feature backbone in DETR with a ViT based one, is problematic due to (i) the quadratic complexity of the self attention module in ViT, and (ii) then again the attention module in the transformer encoder decoder part (which they call "neck"). To remedy, it proposes to build on top of Swin Transformer backbone, by proposing a novel reconfigured attention module (RAM), and further removing the encoder-decoder in the neck, replacing it with only a lightweight decoder.
main_review: The description of the cross-attention is a bit informal and hard to understand. There is a lot going on in Fig 3 which could be formally described for the readers benefit.

YOLOS is not scalable because of global attention between [PATCH] tokens and [DET] tokens, however here the same issue arises with cross-attention (p4 last para) and is resolved by only considering global attention at the last stage. Could something similar be done with YOLOS? e.g. removing the global cross attention between patch and det in earlier stages. And if such global attention is removed in earlier stages, Swin type layers can also be used if I am not mistaken? In that case does the network correspond to YOLOS largely (without the neck?). This should be discussed as well.

DETR (vanilla and deformable) results are not given with the base models. I understand that the FPS would be low, but the results should be reported. In addition the DeiT-tiny and DeiT-small should also be the distillation based training versions. Since they are +2 points better than the non distillation ones, one could suspect that the (a) Deformable DETR + DeiT-tiny will improve from 39.2 to ~41, which would be then higher than (b) ViDT+Swin-nano (which is comparable at 16M params vs. 18M for (a)). This would also be fair wrt to backbone, from Tab1 and Sec4(Algothims) discussion, it is mentioned that the DEIT tiny, small are eqv. to Swin nano and tiny.

I am also wondering why are DETR(vanilla or Deformable) results with any base model (DeiT-base or Swin-base) not given in Tab2. While they might not be comparable wrt FPS, they should be given.

Similarly, the results with Conv backbones should also be given, since there are higher reported results in the literature. I wouldn't penalize the paper for not having "state-of-the-art" results. This request is so that the reader can be aware of the holistic landscape.


--- Post rebuttal update ---
I am not entirely convinced that a simpler baseline improvement to YOLOS can not be devised. But the proposed method is surely one way to go forward. The results and discussions could also be valuable to the readers. I am not upgrading my rating but I wouldn't argue strongly for rejections, specially when there is some support from the other reviewers.
summary_of_the_review: The paper is generally well written, however one of the main contribution, the reconfigurable attention module, should be more precisely and formally described, at least an appendix. Otherwise it is a little hard to get with Fig3 largely.

The contextualization of the paper wrt YOLOS is also something that could be improved. It seems that the model is more closely related to YOLOS than seems at first read.

The experiments should be made more complete with fair comparison for baseline backbones etc. And the results with conv backbones should also be given for completeness and benefit of the reader.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose an efficient transformer-based detector. Specifically,  the proposed detector introduce Reconfigured Attention Module (RAM) , encoder-free neck and token-based knowledge distillation to boost performance with low computational overhead. 
main_review: Strengths
- Compared with other transformer-based detectors, the proposed detector is effective and efficient, which has a high performance with a fast speed.
- The comparison with other transformer-based detectors is written well and readable.

Weaknesses
- Though the proposed detector is effective and efficient, the novelty is limited. To my knowledge, Reconfigured Attention Module (RAM) and token-based knowledge distillation is not novel. Also, other techniques, such neck structure and auxiliary decoding loss, are not original, which are borrowed from DETR and Deformable DETR. 
- There are some concerns about  Reconfigured Attention Module (RAM). First, there is little about the cross-attention between [DET] $\times$ [PATCH]. Could you provide more details about how to process [DET] $\times$ [DET] and [DET] $\times$ [PATCH] at once? Second, the proposed RAM is not novel and may not have great contribution to the performance. RAM only has cross-attention between [DET] $\times$ [PATCH] in the last stage. The function of this module is the same as the transformer decoder. For the rest stages,  there are two operations: [PATCH] $\times$ [PATCH] and [DET] $\times$ [DET]. [PATCH] $\times$ [PATCH] is a default operation in Swin Transformer. To my knowledge, [DET] $\times$ [DET] is not meaningful and may have no effect on performance. Could you provide an experiment that removing [DET] $\times$ [DET] from the RAM module except the last stage?
- In Table 4 (w. Neck), the detection performance drop largely if all the stages are not involved. In this setting (w. Neck), the transformer decoder also have interaction between [DET] $\times$ [PATCH]. Thus, the performance drop is not only caused by the lack of interaction between [DET] $\times$ [PATCH] in RAM. Could you provide an experiment that only interacting with {4} [PATCH] in the transformer decoder?
- In Table 2, there is not an experiment about Deformable DETR + Swin-base. Could you provide this experiment? 
summary_of_the_review: I appreciate the effective and efficient of the proposed detector, but I prefer to reject this paper for the above weaknesses. If the authors solve my concerns, I would like to raise my rate.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes ViDT, a high-performance Detection Transformer with an impressive accuracy-speed trade-off. A lot of experiments as well as ablation studies are conducted to prove the effectiveness of the proposed detector. The design principle of ViDT can also generalize and inspire future detector design. Moreover, this paper also includes an in-depth analysis of several current Detection Transformer architectures.

main_review: #### Strengths

- This paper is well motivated and clearly written. It proposes a high-performance Detection Transformer coined as ViDT, which synergizes the best of Swin Transformer, YOLOS, as well as DETR (Deformable DETR) architectural design and for the first time demonstrates that the Detection Transformer family can achieve a very competitive accuracy-speed trade-off in the challenging COCO object detection benchmark. A common pitfall is that some people believe Deformable DETR is "fast". Deformable DETR is indeed fast in terms of the convergence, *i.e.*, the total training epochs required by Deformable DETR are much smaller compared with the original DETR. However, Deformable DETR is not fast in terms of inference FPS, which is even slower than DETR. ViDT addresses the latency issues while maintaining a very high detection AP. It shows that Detection Transformer family also has the potential to be as strong as the highly-optimized detectors, *e.g.*, the YOLO series.

- This paper proposes an efficient Detection Transformer design principle that can generalize well and inspire future Detection Transformer design, *i.e.*, the encoder part in current DETR & Deformable DETR design results in high latency. Inspired by YOLOS, this paper proposes to incorporate [DET] tokens in the earlier stage of the model, which can compensate for the effects of the encoder while maintaining high inference FPS. Therefore the Transformer encoder part can be ablated by adding [DET] tokens to stage-4 of the backbone. Meanwhile, this paper finds that the Transformer decoder part is crucial for fast convergence for several Detection Transformer instances, which cannot be easily removed.

- This paper conducts a lot of experiments studying different configurations of DETR & Deformable DETR with ViT backbone, YOLOS, and the proposed ViDT in the same testbed. These results are quite meaningful and can be served as valuable references & baselines to the community as well as future work.

  

#### Weaknesses

- I believe there is one missing detail about the model design. If the [DET] x [PATCH] attention starts at stage-4, I wonder what happens to the [DET] x [DET] attention? Is [DET] x [DET] attention performed across all previous stages? And what are the feature dimensions of [DET] tokens? Are the feature dims always the same as the stage-4 feature, or they are gradually raising during different stages. In a word, I think the details and configurations of [DET] x [DET] attention need to be clarified.


--- Post rebuttal update ---
After reading the rebuttal, other reviewers' comments, and the discussion, overall I think the detector presented in the paper is strong in terms of both precision and efficiency, and the advantages I mentioned above are valuable, especially the first one. I keep my original rating.
summary_of_the_review: Overall, I believe this is a meaningful paper from the perspective of both object detection performance and research. It will also inspire the design of [DET] token based framework of other object- and region- level downstream tasks, *e.g.*, instance segmentation, video instance segmentation as well as panoptic segmentation.
review_title: null

Meta-review: The paper introduces an object detection method that integrates vision and detection transformers through a novel Reconfigured Attention Module (RAM). Among other questions, the reviewers raised concerns about fair comparison with baselines, limited novelty of the RAM module, completeness of experiments, and missing details. The rebuttal adequately addressed these concerns with clarifications and additional experiments. R1 remained unconvinced that a simple modification to YOLOS could not be devised to improve the speed similar to the proposed method, but stated he/she wouldn’t argue strongly for rejection. While this is a legitimate concern, the AC agrees with R2 and R3 that the paper has enough merits to be accepted at ICLR, as the results are strong and are likely to have significant practical value.
Decision: Accept (Poster)