Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper argues that recently developed detection transformers didn't employ pre-training on transformer layers and not benefit from pretraining. Hence, it proposes FP-DETR to fully pretrain transformer on image classification and then finetune it for object detection by changing the task adapter. Compared to previous methods, it only uses encoder-only transformers. Further experiments show reasonable performances and robustness analysis. 
main_review: This paper is well-organized and easy to follow. 

Strength: 
+ Encoder-only transformer design without the need of extra visual backbone
+ Lower parameters compared to concurrent works

Weakness:
- The effeteness of pre-training is not well justified. The paper claims "large-scale pre-training on ImageNet helps FP-DETR learn more generalizable representation." However, the author only conducts experiments on imagenet-1k, which barely counts a large-scale pre-training. In order to further prove the benefits, it would be best to re-run experiments on imagenet-22k and show the continuous improvement .
- The necessary of deformable convolution layer.  The encoder-only transformer design would be more appreciated if not using deformable convolution. How big will the impact be if trained without deformable convolution? 
- Latency. Since author focuses on small models, it would be best to report latency.  
- The robustness analysis is not convincing and not related to the main contribution. The author should include more ablation studies on hyper-parameters instead (such as #encoder, #dim, #token, etc)
summary_of_the_review: This paper shares some insights on the proposed FP-DETR, but it lacks comprehensive experiments to justify the claims. After reading the rebuttal and extra ablation studies, most of my questions have been answered. I believe this paper has some merit and worth accepting for a poster. Hence, I have raised my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a new transformer framework for detection task by masking better use of features in pretraining phase. Inspired by ‘prompting’ in the NLP task, it treats query positional embeddings as visual prompts to better localize objects. With the similar architecture design, it mitigates the discrepancy between pretraining and finetuning stages. Task adaptor is introduced to capture inter-object relationships. The proposed framework achieves competitive performance and becomes more robust and generalizable.
main_review: - This work introduced the ‘prompting’ to detection task. By mitigating the gap between pre-training and finetuning task, it shows a better performance on COCO dataset. It is also better on robustness and gereralization to small-size dataset. Authors conducted ablation studies to verify the different settings. 

- In Table 7, it shows the FP-EncDec with 300 tokens performs better than that with only 1 token on pre-training and fine-tuning tasks. I wonder if we also apply 300 tokens to FP-DETR on the pretraining, will performance of pretraining and downstream tasks become better as well? It is interesting to see it as it may provide more flexibility to different downstream tasks.
- In table 2, is it possible to have an ablation study in which we have FP-DETR with ResNet-50 backbone? 

- Minor: it seems there is a typo -  PT-DETR in the introduction? 

summary_of_the_review: Although the idea is borrowed from the NLP field, applying it to the CV field may be helpful for the community. It provides a new perspective towards the mitigate the discrepancy between pre-training and finetuning. At the current stage, I prefer to accept this work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel transformer-based detection algorithm (FP-DETR) that could benefit from large-scale pre-training for model robustness and generalization on small datasets. FP-DETR consists of a encoder only transformer on top of a set of lightweight feature extracting layers for pre-training. It is then fine-tuned with a task-adapter that helps the model attending to object location as well as be aware of inter-object relationship for detection task. Evaluation is done on both large scale dataset COCO and relatively small domain CityScapes with ablation analysis on different system modules and design choices. The result indicates that FP-DETR presents better generalization capabilities on smaller dataset with on-part performance compared to other SOTA in general settings.

main_review: Overall, I like the paper but I feel hesitant to accept this paper at its current status . While it is no secret that one can improve model robustness and generalizability through pre-training, I do believe that the method contributes values to the community with an architecture that is designed to better utilize imageNet classification pre-training and close the gap between classification and detection in the fine-tuning stage. However, I found the current experimental protocol is not solid enough to support some of the claimed contributions. I am happy to change my rating if the author could provide justification through discussion or additional experiments. 

Pros:

- The paper is well written and easy to read. The intuition behind the algorithm design is sufficiently discussed. The technical session is described in detail.
- I like the idea of decreasing the model discrepancy between upstream classification tasks and downstream detection tasks through an encoder only transformer. 
- The intuition to provide location attention and inter-object relationship attention is interesting
- The overall validity of the algorithm is backed by performance close to SOTA on COCO and outstanding in Cityscapes. 

Cons:
- The main concern I have in the experiment is whether we can decouple the advantage of the proposed architecture or the fact that the Transformer encoder is being pretrained when compared to other SOTA (Table 2). E.g. What’s the performance of the proposed algorithm with normal ResNet backbones? What’s the performance of Deformable Tranformer with Encoder pretrained？
- The ablation analysis on the lite version (Table 3) is not as strong as one in the base version. It is easier to improve on top of a weak performance.

summary_of_the_review: The paper proposed a novel architecture to empower a transformer-based object detector with smoother pre-training and fine-tuning paradigm. The paper is well-written and provides interesting discussion on its intuition and new perspective on positional embeddings in this context. However, pre-training helps improve the model is not a novel theory by itself. The insufficient experimental comparison decreases the strength of the paper. I am looking forward to seeing the rebuttal and willing to change my rating accordingly.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method of pre-training transformer based object detector (FP-DETR). Compared to previous methods, the authors propose to pre-train DETR's encoder with ConvNet backbones on ImageNet. During the finetuning stage, the authors proposes a task adapter to map the visual prompt to classification token domain to mitigate the gap between image classification and detect object tasks. Experiments on MS-COCO show that the FP-DETR achieves competitive performance and generalizes well on small scale datasets. 
main_review: ## Strengths 

+ The submission is clearly written. The motivation and technical details are easy to understand.

+ The authors find the similarity between textual prompt filling task in NLP and object detection and propose a solution to mitigate the gap.

+ The proposed network (FP-DETR) generalizes well on two small datasets (COCO-C and Cityscapes)

## Weakness

- The technical novelty is not fully verified. According to the section 3, the authors first pre-train the encoder part on the ImageNet. In Figure 1, the pre-training leverages class token as a placeholder which could later be replaced by visual prompt information. However, the class token domain has a large gap between the visual domain with visual contents. As a solution, the authors propose a task adapter (which is an self-attention layer or a Bi-LSTM) to mitigate the gap. According to the sentence above eq5, the major function of the prompt encoder is to `improve the modeling of the inter-object relationships`. In such sense, it would be confusing whether the task-adapter is used to better model the object interaction, or to adapt the pre-training image classification task to object detection task. I would suggest the authors to calculate some embedding distance to show that task-adapter-processed embeddings' distances to class token embeddings in pre-training are smaller than pre-task-adaptor embeddings' distances. If the distances are smaller, the task adapter is verified that it could be used to transform the new query embeddings from visual prompts to class token embedding spaces, so that the FP-DETR pre-training could be easily adapted to object detection task. 

- The generalization ability is inconsistent with pre-training datasets' sizes. According to the findings in section 4.2, pre-training on ImageNet-21k has lower performance than that from ImageNet-1k. The authors conjectured that the random initialization of transformer may be the cause. However, if the pre-training strategy cannot be scaled to larger datasets to further improve the performance, this method may have smaller scope of application. It may need more efforts to solve the scalability problem. 
summary_of_the_review: This submission provides a novel way of pre-training DETR's encoder part on image classification task first, and then finetuning DETR on object detection task. However, there is one novelty not fully verified. Besides, the method may have some problem in scalability. Thus, I would wait for authors' response before making the decision.
review_title: null

Meta-review: Summary: Authors present an approach for transformer based object detection that “fully pretrains” the encoder structure of the transformer, and drops the pretrained convolutional backbone used in other works. 

Pros:
- Eliminates need of extra visual backbone
- Fewer parameters than other works
- Achieves competitive performance, especially controlling for model size

Cons:
- Multiple reviewers raised concerns about authors only evaluating their approach with pretraining from ImageNet 1K, which is not considered large scale. Authors replied with new experimental data including pretraining from ImageNet 22k, which improved results. 
- Multiple reviewers raised concerns about the need for more ablation experiments, which the authors addressed.

Reviewer scores lean toward accept. Those that lean toward reject raised issues that the authors appear to have addressed sufficiently. Not all reviewers have replied to authors, though understood at least some reviewers on this paper are on end-of-year time-off.

Overall recommendation: accept.
Decision: Accept (Poster)