Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper suggests a simple way of turning a classification pre-trained ViT into a detection model, without imposing any more image-specific inductive biases than the original ViT model itself. This is achieved by appending 100 [det] tokens to the input, and then optimally matching these with the ground-truth in the loss during training. The results are competitive with, but do not exceed SOTA.
main_review: I think this is a good paper and do not have much criticism. The writing is upfront about not aiming for SOTA but seeing how far one can go with an as "objective" as possible architecture, and show that indeed one can go far!

1) The few sections about model scaling seem out of place. They do matter at all for the paper's main message. They reads as if it was something someone tried, is "kinda meh", but does not want to throw away the results. I would have preferred if they were not included, but more analysis/details of the model was

2) An interesting thing to analyze would be, what is learned for the [det] tokens. For example, do those which are close to eachother (in latent space) also lead to mostly nearby detections?

3) As the first sentence mentions: "transformers are born to transfer" ; it would have been interesting to see how transformers pre-trained on larger datasets, such as ImageNet-21k, perform.

4) It is interesting that with this scheme, a DINO pre-trained ViT, which in the DINO paper is shown to have much better "spatial alignment", does not really lead to better detection results. I think this is worth explicitly highlighting.

5) Nitpick: footnote 3 belongs to social media, not in a paper.

Overall, my comments are not really shortcomings of the method, but ways in which the paper could be changed to turn it from "ok" to "great", in my opinion.
limitations_and_societal_impact: None
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents the design and evaluation of a Transformer-based object detector which works in a pure sequence-to-sequence manner. The proposed YOLOS detector is built from ViT by borrowing the [DET] token from detection transformer (DETR). A rather interesting result is that such a pure seq-to-seq architecture, which only takes non-overlapping image patches as input, can do object detection with a reasonable accuracy. 
main_review: Object detection is one of the fundamental problems in CV and image understanding. However, the solution to this problem is not as neat as image classification. It would be nice if an object detector can get rid of the complex and diverse design of the detector head. This paper presents a neat design of object detector and shows that a pre-trained Transformer learned from image classification task can be transferred to object detection to achieve a reasonable performance. This reviewer finds this paper quite interesting and is a meaningful investigation toward a unified Transformer for image classification and object detection tasks. 

However, there is one point of view that this reviewer cannot agree with the authors. The authors mentioned that some recent works [34, 53, 55] introduce the pyramidal feature hierarchy and locality to Vision Transformer design, which largely boost the performance. But the authors think that these architectures are performance-oriented and cannot reflect the properties of the vanilla ViT. This reviewer thinks that there is nothing wrong to make performance-oriented design for vision transformers. Images have their own characteristics including locality and large scale span. It does not make much sense to stick to the vanilla ViT only because it is the first vision Transformer design or it has high similarity with the Transformer used in NLP. This reviewer thinks that the authors should encourage fellow researchers to investigate the proposed YOLOS on other more efficient vision Transformers. 

== after rebuttal ==
The authors have addressed my concerns in the rebuttal, and they agree to include the related discussions in the final version. Besides, the authors provide convincing additional experimental results in the response to other reviewers. Taking all these into consideration and after the calibration with other papers I reviewed, I decided to raise my score. 
limitations_and_societal_impact: This reviewer does not see any serious limitations or potential negative societal impact of this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper explores how to transfer the pure ViT that is pre-trained on ImageNet to the more challenging object detection task. To cope with the object detection task, the proposed YOLOS first drops the CLS tokens in ViT and appends learnable DET tokens. Besides, the bipartite matching loss is utilized to perform set prediction for objects. Extensive experiments are provided to analyze the impacts of pre-training schemes. The proposed YOLOS also shows good performance for object detection on COCO benchmark.
main_review: **Strengths** 
-	The topic of this paper is actually quite simple yet interesting: could the vanilla pre-trained ViT be transferred to more complex task like object detection? Prior transformer-based object detection either utilizes transformer as feature extraction (ViT-FRCNN) or prediction head (DETR), whether transformer could perform object detection from sequence-to-sequence perspective is less explored and is thoroughly discussed in this paper.
-	Extensive experiments are provided to show how the pre-training schemes for ViT affect the performance of object detection.
-	The proposed YOLOS shows good performance on COCO.
**Weaknesses** 
-	As discussed in Line 108, one of the main difference between the proposed YOLOS and DETR is that DETR uses a randomly initialized transformer while YOLOS utilizes pre-trained ViT. However, there are some prior works that discuss the pre-training of DETR, such as UP-DETR [Dai et. al, CVPR 2021]. What is the difference between the proposed method and these DETR pre-training method? Why the proposed method might be better?
-	There are also some prior methods that discuss and conclude that the decoder is not necessary for transformer-based object detection [Rethinking Transformer-based Set Prediction for Object Detection, arXiv 2020]. What is the different between the proposed YOLOS and these prior encoder-only transformer-based detectors?
-	Though the authors claim that YOLOS is not designed to be yet another high-performance  object detector, the performance is not quite competitive with state-of-the-arts detectors, especially compared with DETR.
-	I do not quite understand how the visualization in Figure 2 comes to the conclusion of “We observe that each [DET] token learns to specialize on certain regions and sizes.”
-	I am quite confused with the model scaling part. What does f(Lin.)/f(Att.) actually impact the scaling strategy? Can I conclude from the section “Towards Better Bigger Transformer” that the proposed scaling method is not suitable for YOLOS?

I have mixed evaluation for this manuscript and my initial rating is around borderline. On one hand, I do think that the topic discussed in this paper is quite interesting. But on the other hand, the proposed YOLOS does not present much novel design and the performance is also not quite strong.

limitations_and_societal_impact: The limitations have been discussed in the last section of the paper.
review_title: null

Meta-review: The authors present a simple algorithm for converting a Vision Transformer (ViT) trained for classification into a detection model. The approach is based on dropping the the CLS tokens in ViT and appends learnable DET tokens, followed by matching these to the ground-truth during training. Empirical results are not SOTA, but are competitive. The reviewers appreciate the simplicity and the relatively strong performance of the resulting model. During the discussion the reviewers agreed that the rebuttal clarified the remaining points. I will hence recommend acceptance. I urge the authors to include suggested improvements in the revised version.
Decision: Accept (Poster)