FP-DETR: Detection Transformer Advanced by Fully Pre-training
Authors: Wen Wang
Yang Cao
Jing Zhang
Dacheng Tao
Venue: ICLR 2022 Poster (2022)
Keywords: Object Detection
Detection Transformer
Pre-training
Visual Prompt

Abstract: Large-scale pre-training has proven to be effective for visual representation learning on downstream tasks, especially for improving robustness and generalization. However, the recently developed detection transformers only employ pre-training on its backbone while leaving the key component, i.e., a 12-layer transformer, being trained from scratch, which prevents the model from above benefits. This separated training paradigm is mainly caused by the discrepancy between the upstream and downstream tasks. To mitigate the issue, we propose FP-DETR, a new method that Fully Pre-Trains an encoder-only transformer and smoothly fine-tunes it for object detection via a task adapter. Inspired by the success of textual prompts in NLP, we treat query positional embeddings as visual prompts to help the model attend to the target area (prompting) and recognize the object. To this end, we propose the task adapter which leverages self-attention to model the contextual relation between object query embedding. Experiments on the challenging COCO dataset demonstrate that our FP-DETR achieves competitive performance. Moreover, it enjoys better robustness to common corruptions and generalization to small-size datasets than state-of-the-art detection transformers. Code will be made publicly available at $\url{https://github.com/encounter1997/FP-DETR}$.

Published as a conference paper at ICLR 2022
FP-DETR: DETECTION TRANSFORMER ADVANCED BY FULLY PRE-TRAINING
Wen Wang', Yang Caol:2t, Jing Zhang', Dacheng Tao
l University of Science and Technology of China Institute of Artificial Intelligence, Hefei Comprehensive National Science Center The University of Sydney, *JD Explore Academy, China
wangenêma i1. ustc. edu. cn, forrest c edu cn, jing. zhangl è esydney... edu. au, dacheng. taoθ gma com
ABSTRACT
Large-scale pre-training has proven to be effective for visual representation learn- ing downstream tasks, especially for improving robustness and generalization. However, the recently developed detection transformers only employ pre-training on its backbone while leaving the key component, i.e., 12-layer transformer, being trained from scratch, which prevents the model from above benefits. This separated training paradigm is mainly caused by the discrepancy between the up- stream and downstream tasks. To mitigate the issue, we propose FP-DETR, a new method that Fully Pre-Trains an encoder-only transformer and smoothly fine- tunes it for object detection via task adapter. Inspired by the success of textual prompts in NLP, we treat query positional embeddings as visual prompts to help the model attend to the target area (prompting) and recognize the object. To this end, we propose the task adapter which leverages self-attention to model the con- textual relation between object query embedding. Experiments the challenging COCO dataset demonstrate that our FP-DETR achieves competitive performance. Moreover, it enjoys better robustness to common corruptions and generalization small-size datasets than state-of of aef-the-art detection transformers. Code will be made publicly available https: / thub. com/ encounter1991 -DERD....
INTRODUCTION
Since the surge of deep learning-basee object detector (Girshick et al., 2014), pre-training the back- bone on large-scale datasets like ImageNet (Deng et al., 2009) and fine-tuning the model on down- stream tasks has become well-established paradigm. Pre-training significantly improves robust- ness (Hendrycks al., 2019) and may enhance the model performance, especially on small datasets.
Following this paradigm, modern object detectors, like Faster RCNN (Ren et al., 2016) and YOLO (Redmon al., 2016), generally add a lighteeight task-specific layers on top ImageNet pre-trained backhones, so that the model can enjoy the aforementioned benefits of pre-training. However, the recently developed detection transformers' c.g., DETR (Carion et al., 2020) and De- formable DETR (Zhu et al., 2020), only pre-trains its CNN backbone, while leaving the core module, a 12-layer transformer (including both encoder layers and decoder layers), trained from scratch in downstream tasks. As result, the model suffers from limited robustness against common corrup- tions and the reliance on large amount of training data for fine-tuning. Though UP-DETR (Dai al., 2020) attempts mitigate this problem by unsupervised pre-training. requires an off-the- shelf CNN backbone that has already be pre-trained. Moreover, the two-stage separate pre-training methods may impair the model performance, since the two parts work together for object detection during fine-tuning,
The key reason that hinders existing detection transformers from benefiting from large-scale pre- training can be attributed to the discrepancy between the upstream task and the downstream task.
"This work was done during Wen Wang's 's internship at JD Explore Academy. 'Corresponding author. In this paper, use detection transformer represents class of object detectors built upon transformers, while DETR represents the seminal work by (Carion al., 2020).

---

Published as a conference paper at ICLR 2022
Firstly, the object inrecctiooooriennte model structures are difficult to adapt to the ImageNet classifi- cation task. For example, the transformer decoder requires multiple query embeddings for detecting objects, while for ImageNet classification, there is only a single query embedding (class token) used. If the decoder is included during pre-training, both the self-attention layers and the projections on query embeddings in cross-attention layers may easily overfit the single class token, making it is hard to pre-train the decoder. Secondly, the upstream ImageNet classification task misses some crucial components for the downstream object detection task. For example, while the downstream object detection task requires both localization and classification for the objects of interest, the up- stream classification task only focuses on the latter. Moreover, the object relation modeling object detection, which is important for removing heuristic post-processing like non-maximun suppression (NMS) (Carion et al., 2020), is absent in image classification.
To mitigate these problems, we propose FP-DETR, a novel method that reformulates the pre-training and fine-tuning phases for detection transformers. It fully pre-trains detection transformer on Im- ageNet classification task and smoothly fine-tune it for object detection task though task adaptor. Concretely, during pre-training, we introduce an encoder-only transformer structure by removing the decoder that can hardly be well pre-trained on the ImageNet classification task. Moreover, since both the CNN backbone and the transformer encoder in detection transformers can be seen as feature extractor, we replace the complex CNN backbone with simple multi-scale tokenizer and only uses the transformer encoder for feature extraction. The resultant architecture is an efficient transformer encoder-only model. During fine-tuning on the object detection task, we take inspiration from the success of textual prompt (Liu et al., 2021a) in NLP and treat the query positional embeddings as visual prompts to help the model attend the target areas (prompting) and recognize the object. Our key intuition is that if the model knows where to look at, it simply needs to identify the category of the object in hand, as did during pre-training Based on this motivation, we devise lightweight task adaptor to enhance the prompt ability of query positional embeddings and hence bridge the gap be- tween upstream classification task and downstream object detection task. It leverages self-attention to emphasize the relation between query embeddings. such that the inter-object relationships. which have been neglected during pre- pre-training, can be captured during fine-tuning.
Experiments show that the proposed FP-DETR achieves competitive performance on the challeng- ing COCO dataset (Lin et al., 2014), and a better trade-off between the number of parameters and detection accuracy. Moreover, FP-DETR is more robust against common corruptions and gen- eralize well to small-size datasets like Cityscapes (Cordts et al., 2016), attributing to the effective fully pre-training.
2 RELATED WORK
Object Detection. Object detection is one of the fundamental tasks in computer vision (CV). Rep- resentative object detection methods can be roughly categorized as the one-stage object detector, e.g., YOLO (Redmon et al., 2016), SSD (Liu et al., 2016), and two-stage object detectors like Faster-RCNN (Ren et al., 2016). While significant progress has been made, these methods are not end-to-end and rely on heuristic components like non-max suppression (NMS) and rule-based label assignments. Recently, DETR (Carion et al., 2020) formulates the object detection task as a set prediction problem and proposes an end-to-end pipeline to exploit transformer and bipartite match- ing for detection. The success of DETR brought the recent surge of detection transformers. For example, Deformable DETR (Zhu al., 2020) proposes deformable attention to significantly accel- erate the model convergence and reduce the computational cost, while allowing the model to benefit from multi-scale features. Conditional DETR (Meng et al., 2021) narrows down the spatial range for localizing the object regions via learning the decoder embedding conditioned on a spatial query. REGO-DETR (Chen et al., 2021) mitigate the training difficulty through Region-of- based detection refinement. While effective, the crucial component in these models, i.c.. a 12-layer transformer, is trained from scratch, which limits the robustness and generalization ability of the detection models.
(Rol)
Pre-training for Object Detection. The seminal work RCNN (Girshick et al., 2014) demonstrates the many benefits of ImageNet pre-training for object detection, e.g., better detection performance and faster convergence. Since then, pre-training and fine-tuning has become a well-established paradigm for object detection. Recently, He et.ll challenge the common belief by showing that
2

---

Published as a conference paper at ICLR 2022
alephant
Predction
Prodiction
xL
xL
Deformable Transformer Encoder Layer
Pre-trained Deformable Transformer Encoder Layer
Projection
Rejection
Prompt Encoder
Plattonod Mulb scale Potchess
Closs Token
Visual Prompls
Pre-training
Fine-tuning
Figure I: Illustration of pre-training and fine-tuning stages of FP-DETR.
training from scratch on datasets like COCO with a longer schedule can also produce a competitive detection performance (He al., 2019). However, Hendrycks et.al show that though comparable in-domain performance can be achieved, training from scratch results in much worse out-of-domain generalization (Hendrycks al., 2019). Inspired by the success of pre-training for CNN-based object detectors, UP-DETR (Dai et al., 2020) proposes a novel random query patch detection pre- training task to improve DETR's performance and convergence. Though progress has been made, their method relies on pre-trained off-the-shee CNN backbone. Moreover, the improvement is limited, since the CNN backbone and the transformer are pre-trained separately while they need to work jointly in the downstream task. Probably the most similar work to ours is YOLOS (Fang al., 2021), which also pre-trains and fine-tunes an encoder-only transformer for object detection. How- it aims demonstrating the feasibility of detecting objects using BERT (Devlin et al., 2018), with minimum inductive bias. As a result, the discrepancy between the upstream and downstream tasks, which significantly degenerates the model performance, is not considered in YOLOS. By trast, we aim at maximizing the benefits of pre-training for the downstream object detection task. To this end, we treat query positional embeddings visual prompts and devise novel task adaptor to smoothly fine-tune the pre-trained model on downstream tasks. Moreover, since object detec- tion task often requires high-re reghuuesolution images as input, inductive bias like sparsity introduced to reduce the computational costs.
Prompt-based Learning in NLP. Large-scale pre-trained models like GPT-3 (Brown et al., 2020) advance the field of NLP. However, such large-scale models are not designed for fine-tuning. To bet- ter utilize these models for downstream tasks, the "pre-train, prompt, and predict" paradigm emerges in the NLP community, and has drawn increasing attention (Liu et al., 2021 The new paradigm reformulates the downstream task to mimic the task during pre-training with the help of a textual prompt, bridge the gap between upstream and downstream tasks. Specifically, pioneer work intro- duces manually designed prompts to improve the fine-tuning (Schick & Schüizee, 2020), or searches the textual prompt for downstream tasks (Gao al., 2020; Jiang et al., 2020). Though improvements have been made, these prompts correspond to natural language phrases in discrete space can be sub- optimal, since the neural networks inherently continuous. To tackle this problem, P-tuning (Liu et al., 2021b) proposes to optimize contiguous prompt embedding to better close the gap between pre-training and fine-tuning. In this paper, instead of reformulating the downstream task, we pro- pose a new perspective and treat the query positional embeddings in detection transformers as visual prompts. To this end, task adaptor devised to facilitate fine-tuning on the downstream task.
METHOD
In this section, we first revisit the preliminaries on detection transformers, then introduce the pre- training and fine-tuning phases of our method shown in Figure 1. Through exploiting the large- scale pre-training on ImageNet, our detection transformer can enjoy better detection performance, improved robustness, and enhanced generalization.
3

---

Published as a conference paper at ICLR 2022
3.1 PRELIMINARIES ON DETECTION TRANSFORMERS
Detection transformers generally consist of the following parts: a backbone, a transformer encoder, transformer decoder, and a task-specific feed-forward network (FFN). The backbone F can be decomposed as multiple stages, i.e., F = o .... f o s², where each stage S takes the output from the previous stage as input and outputs the down-sampled feature map at stage For an input image I E IR xx333 the backbone extracts multi-scale features, which are flattened and projected to 1-D sequence representation. Afterwards, the sequence representation is added with position and level embedding to obtained r E RN*D, where N = Σ'= HW/S? is the sequence length, St is the down-sampling rate at 1-th feature level, and D is the dimension of the feature embedding. The transformer general consists both encoder and decoder, and takes the sequence representation extracted from the image and the query embeddings input for context modeling. The final prediction is made by applying FFN to the object representations.
While Deformable DETR (Zhu al., 2020) contains four different feature levels with different down-sampling rates, most other detection transformers, including DETR (Carion et al., 2020), Conditional DETR (Meng et al., 2021), contains only a single feature level.
3.2 PRE-TRAINING
Encoder-only Transformer. As described in Section 1, for image classification, only a single class token required for context aggregation. The six decoder layers trained on only one class token is prone to overfitting. To tackle this problem, we remove the decoder during pre-training on the image classification task, which leads an encoder-only transformer structure. We follow Deformable DETR (Zhu et al., 2020) to adopt multi-scale deformable attention with four different feature levels to build our model. It largely alleviates the slow convergence of DETR (Carion al., 2020), with the help of inductive bias like sparsity.
To perform image classification, an additional class token added with corresponding positional em- bedding Tels E RDD is concatenated with the sequence feature, for aggregating global context from the image tokens. Thus the input to the encoder-only transformer 24 can be written as,
20 Ieis: x :.:::..;;"""
(1)
Afterward, the input sequence feature is processed by T' different transformer encoder layers, each consists of multi-scale deformable self-attention (MSDSA) layer and an MLP layer. Residual connections and layer norm (LN) are applied after each sub-layer, i...
z't LN(MSDSA(z--)) + 21-1)
1 1...7, ., t l...T.
= LN(MLP(2' +
(3)
Final predictions made on class token the 7-th layer, i.e., 24. During implementation, the class token is treated as an additional feature level, with only one resolution. Since multi-scale deformable attention assumes different levels of feature maps are spatially aligned, which does not hold for the class token in Vets, attention mask is utilized to remove the information flow from Eets-
Lightweight Multi-scale Tokenizer. The transformer encoder's 's ability to serve as general feature extractor has been proved by recent advances in vision transformers (Dosovitskiy et al., 2020). While existing detection transformers contain both sophisticated backbone and transformer encoder for feature extraction, we argue that such design brings unnecessary complexity to the detection model. Instead, we replace the complex backbone with simple convolutional layers that work as a multi-scale tokenizer. Specifically, the feature extractor at each stage fi is a simple single-layer convolutional layer that designed only for www.-sam the feature map to expected resolution. Such design significantly simplifies the model and makes it easy to pre-train the detection model on ImageNet classification task. The detailed structure of our multi-scale tokenizer is shown in Appendix A.
Discussion. Most of the existing vision transformers adopt an encoder-only structure (Dosovitskiy et al., 2020; Liu et al., 2021c; Xu et al., 2021). By reformulating the detection transformer to an encoder-only structure, we solve the problem that the decoder is difficult to pre-train and simplify the model structure. Moreover, it allows us take advantage of the existing experience in training vision transformers. In Appendix A.2, we explore pre-training on ImageNet using the encoder- decoder structure, however, the performance is much worse than that of the encoder-only structure.
4

---

Published as a conference paper at ICLR 2022
Textual (what
Visual Prompt (where to attend)
I love this movie. Overall, it [mask] movie.
Language Model Pre- tre-trained on Masked Language Modeling Task
Visual Transformer Pre-trained on Image Classification Task
love this movie. Overall, is good movie.
Positive
Person
Textual Prompt
Visual Prompt
Figure 2: Analogy between textual prompt in NLP and visual prompt in CV.
3.3 FINE-TUNING
During fine-tuning, the single class token is replaced by Ng query content embeddings (Carion et al., 2020) Tobj E xD for object detection. The content embeddings are also added with query positional embeddings (Carion al., 2020) p E RNNxD before feed into the transformer encoder. Thus the input to the transformer encoder is:
Zo obj p² 2b p²: ... F obj
No +
(4)
The class token for image classification and the query content embeddings for object detection are both designed for context aggregation. While the former focuses on global context aggregation, the latter focus on aggregating the local context for one specific object instance.
Query Positional Embeddings as Visual Prompts. The textual prompt in NLP is shown in Fig- ure 2. It reformulates the downstream task mimic the pre-training one, so that the pre-trained model can better handle the downstream task. However, finding the optimal textual prompt is not easy, recent work such as P-tuning (Liu et al., 2021b) searches prompts contiguous space to bridge the gap between the upstream and downstream tasks.
Object detection can be decomposed into two subtasks, i.e., localization and classification. Our intu- ition is that if the classifier pre-trained on ImageNet knows where to look at, it can easily recognize the object within the specified region, as did during ImageNet classification pre-trate pre-training. In our de- tection transformer, the query position embeddings different query content embeddings to sample the corresponding context from the image content. In other words, the query position embedding works visual cue point the image region that the model should focus on, as shown in figure 2 (b). From this perspective, the query position embedding serves the process of training the model to localize objects corresponds to the process of searching textual prompts in contiguous space. And the process of final classification corresponds to the process of filling in the blanks and answers mapping in NLP (Liu et al., 2021a).
mapped to the reference points, which guide
visual prompt, which analogous the textual prompt in NLP. Further,
Improving Fine-tuning with Task Adaptor. Based on the above motivation, we propose task adaptor to improve the query positional embeddings' ability to prompt. Specifically, the inter-object relationship can help the model better identify object regions. However, this crucial part missing during ImageNet classification pre-training, since only one class token is used. To this end, we devise the task adapter to improve the modeling of the inter-object relationships. It pre-processes the visual prompts before they sent to each pre-trained encoder layer, i.c.,
O:N₂ Task Adaptor (=t 0:NN
Zt
By default, the task adaptor is instantiated as a self-attention layer. Afterward, the sequence is pro- cessed by the transformer encoder layers, as did in Equation and 3. The object pattern embedding in the last encoder layer, i.e., 2t ,0:N, is used for final prediction. Our task adaptor helps the visual
5

---

Published as a conference paper at ICLR 2022
Model
Layers
Hidden size D
MLP size
Heads
Params
Acc Top-1
Lite
12
256
1024
8
11M
77.3
Small
12
384
1536
8
23M
80.8
Base
12
480
1920
10
35M
82.0
Table I:
Details of our FP-DETR variants. Parameters (Params) and top-1 accuracy (Ace @ Top-1)
here are calculated on the upstream ImageNet-1k (Deng al., 2009) classification task.
prompts for individual objects to make associations with other instances, making them more suitable for the downstream object detection task.
Discussion. Recently, Shen et al. (2021) find the connection between named entity recognition (NER) in NLP and object detection in CV, and propose a two-stage method for NER. Similarly, We propose new perspective to view query positional embeddings as visual prompts. However, the existing pre-trained visual models are not powerful pre-trained language models (Brown al., 2020), As a result, we are unable to fix the pre-trained models and only tune the visual prompts, to perform few-shot or even zero-shot learning downstream tasks, as did in NLP. But still, this new perspective helps us to better understand the detection transformers and the discrepancy between the pre-training and downstream tasks, which further helps design the task adaptor smoothly fine-tune the pre-trained model on downstream tasks.
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Datasets.
Following the common practice, detector is pre-trained ImageNet (Deng et al., 2009) and fine-tuned on COCO 2017 (Lin et al., 2014) train set. Evaluation results on the val set of COCO 2017 are reported. To evaluate the models' robustness against common corruptions, report the model performances on COCO-C (Michaelis et al., 2019), which is obtained by applying corruption synthesis algorithm COCO. Besides, we evaluated the model's generalization ability by fine-tuning on the small-size dataset, i.e., Cityscapes dataset (Cordts et al., 2016).
Model Variants The lite version of our model follows the hyper-parameters of the Deformable Transformer in Deformable DETR (Zhu et al., 2020). Specifically, the encoder-only transformer contains 12 self-attention layers, each layer with heads, the dimension each head is 32, and the dimension of the MI.P layer in FFN is 1024. Furthermore, we introduce two other model variants by changing the number of heads, or dimensions per head, while keeping the model depth unchanged. Table 1 summarizes the different variants of our model.
Training & Fine-tuning. By default, our FP-DETR is pre-trained on ImageNet-Ik Ik (Deng et al., 2009) for 300 epochs with AdamW (Loshchilov & Hutter, 2018) optimizer and cosine learning rate scheduler. Training strategies in DeiT (Touvron et al., 2021a) are adopted, and the image size is set as 224 x 224. We use a batch size of 1,024 for training, and the initial learning rate is set as 5 x 10 After pre-training, models are fine-tuned tuned for 50 epochs with Adam optimizer on the downstream tasks. The learning rate is initialized as 1 10-1 and decreased by factor of 0.1 at the 40th epoch. Deformable DETR to apply deep supervision (Lee et al., 2015) on
We follow the implementation the last six encoder layers. The class token and the query content embedding are concatenated with the image tokens at an intermediate (the 7th) encoder layer, following CaiT (Touvron et al., 2021b), Besides, we set both the number of sampling points and the feature levels multi-scale deformable attention as 4, and the number of object query embeddings as 300. Models are fine-tuned with batch size 32, All experiments are implemented on the NVIDIA A100 GPU.
4.2 COMPARISON WITH STATE-OF OF-THE-
As can be seen in Table 2, FP-DETR achieves competitive performance COCO 2017 val set. Specifically, FP-DETR- -Base is comparable with state-of-the-aan the-art Deformable DETR, while its small variant with 24M parameters DETR and DETR with about 40M parameters. It performs better on detecting small objects and has better localization ability.
match the performance of UP-DETR and outperforms Conditional

---

Published as a conference paper at ICLR 2022
Table Comparision of FP-DETR with other detection transformers on COCO 2017 val set. Mod- els are categorized as encoder-only (Enc) and encoder-decoder (Enc-Dec) according to the trans- former structure. indicates the model is pre-trained on ImageNet-2Ik,
Method
Structure
Backbone
Epochs
AP
AP
AP
APs
APM
AP1
Params
DETR
Enc-Dec
ResNer-50
500
42.0
62.4
44.2
20.5
45.8
61.1
41M
UP-DETR
Enc-Dec
ResNet-50
300
42.8
63.0
45.3
20.8
47.1
61.7
41M
Conditional DETR
Enc-Dec
ResNet-50
50
40.9
61.8
43.3
20.8
44.6
59.2
44M
Deformable DETR
Enc-Dec
ResNet-18
50
40.1 58.4
43.7
22.0
43,4
53.0
23M
Deformable DETR
Ene-Dec
ResNet-34
50
42.3
60.7
46.0
24.2
45.8
56.1
33M
Deformable DETR
Enc-Dec
ResNet-50
50
43.8
62.6
47.7
26.4
47.1
58.0
40M
YOLOS-S
Enc
150
36.1
56.5
37.1
15.3
38.5
56.2
31M
YOLOS-B
Enc
150
42.0
62.2
44.5
19.5
45.3
62.1
127M
FP-DETR-Lite
Enc
50
37.2
56.5
40.4
21.7
40.0
48.6
11M
FP-DETR-Small
Enc
50
42.5
62.6
45.9
25.3
45.5
56.9
24M
FP-DETR-Base Enc
50
43.3
63.9
47.7
27.5
46.1
57.0
36M
FP-DETR-Base Enc
50
43.7
64.1
47.8
26.5
46.7
58.2
36M
Moreover, FP-DETR shows a better trade-off between model parameters and detection accuracy compared Deformable DETR. For example, FP-DETR-Small outperforms Deformable DETR with ResNet-18 (He et al., 2016) backbone, which has a similar amount of parameters. It also slightly outperforms Deformable DETR with ResNet-34 backbone, which has about 10M more parameters. The smallest version of our model, FP-DETR-lite, achieves 37.2 mAP on COCO 2017 with only 11M parameters.
Besides. FP-DETR significantly outperforms YOLOS-Bas which also adopts a transformer encoder-only structure, in terms of both model efficiency and detection accuracy. This is mainly due to the inductive bias introduced in our model as Deformable DETR and the prompt-inspired task adaptor for more effective fine-tuning, which better bridges the gap between pre-training and fine-tuning.
The last row shows the result of our FP-DETR Base fine-tuned from ImageNet-21 pre-training, de- noted FP-DETR-Bas Base As can be seen, FP-DETE can benefit from pre-training at larger scale, and FP-DETR-Bass 40M parameters. In Appendix A.3, also show the result of Deformable DETR with ResNet-50 backbone pre-trained on ImageNet-21k. Interestingly. the result slightly worse than Deformable DETR with ImageNet-Ik 1k pre-trained backbone. We conjuncture that only pre-training part of the model makes hard for the randomly initialized transformer adapts to the pre-trained CNN back- bone, especially when the backbone has already been well-trained.
with 36M parameters matches the performance of Deformable DETR with
Table Ablations on the task adaptor on COCO 2017 val set.
Task Adaptor
Shared N/A
AP
AP
AP
APs
APM
AP
Params
w/o Bi-LSTM
32.8
50.9
35.2
18.2
35.8
43.6
11M
34.6
52.9
37.3
19.1
38.1
45.4
11M
Bi-LSTM
34.9
53.0
37.5
20.2
38.0
46.9
11M
Self-attention
36.8
56.0
39.5
20.9
39.8
49.1
12M
Self-attention
37.2
56.5
40.4
21.7
40.0
48.6
11M
4.3 ABLATIONS ON THE TASK ADAPTOR
To gain better understanding of the effectiveness of the proposed task adaptor, perform ablation studies on the lite version of FP-DETR, as shown in Table We have the following observations. First, removing the task adaptor results in significant drops in model performance. Without the task adaptor, the model can not well capture the inter-object relationships, which is crucial for re- moving duplicates and improving object recognition. Second, the task adaptor instantiated with a bidirectional-LSTT layer also helps to adapt the pre-trained model to the downstream task. but works slightly worse than self-attention layer. We conjuncture that the bidirectional-I.STM is

---

Published as a conference paper at ICLR 2022
slightly worse modeling long-range dependency compared the self-attention layer, especially when the number query embeddings reaches 300. By default, we adopt the self-attention layer for the task adaptor. In this way, FP-DETR is a pure transformer encoder structure. Third, in- terestingly. a single task adaptor shared by visual prompts from different transformer encoder layers performs slightly better than specialized task adaptors for each layer. Since the task adaptor is trained from scratch on the downstream task, we conjuncture that the shared task adaptor sees diverse data from different levels compared the non-shared ones, and is thus more sufficiently trained. These experiments validate the effectiveness of our task adaptor on modeling object relationships, which crucial for object detection.
Table 4: Comparison of model robustness the COCO-C dataset.
Noise Shot
Blur
Method
Mean
Gauss
Impul
Defocus
Glass
Motion
Zoom
DETR UP-DETR
19.1
16.8
16.5
12.7
20.0
13.8
17.7
6.5
21.6
22.1
22.2
18.7
20.1
13.4
18.3
6.9
Conditional DETR
20.3
18.6
18.6
16.0
20.5
14.8
18.4
7.1
DeformableDDETR
20.7
19.8
19.6
16.4
20.1
13.8
17.9
6.6
YOLOS-S
18.9
15.3
15.1
14.0
19.4
15.7
20.2
7.1
FP-DETR-lite
18.9
17.0
17.0
15.2
18.6
15.1
17.5
6.5
FP-DETR-Smal
22.8
20.3
20.3
18.1
22.3
18.3
21.2
8.0
FP-DETR-Base
23.7
22.5
22.7
20.4
22.9
18.9
21.7
8.0
Weather
Digital
Method
Snow
Frost
Fog
Bright
Contrast
Elastic
Pixel
JPEG
DETR
16.4
21.4
29.0
35.0
18.8
23.2
18.3
19.8
UP-DETR
19.3
24.2
31.6
36.9
21.4
23.0
20.7
24.6
Conditional DETR
30.0
34.7
24.0
19.4
20.5
Deformable-DEi
18.3
23.6
31.7
37.0
21.2
24.0
19.3
21.0
YOLOS-S
17.7
21.2
26.5
29.5
19.7
22.8
17.1
21.8
FP-DETR-Lite
18.2
22.7
28.4
32.5
20.5
21.3
14.8
18.2
FP-DETR-Smal
22.9
27.1
33.1
37.3
25.5
20.2
23.1
OP-DETR-Baad
22.4
27.1
33.4
38.0
25.3
26.1
20.0
25.6
Table Comparison of model generalization on the small-size Cityscapes dataset.
Method
Structure
Epochs
AP
APso
AP3s
APs
APM
APt.
Params
DETR
Enc-Dec
500
15.9
34.8
12.7
2.9
13.5
33.8-2.1
41M
UP-DETR
Ene-Dee
300
23.8
45.7
20.8
4.0
20.3
46.6a19
41M
Conditional DETR
Ene-Dec
150
21.1
42.7
18.8
3.610a
19.8
41.121.0
44M
Deformable DETR
Enc-Dec
50
27.3
49.2
26.3
8.740.2
28.2+n.
45.7
40M
YOLOS-S
Fnc
150
9.810.1
25.3
6.1
1.910.2
8. 8.1
20.7
3IM
FP-DETR-Lic
Enc
50
26.7
49.1
25.7
9.710.7
28.6.a.
42.8.0.7
11M
TP-DETR-Smal
Enc
28. 28.620.2
52.020.6
26.920
10.1
30.220.6
46.3
24M
FP-DETR-
Enc
29.6
53.6
28.4+0.
11.2
30.9
47.4.1.
4.4 ROBUSTNESS TO COMMON CORRUPTIONS
Model robustness (He & Tao, 2020) is critical trustworthy AI applications like autonomous driv- ing. To this end, we evaluate the object detectors' robustness against common corruptions on COCO- C (Michaelis al., 2019). As shown in Table all detectors suffer significant performance drops under this rigorous condition. However, FP-DETR suffers the least performance drop compared to existing detection transformers. Notably, FP-DETR corruptions, though it has comparable performance Deformable DETR the clean COCO 2017 val set. This is because the pre- pre-training on ImageNet helps FP-DETR Iearn more generalizable representation. This is coherent with the observation in Hendrycks et al. (2019).
performs best on 14 out of 15 types of
The first two rows in Figure provide some qualitative results of our FP-DETR-BB Base on the COCO- C dataset. As be seen, the images significantly degenerated due the existence of various corruptions, like elastic transform, zoom blur, fog, etc. However, FP-DETR Base still manages to produce plausible results under low visibility and large distortion, which manifests its robustness.
8

---

Published as a conference paper at ICLR 2022
STOP
Figure 3: Qualitative detection results of FP-DETR on the COCO-C and Cityscapes datasets are shown in the first two rows and the last row, respectively.
4.5 GENERALIZATION SMALL-SIZE DATASET
In real-world applications, collecting large amount of data is often infeasible. As result, the models are required to perform well by training on only limited data. Table 5 shows the result of models fine-tuned on the Cityscapes dataset with only 2.975 training images. All models are trained with a batch size of to guarantée enough training iterations. The results are averaged over five repeated runs with different random seeds. As can seen, most detection transformers. including DETR, UP-DETR, Conditional DETR, and YOLOS-B, perform poorly under this condition. This partly due to the lack of inductive bias those models. As result, the model requires large amount of data to learn the sparsity that is beneficial for object detection on images. Deformable DETR performs better, thanks to the sparsity inductive bias introduced the deformable trans- former. However, still performs worse than our FP-DETR, since the transformer in Deformable DETR is trained from scratch. Specifically, both FP-DETR-Bas and LEDERR-Smaa outperform Deformable DETR with more parameters. Notably, FP-DETR-Lit with only 1 11M parameters matches the performance of 40M Deformable DETR, attributing to the end-to-end fully pre-training of the transformer.
The last row in Figure 3 shows some qualitative results of our FP-DETR-Bass on the Cityscapes dataset. As be seen, FP-DETR quickly adapts to the small-size dataset, and produces accurate object detection results, even for the small cars in the distant area. These results demonstrate the generalization ability of our method.
5 CONCLUSION
In this paper, we propose a novel detection transformer named FP-DETR that can take advantage of pre-training on upstream datasets to enable better robustness and generalization. FP-DETR contains simple encoder-only structure for fully pre-training and can be smoothly fine-tuned for object detection via a task adapter. leverages query positional embeddings as visual prompts to help the model attend to the target area (prompting) and recognize the object, effectively mitigating the gap between the upstream ImageNet classification task and the downstream object detection task. Experiments show that FP-DETR not only achieves competitive performance the challenging COCO dataset, but also gains better robustness against common corruptions and generalization to small-size datasets.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work is supported by National Key R&D Program of China under Grant 2020AAA0105701, National Natural Science Foundation of China (NSFC) under Grants 61872327, and Major Special Science and Technology Project of Anhui (No. 012223665049). Dr Jing Zhang is supported by ARC FL-170100117.
ETHICS STATEMENT
We acknowledge that all co-authors of this work have read and commit adhering the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
The authors strive to ensure the reproducibility of the experimental results. Specifically, implemen- tation details are provided in Section 4.1, Section A.I, and Section A.3. The related studies with the same experimental setup carefully cited and referred to. Moreover, the code will be made publicly available to ensure reproducibility.
REFERENCES
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Zhe Chen, Jing Zhang. and Dacheng Tao. Recurrent glimpse-based decoder for detection with transformer. arXiv preprint arXiv:2112.04632, 2021.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2213-3223, 2016.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. In Proceedings of the IEEE conference computer vision and pattern recognition, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. IEEE, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480) 2018.
Terrance VrVries and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint arXiv: 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbomn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
10

---

Published as a conference paper at ICLR 2022
Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. arXiv preprint arXiv:2106.0066 2021.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint 2rXv:22012.15725 2020.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for ac- curate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580-587, 2014.
Fengxiang He and Dacheng Tao. Recent advances in deep learning theory. arXiv preprint arXiv:2012.10931, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4918-4927,, 2019.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In International Conference on Machine Learning, pp. 2712-2721. PMLR, 2019.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
Chen-Yu Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply- supervised nets. In Artificial intelligence and statistics, pp. 562-570. PMLR, 2015.
Tsung-X Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. arXiv preprint arXiv:2107.13586 2021a.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pp. 21-37. Springer, 2016.
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2103. 10385, 2021b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030. 2021c.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018
Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jing- dong Wang. Conditional detr for fast training convergence. In Proceedings of the IEEE interna- lional conference computer vision, 2021.
Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexan- der S. Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detec- tion: Autonomous driving when winter is coming. arXiv preprint arXiv:1907.07484 2019.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.

---

Published as a conference paper at ICLR 2022
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, 3966):1177-1143) 2016.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21K pretraining for the masses. arXiv preprint arXiv:2 10972, 2021.
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.071 2020.
Yongliang Shen, Xinyin Ma, Zeqi Tan, Shuai Zhang, Wen Wang, and Weiming Lu. Locate and label: A two-stage identifier for nested named entity recognition. In ACL, 2021.
Lestie N Smith. A disciplined approach neural network hyper-parameters:: Part 1-learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv:1803.09930 2018.
Christian Szegedy. Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 42, 2021b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Conference Neural In- formation Processing Systems, 2017.
Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. Advances in Neural Information Processing Systems, 34, 2021.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: De- formable transformers for end-to-end object detection. In International Conference on Learning and Representations, 2020.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 STRUCTURE OF THE MULTI-SCALE TOKENIZER
Our lightweight multi-scale tokenizer extracts feature maps of 4 different levels to construct the input sequences to the encoder-only transformer. Different from existing detection transformers, which use a sophisticated ResNet-50 backbone for feature extraction, our multi-scale tokenizer simply downsamples the input image to desired feature resolution. Specifically, we perform non- overlapping patch embedding, similar to the seminal ViT (Dosovitskiy et al., 2020). The only differ- ence is that we perform multi-scale tokenization and adopt a hierarchical structure to progressively embed the tokens of large resolution. The detailed structure of our multi-scale tokenizer is shown in Table 6. The dimension D' is 192, 384, and 384, respectively for our Lite, Small, and Base model. After patch embedding, the 2D sequence is flattened to ID sequence and further projected to tokens with dimension D.
Table 6: The architecture of the lightweight multi-scale tokenizer.
Multi-scal Tokenizer
Conv 8 x 8 x D', stride 8, pad 0
LayerNorm
Conv 2 2 x D', stride 2, pad 0
LayerNorm
Conv 2 2 x D', stride 2, pad 0 LayerNorm
Conv 2 2 x D', stride 2, pad 0 LayerNorm
A.2 EXPLORING RANODER-DECODEE STRUCTURE
While most detection transformers contain an encoder-decoder transformer, existing study on vi- sion transformers extensively explores the encoder-only transformer for image classification. This results in discrepancy between the upstream ImageNet classification task and the downstream object detection task. To mitigate this problem, we first explore pre-training the encoder-decoder transformer on the ImageNet classification task, using existing pre-training technologies (Touvron transformer structure,
et al., 2021a). Specifically, we pre-traie train and fine-tune an encoder-decodee denoted as En-Ene on the ImageNet-lk Ik dataset, follow the same implementation in 4.1. The model follows the design of our FP-DETR-LEN replaced by 6 decoder layers and the task adaptor is removed. In each decoder layer, standard self-attention layer (Vaswani et al., 2017) and multi-scale deformable attention layer (Zhu et al., 2020) are applied sequentially, following Deformable DETR. Two variants of FP-EncDec are con- sidered: (1) pre-training with a single class token, follow the common practice in training vision transformers (Dosovitskiy et al., 2020): and (2) pre-training with 300 class tokens, which equals the number of object query embeddings for fine-tuning. The 300 class tokens are pooled into a single class token before final classification.
except the last 6 layers of self-attention layers are
Table 7: Comparison of encoder-decoder transformers and encoder-only transformer for pre-training and fine-tuning.
Token Number
Method
Structure
Acc@ 1
AP
AP30 AP1s
APs
APM APL
Params
FP-EncDec
Enc-Dec
72.0
33.1
51.8
35.7
17.9 35.9 43.4
12M
FP-EncDec
Enc-Dec
300
76.4
35.2
54.0
37.9
20.9
38.1
47.0
12M
DP-DETR-Lite
Enc
77.3
37.2
56.5 40.4
21.7 40.0 48.6
11M
The results for both pre- tre-training on ImageNet- 1k and fine-tuning on COCO 2017 dataset are shown in Table 7. As can be seen, pre-training the transformer decoder with single class token performs poorly on the ImageNet classification task, since both the self-attention layers and the projections on the class token in the cross-attentin layer are trained single class token, which could easily
13

---

Published as a conference paper at ICLR 2022
lead to overfitting. The low pre-training accuracy also limits the model's detection performance the downstream task. By contrast, FP-EncDec pre-trained with 300 class tokens performs better on ImageNet- 1k classification, the top-1 accuracy is 4.4 higher than the single class token counter- part, since the decoder is more sufficiently trained. However, it is still inferior to our encoder-only FP-DETR-Lite. We conjuncture that this is caused by (1) existing training techniques for vision transformers have been heavily tuned towards the encoder-only structure, which can be sub-optimal for the encoder-decoder transformer; and (2) the discrepancy between upstream and downstream tasks also degenerates the model's 's performance on object detection, even if the decoder has been pre-trained. We expect these findings may provide useful insights the community to rethink the current paradigm of pre-training vision transformers, and pay more attention to the pre-training of encoder-decoder transformers.
A.3 PRE-TRAINING ON IMAGENET-2IK
For ImageNet-21k pre rre-training.
follow the pipeline in Ridnik et al. (2021). The ImageNet-21k is first pre-processed by three steps: (1) cleaning invalid classes; (2) validation split; and (3) image re- sizing. Afterward, the model is trained the processed dataset (ImageNet-2IK-PP using the seman- tic softmax training (Ridnik et al., 2021). Specifically, our model is pre-trained on ImageNet-22 Ik-P with a batch size of 4096 for 80 epochs. The model is initialized from ImageNet- IK pre-trained weights. The learning rate is initialized as 3e-4 and scheduled using one-cycle policy (Smith, 2018), 8). RandAugment (Cubuk et al., 2020), Cutout (DeVries & Taylor, 2017), Label -mmoothig (Szegedy et al., 2016), and True-weight-decay (Loshchilov & Hutter, 2018) are adopted for regularization. For more details on pre-training, please refer to Ridnik et al. (2021).
For fine-tuning FP-DETR further, we adopt the same implementation details in 4.1. For a fair comparison, we also finetuned Deformable DETR (Zhu et al., 2020) with an ImageNet-21k-2 pre- trained ResNet-50 backbone. The ResNet-50 pre-trained weight is taken from the official release? of Ridnik et al. (2021).
Table 8: Comparision models fine-tuned from ImageNet- Ik and ImageNet-21k pre-trained weights. f indicates the model is pre-trained on ImageNet-2Ik.
Method
Structure
Bone
Epochs
AP
APso
AP15
APs
APM
AP,
params
Deformable DETR
Enc-Dee
R50
50
43,8
62.6
47.7
26.4
47.1
58.0
40M
Deformable DETR
Enc-Dec
R50
50
42.9
62.1
46.6
25.1
46.3
57.7
40M
FP-DETR-Base
Enc
50
43.3
63.9
47.7
27.5
46.1
57.0
36M
SP-DETR-Basc
Enc
50
43.7
64.1
47.8
26.5
46.7
58.2
36M
The result is shown in Table 8. As can be seen, our FP-DETR can benefit from fully pre-training on the larger-scale ImageNet-21k dataset well as fine-tuning with the task adaptor. Counter- intuitively, the performance of Deformable DETR degenerates when using the weights pre-trained on ImageNet-2Ik. We conjuncture that only pre-training the CNN backbone of the model makes it difficult for the randomly initialized transformer to adapt to the pre-trained backbone, especially when the backbone has already been well-trained. More research efforts should be made to further dig into this problem.
Phttps:// : thub. bababbbaaiii 11/ ImageNet221
14


Reviews:
[{"review_id": "rPQvQuwi2fX", "reviewer": "Reviewer_MjUh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-organized and easy to follow. \n\nStrength: \n+ Encoder-only transformer design without the need of extra visual backbone\n+ Lower parameters compared to concurrent works\n\nWeakness:\n- The effeteness of pre-training is not well justified. The paper claims \"large-scale pre-training on ImageNet helps FP-DETR learn more generalizable representation.\" However, the author only conducts experiments on imagenet-1k, which barely counts a large-scale pre-training. In order to further prove the benefits, it would be best to re-run experiments on imagenet-22k and show the continuous improvement .\n- The necessary of deformable convolution layer.  The encoder-only transformer design would be more appreciated if not using deformable convolution. How big will the impact be if trained without deformable convolution? \n- Latency. Since author focuses on small models, it would be best to report latency.  \n- The robustness analysis is not convincing and not related to the main contribution. The author should include more ablation studies on hyper-parameters instead (such as #encoder, #dim, #token, etc)", "summary_of_the_paper": "This paper argues that recently developed detection transformers didn't employ pre-training on transformer layers and not benefit from pretraining. Hence, it proposes FP-DETR to fully pretrain transformer on image classification and then finetune it for object detection by changing the task adapter. Compared to previous methods, it only uses encoder-only transformers. Further experiments show reasonable performances and robustness analysis. ", "summary_of_the_review": "This paper shares some insights on the proposed FP-DETR, but it lacks comprehensive experiments to justify the claims. After reading the rebuttal and extra ablation studies, most of my questions have been answered. I believe this paper has some merit and worth accepting for a poster. Hence, I have raised my score.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kRK6ufQYyI", "reviewer": "Reviewer_Jj3q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths \n\n+ The submission is clearly written. The motivation and technical details are easy to understand.\n\n+ The authors find the similarity between textual prompt filling task in NLP and object detection and propose a solution to mitigate the gap.\n\n+ The proposed network (FP-DETR) generalizes well on two small datasets (COCO-C and Cityscapes)\n\n## Weakness\n\n- The technical novelty is not fully verified. According to the section 3, the authors first pre-train the encoder part on the ImageNet. In Figure 1, the pre-training leverages class token as a placeholder which could later be replaced by visual prompt information. However, the class token domain has a large gap between the visual domain with visual contents. As a solution, the authors propose a task adapter (which is an self-attention layer or a Bi-LSTM) to mitigate the gap. According to the sentence above eq5, the major function of the prompt encoder is to `improve the modeling of the inter-object relationships`. In such sense, it would be confusing whether the task-adapter is used to better model the object interaction, or to adapt the pre-training image classification task to object detection task. I would suggest the authors to calculate some embedding distance to show that task-adapter-processed embeddings' distances to class token embeddings in pre-training are smaller than pre-task-adaptor embeddings' distances. If the distances are smaller, the task adapter is verified that it could be used to transform the new query embeddings from visual prompts to class token embedding spaces, so that the FP-DETR pre-training could be easily adapted to object detection task. \n\n- The generalization ability is inconsistent with pre-training datasets' sizes. According to the findings in section 4.2, pre-training on ImageNet-21k has lower performance than that from ImageNet-1k. The authors conjectured that the random initialization of transformer may be the cause. However, if the pre-training strategy cannot be scaled to larger datasets to further improve the performance, this method may have smaller scope of application. It may need more efforts to solve the scalability problem. ", "summary_of_the_paper": "This paper proposes a new method of pre-training transformer based object detector (FP-DETR). Compared to previous methods, the authors propose to pre-train DETR's encoder with ConvNet backbones on ImageNet. During the finetuning stage, the authors proposes a task adapter to map the visual prompt to classification token domain to mitigate the gap between image classification and detect object tasks. Experiments on MS-COCO show that the FP-DETR achieves competitive performance and generalizes well on small scale datasets. ", "summary_of_the_review": "This submission provides a novel way of pre-training DETR's encoder part on image classification task first, and then finetuning DETR on object detection task. However, there is one novelty not fully verified. Besides, the method may have some problem in scalability. Thus, I would wait for authors' response before making the decision. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_Mo_aTMzXgk", "reviewer": "Reviewer_s2Cu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I like the paper but I feel hesitant to accept this paper at its current status . While it is no secret that one can improve model robustness and generalizability through pre-training, I do believe that the method contributes values to the community with an architecture that is designed to better utilize imageNet classification pre-training and close the gap between classification and detection in the fine-tuning stage. However, I found the current experimental protocol is not solid enough to support some of the claimed contributions. I am happy to change my rating if the author could provide justification through discussion or additional experiments. \n\nPros:\n\n- The paper is well written and easy to read. The intuition behind the algorithm design is sufficiently discussed. The technical session is described in detail.\n- I like the idea of decreasing the model discrepancy between upstream classification tasks and downstream detection tasks through an encoder only transformer. \n- The intuition to provide location attention and inter-object relationship attention is interesting\n- The overall validity of the algorithm is backed by performance close to SOTA on COCO and outstanding in Cityscapes. \n\nCons:\n- The main concern I have in the experiment is whether we can decouple the advantage of the proposed architecture or the fact that the Transformer encoder is being pretrained when compared to other SOTA (Table 2). E.g. What’s the performance of the proposed algorithm with normal ResNet backbones? What’s the performance of Deformable Tranformer with Encoder pretrained？\n- The ablation analysis on the lite version (Table 3) is not as strong as one in the base version. It is easier to improve on top of a weak performance.\n", "summary_of_the_paper": "This paper proposes a novel transformer-based detection algorithm (FP-DETR) that could benefit from large-scale pre-training for model robustness and generalization on small datasets. FP-DETR consists of a encoder only transformer on top of a set of lightweight feature extracting layers for pre-training. It is then fine-tuned with a task-adapter that helps the model attending to object location as well as be aware of inter-object relationship for detection task. Evaluation is done on both large scale dataset COCO and relatively small domain CityScapes with ablation analysis on different system modules and design choices. The result indicates that FP-DETR presents better generalization capabilities on smaller dataset with on-part performance compared to other SOTA in general settings.\n", "summary_of_the_review": "The paper proposed a novel architecture to empower a transformer-based object detector with smoother pre-training and fine-tuning paradigm. The paper is well-written and provides interesting discussion on its intuition and new perspective on positional embeddings in this context. However, pre-training helps improve the model is not a novel theory by itself. The insufficient experimental comparison decreases the strength of the paper. I am looking forward to seeing the rebuttal and willing to change my rating accordingly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VfawEJ4rwy", "reviewer": "Reviewer_JhDk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- This work introduced the ‘prompting’ to detection task. By mitigating the gap between pre-training and finetuning task, it shows a better performance on COCO dataset. It is also better on robustness and gereralization to small-size dataset. Authors conducted ablation studies to verify the different settings. \n\n- In Table 7, it shows the FP-EncDec with 300 tokens performs better than that with only 1 token on pre-training and fine-tuning tasks. I wonder if we also apply 300 tokens to FP-DETR on the pretraining, will performance of pretraining and downstream tasks become better as well? It is interesting to see it as it may provide more flexibility to different downstream tasks.\n- In table 2, is it possible to have an ablation study in which we have FP-DETR with ResNet-50 backbone? \n\n- Minor: it seems there is a typo -  PT-DETR in the introduction? \n", "summary_of_the_paper": "This work proposes a new transformer framework for detection task by masking better use of features in pretraining phase. Inspired by ‘prompting’ in the NLP task, it treats query positional embeddings as visual prompts to better localize objects. With the similar architecture design, it mitigates the discrepancy between pretraining and finetuning stages. Task adaptor is introduced to capture inter-object relationships. The proposed framework achieves competitive performance and becomes more robust and generalizable.", "summary_of_the_review": "Although the idea is borrowed from the NLP field, applying it to the CV field may be helpful for the community. It provides a new perspective towards the mitigate the discrepancy between pre-training and finetuning. At the current stage, I prefer to accept this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary: Authors present an approach for transformer based object detection that “fully pretrains” the encoder structure of the transformer, and drops the pretrained convolutional backbone used in other works. 

Pros:
- Eliminates need of extra visual backbone
- Fewer parameters than other works
- Achieves competitive performance, especially controlling for model size

Cons:
- Multiple reviewers raised concerns about authors only evaluating their approach with pretraining from ImageNet 1K, which is not considered large scale. Authors replied with new experimental data including pretraining from ImageNet 22k, which improved results. 
- Multiple reviewers raised concerns about the need for more ablation experiments, which the authors addressed.

Reviewer scores lean toward accept. Those that lean toward reject raised issues that the authors appear to have addressed sufficiently. Not all reviewers have replied to authors, though understood at least some reviewers on this paper are on end-of-year time-off.

Overall recommendation: accept.
Decision: Accept (Poster)