DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
Authors: Hao Zhang
Feng Li
Shilong Liu
Lei Zhang
Hang Su
Jun Zhu
Lionel Ni
Heung-Yeung Shum
Venue: ICLR 2023 poster (2023)
Keywords: Object Detection
Detection Transformer
End-to-End Detector

Abstract: We present  DINO (DETR with Improved deNoising anchOr boxes), a strong end-to-end object detector. DINO improves over previous DETR-like models in performance and efficiency by using a contrastive way for denoising training, a look forward twice scheme for box prediction, and a mixed query selection method for anchor initialization. DINO achieves 49.4AP in 12 epochs and 51.3AP in 24 epochs on COCO with a ResNet-50 backbone and multi-scale features, yielding a significant improvement of +6.0AP and +2.7AP, respectively, compared to DN-DETR, the previous best DETR-like model. DINO scales well in both model size and data size. Without bells and whistles, after pre-training on the Objects365 dataset with a SwinL backbone, DINO obtains the best results on both COCO val2017 (63.2AP) and test-dev (63.3AP) with model size under 1 billion parameters. Compared to other models on the leaderboard, DINO significantly reduces its model size and pre-training data size while achieving better results. The code will be available.

Under review as a conference paper at ICLR 2023
DINO: DETR WITH IMPROVED DENOISING ANCHOR BOXES FOR END-TO-END OBJECT DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
We present DINO (DETR with Improved deNoising anchOr boxes), strong end- to-end object detector. DINO improves over previous DETR-like models per- formance and efficiency by using contrastive way for denoising training, look forward twice scheme for box prediction, and a mixed query selection method for anchor initialization. DINO achieves 49. 49.4AP in 12 epochs and 51.3AP in 24 epochs on COCO with a ResNet-50 backbone and multi-scale features, yielding a significant improvement of +6.0AP and +2.7AP, respectively, compared to DN- DETR, the previous best DETR-like model. DINO scales well in both model size and data size. Without bells and whistles, after pre-training on the Objects365 dataset with a SwinL backbone, DINO obtains the best results on both COCO va12017 (63.2AP) and test-dev (63.3AP) with model size under billion parameters. Compared to other models on the leaderboard, DINO achieves bet- ter results with smaller model size and pre-training data size. The code will be available.
Dyta
ES-DETR
610
$2.0
40
AP 10 +22001
Yeeila HNIAA
Dyfled
Farmon
12
24 366
Epuck
100
284
607
Medel stzc (M)
(b)
2000
Figure 1: AP on COCO compared with other detection models. (a) Comparison to models with ResNet-50 backbone w.r.t. training epochs. Models marked with DC5 use dilated larger resolution feature map. Other models use multi-scale features. (b) Comparison to SOTA models w.r.t. pre- training data size and model size. SOTA models are from the COCO test-dev leaderboard. In the legend we list the backbone pre-training data size (first number) and detection pre-training data size (second number). mcans the data size is not disclosed.
INTRODUCTION
Object detection is fundamental task in computer vision. Remarkable progress has been accom- plished by classical convolution-based object detection algorithms (Ren et al., 2017 Tian et al.) 2019 Lin et al., 2020 [Bochkosskiy et al.) 2020 Ge et al.) (021). Despite that such algorithms normally include hand-designed components like anchor generation and non non-maximum suppression
(NMS), they yield the best detection models such DyHead (Dai al. 2021a), Swin (Liu et al.
2021b and SwinV2 (Liu et al. 2021a) with HTC++ (Chen et al. 2019a). as evidenced on the COCO
test-dev leaderboard (pap).
In contrast to classical detection algorithms, DETR (Carion et al., 2020 is a novel Transformer- based detection algorithm. It eliminates the need of hand-designed components and achieves com-

---

Under review as a conference paper at ICLR 2023
parable performance with optimized classical detectors like Faster RCNN (enn et al. 2017). Dif- ferent from previous detectors, DETR models object detection as a set prediction task and assigns labels by bipartite graph matching. It leverages learnable queries to probe the existence of objects and combine features from an image feature map like soft ROI pooling (Liu et al. 2022).
Despite its promising performance, it converges slow and the meaning of queries is unclear. To address such problems, many methods have been proposed, such as introducing deformable atten- tion (Zhu al. 2021), decoupling positional and content information (Meng al.) (222I), providing spatial priors (Gao et al. 202T] Yao et al. 2021] Wang et al. (02T), etc. Recently, DAB-DETR (Liu al., 2022) proposes formulate DETR queries as dynamic anchor boxes (DAB), which bridges the gap between classical anchor-based detectors and DETR- like ones. DN-DETR (Li et al.) (202))
further accelerate convergence by introducing denoising (DN) technique. These improvements promote the development of DETR-like models, while it remains not on the list of first-choice de- tectors in the field.
The best detection models nowadays are based improved classical detectors like DyHead DDai itt al. 2021b) and HTC (Chen et al. 2019a). For example, the best result presented in Swin (Liu et al. a was trained with the HTC++ (Chen et al. 2019a Liu et al. 2021b) framework. Two main reasons contribute to the phenomenon: 1) Previous DETR-like models are inferior to the im- proved classical detectors. Most classical detectors have been well studied and highly optimized, leading to a better performance compared with the newly developed DETR-like models. 2) The per- formance of DETR-like model has not been tested large backbone with large-scale pre-training data. We aim to address both concerns in this paper.
Specifically, by improving the denoising training, query initialization, and box prediction, we design a new DETR-like model based on DN-DETR, DAB-DETR, and Deformable DETR. We name our model as DINO (DETR with Improved deNoising anchOr box). As shown in Fig. | the comparison on COCO shows the superior performance of DINO, In particular, DINO demonstrates strong performance, setting new record of 63.3 AP for models with less than billion parameters on the COCO test-dev leaderboard (pap).
As DETR-like model, DINO contains a backbone, multi- Transformer encoder, a multi-layer Transformer decoder, and multiple prediction heads. Following DAB-DETE we formulate queries in decoder as dynamic anchor boxes and refine them step-by-step across decoder layers. Following DN-DETR, we add ground truth labels and boxes with noises into the Transformer decoder layers to help stabilize bipartite matching during training. We also adopt deformable attention (Zhu et al.) 2021) for its computational efficiency. Moreover, we propose three new methods as follows. First, to reduce duplicate predictions, we propose a contrastive denoising training by adding both positive and negative samples of the same ground truth at the same time. After adding two different noises to the same ground truth box, we mark the box with a smaller noise as positive and the other as negative. The contrastive denoising training helps the model to predict more precise boxes and avoid duplicate outputs of the same target. Second, to overcome the shortsightedness of refining boxes in each decoder layer, which is greedy way proposed in Deformable DETR, while keeping the advantages of fast convergence, we propose new look forward twice scheme to correct the updated parameters with gradients from later layers. Third, the dynamic anchor box formulation of queries links DETR-like models with classical two-stage models. Hence we propose a mixed query selection method, which helps better initialize the queries. We select initial anchor boxes as positional queries from the output of the encoder, similar to (Zhu et al.) 2021 Yao et al. 2021 l. However, we leave the content queries learnable queries aligned with CDN part where queries are also learnable queries which encourages the first decoder layer to focus the spatial prior.
We validate the effectiveness of DINO with extensive experiments on the COCO (Lin et al. 2014) detection benchmarks. As shown in Fig. [ DINO achieves 49.4AP in 12 epochs and 51.3AP in 24 epochs with ResNet-50 multi-scale features, yielding a significant improvement of +6.0AP and +2.7AP, respectively, compared to the previous best DETR-like model DN-DETR. In addition, DINO scales well in both model size and data size. After pre-training on the Objects365 (Shao) al. 2019) data set with a SwinL (Liu al. (021bb backbone, DINO achieves impressive results on both COCO val2017 (63.2AP) and test-der (63.3AP) benchmarks, as shown in Table 4 Our DINO reduces the model size to 1/15 compared to Swin V2-G (Liu al. 2021a). Moreover, DINO outperforms Florence (Yuan et al. 2021) with only 1/60 backbone pre- tre-trainig dataset and 1/5 detection pre-training dataset.
2

---

Under review as a conference paper at ICLR 2023
To summarize, our contributions are three-fold. 1) We design new end-to-end DETR-like object detector with several novel techniques, including contrastive denoising training, look forward twice, and mixed query selection for different parts of the DINO model. 2) We conduct intensive abla- tion studies to validate the effectiveness of different design choices in DINO. As a result, DINO achieves 49.4AP in 12 epochs and 51.3AP in 24 epochs with ResNet-50 and multi-scale features, significantly outperforming the previous best DETR-like model DN-DETR 3) We show that, with- out bells and whistles, DINO can achieve the best performance on public benchmarks with model size under 1 billion parameters. After pre-training on the Objects365 (Shao et al. 2019) dataset with a SwinL (Liu al. 2021b) backbone, DINO achieves 63.2AP on COCO va12017 and 63.3AP on COCO cest-dev benchmarks.
2 RELATED WORK
Early convolution- based object detectors are either two-stage stage models, based on hand-crafted anchors or reference points. Two-stage models (Ren et al.) 2015 He al. (017) usually use an region proposal network (RPN) (Ren al. (0155) to propose potential boxes, which are then refined in the second stage. One-stage models (Redmon & Farhadi 2017 2018) directly output offsets relative to predefined anchors. Recently, some convolution- based models such as HTC++ (Chen el al., 2019a) and Dyhead (Dai et al.) 2021a) have achieved top performance on the COCO 2017 (Lin et al. 2014). ). The performance of convolution-base models, however, rely on the way they generate anchors and need hand-designed components like NMS.
Classical Object Detectors:
DETR and Its Variants: Carion et al. (Carion al., 2020) proposed Transformer- end- to-end object detector named DETR (DEtection TRansformer) without using hand-designed com- ponents like anchor design and NMS. Many follow-up papers have attempted to address the slow training convergence issue of DETR introduced by decoder cross-attention. For instance, Dai et al. (Dai et al. 202 proposed a dynamic decoder to focus on important regions from multiple feature levels. Another line of works is towards deeper understanding of decoder queries in DETR. Many papers associate queries with spatial position from different perspectives. Deformable DETR (Zhu et al. (021)) predicts 2D anchor points and designs deformable attention module that only attends to certain sampling points around a reference point. DAB-DETR (Liu et al. 2022) further extends 2D anchor points to 4D anchor box coordinates represent queries and dynamically update boxes in each decoder layer. Recently, DN-DETR (Li et al. 2022) introduces a denoising training method to speed up DETR training. It feeds noise-added ground-truth labels and boxes into the decoder and trains the model to reconstruct the original ones. Our work is based on DAB-DET and DN-DETR, and also adopts deformable attention for its computational efficiency.
Large-scale Pre-training for Object Detection: The best performing detectors nowadays are mostly achieved with large backbones pre-trained on large-scale data. For example, Swin V2 (Liu et al. 2021a) extends its backbone size to 3.0 billion parameters and pre-trains its models with 70M privately collected images. Florence (Yuan et al. (02T)) first pre-trains its backbone with 900M privately curated image-text pairs and then pre-trains its detector with 9M images with annotated or pseudo boxes. In contrast, DINO achieves better results with a publicly available SwinL (Liu et al.)
2021b) backbone and public dataset Objects365 (Shao et al., 2019) (1.7M annotated images) only.
DINO: DETR WITH IMPROVED DENOISING ANCHOR BOXES
3.1 PRELIMINARIES
As studied in Conditional DETR (Meng et al. 2021 and DAB-DETR (Liu et al. 2022). queries in DETR (Carion al. 2020) are formed by two parts: a positional part and content part, which are referred to as positional queries and content queries in this paper. DAB-DETR explicitly formulates each positional query in DETR as a 4D anchor box (I, y, w, h). where x and y are the center co- ordinates of the box and and h correspond to its width and height. Such an explicit anchor box formulation makes it easy to dynamically refine anchor boxes layer by layer in the decoder.
2022 introduces denoising (DN) training method to accelerate the train- DN-DETR Li et al.
ing convergence of DETR-like models. It shows that the slow convergence problem in DETR is caused by the instability of bipartite matching. To mitigate this problem, DN-DETR proposes to
3

---

Under review as a conference paper at ICLR 2023
Query Selection
Matching
CDN
Multi Stale
Features
Key& Value
Encoder Layers
Decoder 1 Layers x M
Leamaibe Qontent
Flatton
Positional
Embeddiigs
GT-Nuise
Init
Figure 2: The framework of our proposed DINO model. Our improvements are mainly in the Trans- former encoder and decoder. The top-K encoder features in the last layer are selected to initialize the positional queries for the Transformer decoder. Our decoder also contains Contrastive DeNoising (CDN) part with both positive and negative examples.
additionally feed noised ground ground-truth (GT) labels and boxes into the Transformer decoder and train the model to reconstruct the ground-truth ones. The noise (Ax, Ay, Aw, Ah) is constrained by Ave
|Ar| < 2 |Ayl
2 AAt |Aw| < Aw, and |Ayl < Ah, where (z, y, W, h) denotes a GT box and
to control the scale of noise. Since DN-DETR view decoder queries as anchors, a noised GT box can be viewed
D is a hyper-paramen
special anchor with GT box nearby as À is usually small. In addition to the orginal DETR queries, DN-DETR adds DN part which feeds noised GT labels and boxes into the decoder to provide an auxiliary DN loss. The DN loss effectively stabilizes and speeds up the DETR training and can be plugged into any DETR-like models.
Deformable DETR (Zhu al., 2021) is another early work to speed up the convergence of DETR. To compute deformable attention, it introduces the concept of reference point so that deformable attention can attend to a small set of key sampling points around a reference. The reference point concept makes it possible to develop several techniques to further improve the DETR performance. The first technique is query selection (or "two stage"), which selects features and reference boxes from the encoder as inputs to the decoder directly. The second technique is iterative bounding box refinement with a careful gradient detachment design between two decoder layers. We call this gradient detachment technique "look forward once" in paper.
Following
and DN-DETR, DINO formulates the positional queries as dynamic anchor boxes and is trained with extra DN loss. DINO additionally introduces three methods, which will be described in Sec.
3.3
Sec. 3.4
and Sec. 3.5 respectively.
3.2 MODEL OVERVIEW
As DETR-like model, DINO is an end-to-end architecture which contains a backbone, multi- layer Transformer (Vaswani et al. 2017) encoder, multi-layer Transformer decoder, and multiple prediction heads. The overall pipeline is shown in Fig. 2 Given image, extract multi-scale features with a backbone, and then feed them into the Transformer encoder with corresponding po- sitional embeddings. After feature enhancement with the encoder layers, we propose new mixed query selection strategy to initialize anchors as positional queries for the decoder. Note that this strategy does initialize content queries but leaves them learnable. More details of mixed query selection are available Sec. [35] With the initialized anchors and the learnable content queries, use the deformable attention (Zhu et al., 2021) combine the features of the encoder outputs and update the queries layer-by-layer. The final outputs are formed with refined anchor boxes and classification results predicted by refined content features. As in DN-DETR, we have an extra DN branch to perform denoising training. Beyond the standard DN method, we propose new con- trastive denoising training approach by taking into account hard negative samples, which will be presented in Sec. 3.3 To overcome the shortsightedness of the greedy way for box refinement in previous works, novel look forward twice method is proposed to pass gradients between adjacent layers, which will be described in Sec. 3.4
'The DN-DETR paper (I.i et al. 2022 uses d1 and X2 to denote noise scales of center shifting and box scaling, but sets \ Ag. In this paper, we À in place of X and for simplicity.
4

---

Under review as a conference paper at ICLR 2023
3.3
CONTRASTIVE DENOISING TRAINING
recomsrracton loss
reconstruction loss
Do object
objeet
Deganivv
poiliiv
Decoder Layers M
positive
negative
positive
negative
CDN group0
CDN group!
Figure 3: The structure of CDN group and demonstration of positive and negative examples. Although both positive and negative examples 1D anchors that can be represented points in 4D space, illustrate them as points in 2D space on concentric squares for simplicity. Assuming the square center is a box, points inside the inner square are regarded as positive example and points between the inner square and the outer square are viewed as negative examples.
DN-DETR is effective in stabilizing training and accelerating convergence. With the help of De- Noising (DN) queries, it learns make predictions based on noised Ground -ruth (GT) boxes, which leads to fast convergence. However, each DN query in DN-DETR is matched with a GT box and lacks the ability to predict background for "no object". Since predicting background is also important for DETR-like model to reduce duplicate predictions, we propose Contrastive DeNois- ing (CDN) approach to rejecting hard negative examples. To maximize the utilization of denoising queries, we also propose to adaptive number of denoising groups.
Implementation: DN-DETR has a syper-parameter À to control the noise scale. The generated noises are larger than À as DN-DETR wants the model to reconstruct the ground truth (GT) from moderately noised queries. In our method, we have two hyper-parameters a and A2, where A < A2. As shown in the concentric squares in Fig. 3 we generate two types of CDN queries: pos- itive queries and negative queries. Positive queries within the inner square have a noise scale smaller than \ and are expected reconstruct their corresponding ground truth boxes. Negative queries between the inner and outer squares have a noise scale larger than \ and smaller than X2. They are expected to predict "no object". We usually adopt a small X2 because hard negative samples closer to GT boxes can better help the model suppress duplicate predictions. As shown in Fig. each CDN group has a set of positive queries and negative queries. an image has n GT boxes, a CDN group will have 2 queries with each GT box generating both positive and a negative queries. Similar to DN-DETR, we also use multiple CDN groups to improve the effectiveness of our method. The reconstruction losses are I and GIOU losses for box regression and focal loss (Lnn al. 2020) for classification. The loss to classify negative samples as background is also focal loss. Furthermore, to better utilize DN queries. We improve DN-DETR's design of using fixed number of denoising groups with an adaptive number of denoising groups. For each image, we fix the total number of denoising queries as N. For an image with n objects, the number of CDN groups is 2n Analysis: The reason why CDN works is because it explicitly introduces hard negative examples that are very similar to positive example. Such negative examples encourage the model learn subtle differences between positive and negative boxes for more precise box predictions. The abil- ity to distinguish positive and negative example also enables the model to further reduce duplicate predictions on the basis of DETR. DETR eliminates the need of using NMS to suppress duplicate boxes. Instead, it relies on bipartite matching to pick up only one query for each GT box and sup- press other queries by pushing them away or lowering their confidence. However, the suppressed queries are normally not hard negative. As a result, DETR cannot completely avoid duplicate boxes, especially for low confidence boxes. CDN addresses this issue by introducing explicitly designed negative queries, which further enhance the effect of bipartite matching on avoiding duplicate boxes. For example, on the COCO dataset, we compare CDN with its counterpart DN, both using 300 pre- dictions. The numbers of duplicate predictions for each method are shown in Table 333 For all the thresholds from to 0.3, CDN constantly predicts fewer duplicate boxes than DN.
5

---

Under review as a conference paper at ICLR 2023
Threshold
0.00
0.05
0.10
0.15
0.20
0.25
0.30
total
292.65
158.51
59.91
24.16
10.26
3,94
0.52
duplicate
67.91
31,53
9.63
3,53
1.30
0.53
0.26
CDN
total
292.65
164.62
60.21
23.45
9.90
3.83
0.62
duplicate
53.72
25.87
7.35
2.55
0.97
0.40
0.20
Table 1: For a fair comparison, we only change CDN to DN and keep other
unchanged. For
each model, we choose the top 300 predictions and filter them according confidence scores with 7 thresholds from to 0.3. For each threshold 4, "total" and "duplicate" denote the numbers of total and duplicate predic- tions with scores greater than E, respectively, We view predictions with IoU > 0.8 as duplicate predictions.
gralicas fon cuncelleur
prodictinn a
preticke
prodctess
producio Levee
112
Mhn
MR-1
6b
Layer 1d [c]
(a) Ouce
(b)
Figure 4: (a)(b) Comparison box update in Deformable DETR and our method. (c) APs of look forward once and look forward twice in each decoder layer. "LFO" and "LFTT" denote look forward once and look forward twice, respectively.
3.4 LOOK FORWARD TWICE
We propose new approach improving box prediction in this section. The iterative box refine- ment in Deformable DETR blocks gradient back propagation to stabilize training. We name the method look forward once since the parameters of the i-th decoder layer L, are updated based on
the auxiliary loss of boxes ((rred) only, as shown in Fig 4 (a). where b. (pred)) denotes the predicted boxes in T., Such parameter update approach is greedy method, in which each decoder layer approximates ground truth boxes individually while trying not to influence its previous layers by blocking gradient. Such a method stabilizes training and helps convergence early training stages. However, it may lead to a sub optimal result. On the other hand, allowing gradient to propagate from all latter layers will make the model hard converge. To address this issue, we propose to only allow -,_ to be influenced by gradients from itself and L; as shown in (b). Since parameters in Li-1 are optimized to approximate ground truth boxes both L,-1 and L,, we name our method as look forward twice, which is more comprehensive compared with the look forward once method. Implementation: We compare the implementations of look forward once (LFO) and look forward twice (LFT) follows. Since LFO and LFT share the same process from 6/_ to 6/. we first show this process. Denote _ and b;-1 as the boxes before and after stopping gradient. We have
b sg [b[-]]
where sg/:] denotes stopping gradient. b 1 is used as the input anchor box in I, to obtain Ab, as follows.
Ab; L;(b;-1;0,), 0,),,
where L, denotes the i-th Decoder layer with 0, as its parameters. We ignore other inputs to Lę for simplicity. ig is obtained as follows.
(0- Ab.).
where (-) and -1 denote the sigmoid and inverse sigmoid functions. Note that such box update approach is to guarantee that the updated boxes have normalized æ, y, w, h values between 0 and 1. Equation is marked with green line in Fig. 4(a) where gradients are propagated from (freed) to 0, through Ab,. In LFO, the prediction 6. g.rred)) is equal to b/. While in LFT, we update box predictions o ((pred) based on 8 instead of b as follows.
B (pred) = (a- ((, + Ab:),
(4)
Equation is marked with green line in Fig. b). Similarly, (pred) is obtained as follows.
B (pred)
(1)) + Abt1) (o (o-1((()1) Ab; Abit1)
(5)

---

Under review as a conference paper at ICLR 2023
marked with red line in Fig. b), where the gradients from b (pred) 141 are propagated to
Equation
Og through Abe
Fig. 1 (c) shows comparison the performances look forward once (LFO) and look forward twice (LFT) in different layers. For layer to 2, LFO performs better than LFT. While LFT exceeds
LFO layer 6. This observation verifies our intuition that LFT sacrífices performance in early layers to achieve better final performance.
3.5
MIXED QUERY SELECTION
Uaery Selection
Query Seletiie
Kortc Content
Quaiko
Static Ancher Centor Querios
Dysonio Anchore Dynanic Quatics
Dyromic
(3) Static Queries
(b) Vanilla Query
(c) Mixed Query Selection
Figure 5: Comparison of three different query initialization methods. "static" the same for different images in inference. common implementation for these statie queries is make them learnable. Note that (b) the selected reference points go through a positional encoding and linear transform to obtain the query embeddings as implemented in deformable DETR.
that queries will keep
In DETR (Carion al. 2020) and DN-DETR (Li al. 2022), decoder queries static embeddings
without taking any encoder features from an individual image, as shown in Fig. 5 (a). They learn anchors positional queries from training data and set the content queries vectors. Deformable DETR (Zhu et al. 202T) learns both the positional and content queries, which is another implemen- tation of static query initialization. To further improve the performance, Deformable DETR (Zhu et al., 2021] has query selection variant (or "two-stage"). It selects positions with top K clas- sification scores as reference points and the content queries are linear transform of the positional embeddings of the reference points. In addition, features in the selected positions go through clas- sification head and a box head to calculate auxiliary loss. We call the implementation in Deformable DETR as vanilla query selection as shown in Fig. 5 Vanilla query selection helps the model con- verge especially in early training epochs. However, its content queries are not aligned with those in CDN part-the the content queries in CDN part are learnable class embeddings. Therefore, to use selected positions as anchors and learnable query embeddings as the content queries. We call our method as mixed query selection. We show in Table that simple and intuitive method achieves better result.
propose
4 EXPERIMENTS
4.1 SETUP
Dataset and Backbone: We conduct evaluation on the COCO 2017 object detection dataset (Ln et al. 2014), which is split into train2017 and 10177 (also called minitu
We report
results with two different backbones: ResNet-50 (He et al. 2016) pre-trained on ImageNet-1k (Deng 2021b) pre-trained on ImageNet-22k (Deng et al. 2009). DINO al. 2009) and SwinL (Liu et al.
with Res is trained on t ra n2017 without extra data, while DINO with SwinL is first pre- trained on Object365 (Shao al. 2019 and then fine-tuned on train2017. We also report the test-dev results for DINO with SwinL.
Implementation Details: In appendix F we provide implementation details, including all the hyper- parameters and engineering techniques used in our models.
4.2 MAIN RESULTS
12-epoch setting: With our improved anchor box denoising and training losses, the training process can be significantly accelerated, As shown in Table compare method with strong baselines

---

Under review as a conference paper at ICLR 2023
Model
Epochs
A3
AZ1
A2F
OFLOFS
Plaratis
Fedts- Nareea liee clal. (005
12
$1.3
55.8
41.1
122.1
31.1
1011
2007
9.M
217
DETRIDCS (Crim
12
15.5
20.4
14.5
6.3
15.1
26.7
225
6.M
30
Deformable DET Mireeeoora MirEl
2011
12
11.1
106
0.M
24
DAB DAB-DETR-DCS) Lap C P3p44
12
58.0
60.3
39.5
1822
40.0
56.4
206
61M
17
Dynamic jikRR1se Ryseci ADDERR
12
12.9
61.0
06.3
21.0
01.9
56.4
56M
Dynmii Seadi burrain HUMAI
12
13.0
60.7
06.8
21.7
16.0
53.9
HTCiSscale Chez ah 20190
12
42.3
441
90M
5"
DN-Dilemiee-r De (1atle List
2022
12
45.4
61.9
47.
24.8
36.6
59.4
265
65M
28
DINO-1ccle! DINO-Sscale
12
66.6
53.5
32.0
52.5
6.31
779
67M
24
12
49.4--6.11
66.9
53,8
52.5
63.9
800
47M
10
Results for DINO and other detection models with the ResNet50 backbone on COCO val2C17 trained with epochs (the so called 1x setting). For models without multi-scale features, we test their GFLOPS and FPS for their best model ResNet-50-DCS. DINO uses 900 queries. indicates models that use 900 queries or 300 queries with 3 patterns which has similar effect with 900 queries. Other DETR-like models except DETR (100 queries) uses 300 queries. indicates that they are tested using the mmdetection (2019b) framework.
Table 2:
CranetectionChhnnnnnn Chen et al.
Movel
Ppochs
Faster RCNN) Ken et
2015
I0S
42.0
02.4
44.2
20.5
45.8
G1.1
DETROCS) D 2atrre
500
43.3
63.1
45.9
22.5
47.3
61.1
Deformable DETR
19.2
SMCA-R Gao ef 70:4
50
43.7
63.4
47.2
24.2
47.0
60-4
ASP-RCNX-
2020
96
45.0
64.5
29.6
29.7
27.7
58.0
Dynamic DET Rascale
202
50
17.2
65.9 9
31.1
29.3
39.1
RDD-Deformabl-DEE
50.4
DN-Deformile--DDe
50
48.6
67.4
52.7
31.0
52.0
63.7
DINO-bscal
24
50,4
68.3
51.8
33.3
53.7
61.8
DINO-
54,2
36
50.9
69.0
35.3
34.6
54.1
64.6
DINO-Sseale
36
51.2
69.0
30.8
35.0
54.3
63.3
Table 3: Results for DINO and other detection models with the ResNect50 backbone on COCO va va12217 trained with more epochs (24, 36, or more).
including both convolution-basee methods (Ren et al. 2015 Chen et al.,
2019a Dai al. 202
and -ekee methods (Carion et al.) 2020; Zhu et al. 2021 Dai et al.. 2021b Liu et al.
2022
Li al. 2022). For a fair comparison, we report both GFLOPS and FPS tested on the same A
NVIDIA GPU for all the models listed in Table 2 All methods except for DETR and DAB-DETR
use multi-scale features. For those without multi-scale features, we report their results with ResNet- DCS which has better performance for its use of a dilated larger resolution feature map. Since some methods adopt 5 scales of feature maps and some adopt 4, we report our results with both and 5 scales of feature maps.
As shown in Table 2 our method yields an improvement of +5.6 AP under the same setting using Re with 4-scale feature maps and +6.0 AP with 5-scale feature maps. Our 4-scale model does not introduce much overhead in computation and the number of parameters. Moreover, our method performs especially well for small objects, gaining +7.2 AP with 4 scales and +7.5 AP with 5 scales.
Comparison with the best models with ResNet-50 backbone: To validate the effectiveness of our method in improving both convergence speed and performance, we compare our method with several strong baselines using the same ResNet-50 backbone. Despite the most common 50-epoch setting, we adopt the 24 (2x) and 36 (3x) epoch settings since our method converges faster and yields only a smaller additional gain with 50-epoch training. The results in Table 3 that, using only 24 epochs, our method achieves an improvement of +1.8 AP and +2.7 AP with and scales, respectively. Moreover, using 36 epochs in the 3x setting, the improvement increases to +223 and +2.6 AP with 4 and 5 scales, respectively. The convergence curve comparison is shown in Fig, 6 We also show our results using Swinl. backbone without bells and whistles in Appendix B
4.3 COMPARISON WITH SOTA MODELS
To compare with SOTA results, we use the publicly available SwinL. (Liu et al. 2021b) backbone pre-trained on ImageNet-22K. We first pre-train DINO on the Objects365 (Shao et al. 2019) dataset and then fine-tune it on COCO. As shown in Table DINO achieves the best results of 63. 2AP and 63.3AP on COCO va12017 and demonstrate its strong scalability to larger model size and data size. Note that all the previous SOTA models in Table 4 do not use Transformer Recoder-based detection heads (HTC+ (Chen et al,, 2019a) and DyHead (Dai et al.] (022a)). It is the first time that an end-to-end Transformer detec- for is established SOTA model on the leaderboard (pap). Compared with the previous SOTA
with model size under 1 billion parameters, which
8

---

Under review as a conference paper at ICLR 2023
DM Eetoraatic DETH
Figure 6: Training convergence curves evaluated on COCO va12017 for DINO and two previous state-of- ahe-art models with ResNet-50 using multi-scale features.
Backberz Paiin
Deiecaos Pre-taaning
+/20A
Swirt
254M
1N-11K-16M
O355
66.0
57.2
$6.7
DyRic
Dare
IN-2KK 14M
08.4
668
Set Routerr
pIc1
2544
18-228-148
0355
6011
63.3
381M
IN: 1323111
2103
2001
100
TIND Sorrl (Uen))
208M
O1S
60.
Table 4: Comparison of the best detection models on MS-COCO. Similar to DETR Carion et al. (2020), we the term "end-to-end" to indicate if a model is free from hand-crafted components like RPN and NMS. The term "use mask" means whether model is trained with instance segmentation annotations. terms "IN" and "O365" to denote the ImageNet[Deng et (2009) and Objects365 [Shao et al. (2019) datasets, respectively, Note that "0365" is subset of "FourODs" and "FLD-9M". DyHead does not disclose the details of the datasets used for model pre-training.
use the
models, we use a much smaller model size (1/15 parameters compared with SwinV2-G (Liu et al, 2021ab). backbone pre-training data size (1/60 images compared with Florence), and detection pre- training data size (1/5 images compared with Florence), while achieving better results. In addition, reported performance without test time augmentation (TTA) is neat result without bells and whistles. These results effectively show the superior detection performance of DINO compared with traditional detectors.
4.4 ABLATION
#Row
QS
CDN LFT
AP
APso
AP3s
APs
APM
APL
Optimized DN-Detommable DETR ctal
No
16.3
63.8
50.3
28.2
19.6
GI.T
Row I+CDN'
No
47.2
65.0
51.2
29.4
50.7
62.5
Row2evanilla query selection Zhu al. 2021
Vanilla
47.8
65.6
52.5
31.1
51.1
62.5
Row2emixed query selection
Mixed
48.6
66.0
52.9
31.3
51.9
62.7
DINO (ours, Row-ftlook forward twice)
Mixed
5
19.0
66.6
53.5
32.0
52.3
63.0
Table 5: Ablation comparison of the proposed algorithm components. We use the terms "QS", "CDN", and "LFT" to denote "Query Selection", "Contrastive De-Noising Training". and "Look Forward Twice", respec- tively. We propose optimized DN-Deformable DETR with our technical improvements. The technical details shown in Appendix A We also use adaptive number of denoising groups here.
Effectiveness of New Algorithm Components: We validate the effectiveness of our proposed methods in Table We build an optimized DN-Deformable DETR our strong bascline, which performs better than the one in Table 2 We include all the pipeline optimization and engineering techniques (see section 4ection[4.]]and Appendix F in the strong baseline. The result of the strong baseline available in Rob According to Table[3, our three new methods in DINO further improve the performance significantly even without considering any engineering techniques.
CONCLUSION
In this paper, we have presented strong end-to-end Transformer detector DINO with contrastive denoising training, look forward twice, and mixed query selection, which significantly improves both the training efficiency and the final detection performance. As result, DINO outperforms all previous ResNet-50-basee models on COCO va12017 in both the 12-epoch and the 36-epoch settings using multi-scale features. Motivated by the improvement, we further explored to train DINO with stronger backbone on a larger dataset and achieved a strong result, 63.3 AP on COCO 2017 test-dev. This result establishes DETR-like models as mainstream detection framework, not only for its novel end-to-end detection optimization, but also for its superior performance.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Papers with code coco test-dev benchmark (object detection). htwp://ttoee.co/sst/aaaaeaaaaaee wwtp/oottooooeee/ccceeee detection coco
URL
https:
paperswithcode.
Alexey Bochkovskiy, Yao Wang, and Hong-Yuar Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004. 10934, 2020.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 6154-6162, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229 Springer, 2020.
Kai Chen, Jiangmiao Pang. Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Zi- wei Liu, Jianping Shi, Wanli Ouyang, al. Hybrid task cascade for instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 4974 4983, 2019a.
Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: I /906 0006007155. 155, 2019b.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: 1604:06004 2016.
Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang. Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 7373-7382, 2021a.
Xiyang Dai, Yinpeng Chen, Jianwei Yang. Pengchuan Zhang, Lu Yuan, and Lei Zhang. Dynamic detr: End-to-end object detection with dynamic attention. In Proceedings of the IEEE/CVF Inter- national Conference Computer Vision (ICCV), pp. 2988-2997 October 2021b.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Fast convergence of detr with spatially modulated co-attention. arXiv preprint arXiv:2101.07448, 2021.
Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint wwxiv:2107008430 2021.
Agrim Gupta, Piotr Dollár, and Ross Girshick. Lvis: A dataset for large vocabulary instance seg- mentation, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Car- ion. Mdetr modulated detection for end-to-end multi-modal understanding. arXiv: Computer Vision and Pattern Recognition, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.690,, 2014.
Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. arXiv preprint arXiv:2203.0/ 305.0 2022.
10

---

Under review as a conference paper at ICLR 2023
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre- training, arXiv preprint arXhv:2112.0388: 2021.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(2):318- 327, 2020.
Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. DAB- DETR: Dynamic anchor boxes are better queries for DETR. arXiv preprint arXiv:2201.1233: 2022.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. arXiv preprint arXiv:2111.096
2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
1rXiv:1711.05 2017.
Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jing- dong Wang. Conditional detr for fast training convergence. arXiv preprint 1rXiv:2108.06152, 2021.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. 2018.
Joseph Redmon and Ali Farhadi. Yolo9000:
better, faster, stronger. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7263-7271, 2017.
Joseph Redmon and Ali Farhadi. Yolov3:
An incremental improvement. arXiv preprint
1rXiv:1884.00266, 2018
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6): 137-1 149, 2017.
Hamid Rezatofighi, Nathan Tsoi, Jun Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In Pro- ceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 658-666, 2019.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8430-8439, 2019.
Zhiqing Sun, Shengcao Cao, Yiming Yang, and Kris Kitani. Rethinking transformer- set pre- diction for object detection. arXiv preprint 2011 2020.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In 2019 JEEE/CVF International Conference on Computer Vision (ICCV), pp. 9627- 9636, 2019.
11

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for transformer-based detector. arXiv preprint arXiv:2109.07107, 2021.
Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai, and Zicheng Liu. End-to-end semi-supervised object detection with soft teacher. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3060-3069, 2021.
Zhuyu Yao, Jiangbo Ai, Boxun Li, and Chi Zhang. Efficient detr: Improving end-to-end object detector with dense prior. arXiv preprint arXiv:2104.01318 2021.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao. Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11433 2021.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. ICLR 2021: The Ninth International Conference on Learning Representations, 2021.
12

---

Under review as a conference paper at ICLR 2023
A OPTIMIZED DN-DEFORMABLE DETR
The optimized DN-Deformable DETR differs from the original DN-Deformable DETR the fol- lowing three parts. Firstly, the optimized DN-Deformable DETR adopts deformable attention in both encoder and decoder while the original one only adopts deformable attention in encoder. With deformable attention in decoder, the optimized one is able to more decoder queries. For exam- plc, we use 900 here. Secondly, the optimized one use different weight for matcher and loss, while the original one follows DETR use same weight for loss and matcher. For example, use class weight 1.0 for loss and 2.0 for matcher. Finally, we set dropout rate to O. We find these three technical improvements can improve the performance.
B RESULTS USING SWINL BACKBONE WITHOUT TRE-TRAINING ON OBJECT 365
We also evaluate our method on COCO va 12017 with SwinL as backbone without pre-training on Object 365, The results are without any bells and whistles. We compare with other methods using Swin-L backbone.
Model
Ppochh
AP
APRo
APps
AP2
AP
AP2
Cascade Mask RCNN-SwiLL Vasconcelos 2018
55.0
HTCC++-Swin| [Chen et (20198
57.1
JINO--scale-SWWm
41.3
DINO-Sicale-Swinl
36
58.5
77.0
64.1
41.5
62.3
74.0
Table 6: Results for DINO other detection models with the SwinL. backbone on COCO va12017.
C TEST TIME AUGMENTATIONS (TTA)
We aim to build an end-to-end detector that is free from hand-crafted components. However, to compare with traditional detection models, we also explore the use of TTA in DETR-like models. We only use it in our large model with the SwinI. backbone. Our TTA does not obtain an inspiring gain compared with traditional detectors, but we hope our exploration may provide some insights for future studies.
We adopt multi-scale test and horizontal flip as TTA. However, the way of ensembling different augmentations our method different from that in traditional methods which usually output du- plicate boxes. In traditional methods, the ensembling is done by first gathering predictions from all augmentations and ranked by confidence score. Then, duplicate boxes are found and climinated by NMS box voting. The reason why predictions from all augmentations are gathered first is that duplicate boxes appear not only among different augmentations but also within one augmentation. This ensembling method decreases the performance for our method since DETR-like methods not prone to output duplicate boxes since their set-based prediction loss inhibits duplicate predic- tions and ensembling may incorrectly remove true positive predictions (Carion al., 2020). To address this issue, we designed a one-to-one ensembling method. Assume we have 7i augmenta- tions Augo, Aug1, .... Augn-1, where Aug, has predictions O' and a pre-defined parammtr weight w'. O = { (66, 45 s)) (6f 14 si), j-th boundbox, label and score, respectively. We let Augo be the main augmentation which the most reliable one. For each prediction in O we select the prediction with the highest TOU from predictions of each other augmentations O² ",.,. 0 and make sure the IOU is higher than a predefined threshold. Finally, we ensemble the selected boxes through weighted average follows
where bj. 1J and st denote the
Σ To stantt Bidett)
(6)
Στ
where when there at least one box in O' with IOU higher than the threshold and otherwise. idd denotes the index of the selected box in O'.
13

---

Under review as a conference paper at ICLR 2023
D TRAINING EFFICIENCY
We provide the GPU memory and training time for our base model in Table 7 All results are reported 8 Nvidia A100 GPUs with ResNet-50 (He al.) 20T6). The results demonstrate that our models are not only effective but also efficient for training.
Model
#images per GPU
Traning Time
GPU Mem.
Epoch
AP
Faster RCNN (Ren et al. 2015
~ 60min/cp
13GB
108
42.0
DETR (Carion et al. 2020
16min/ep
26GB
300
41.2
Deformable DETR (Zhu et al. 2021
55min/ep
16GB
50
45.4
DINO(Ours)
55min/ep
16GB
12
49.0
Table 7: Training efficieny for different models with ResNet-50 backbone. All models trianed with 8 Nvidia A100 GPUs. All results are reported by us. The results of Faster RCNN are tested with the mmdetection framework. We use the vanilla Deformable DETR without two-stage and bbox refinement during testing.
E ADDITIONAL ANALYSIS ON OUR MODEL COMPONENTS
Encoder/Decoder
6/6
4/6
3/6
2/6
6/4
6/2
2/4
2/2
AP
47.4
46.2
45.8
45.4
46.0
44.4
44.1
41,2
Table 8: Ablation on the numbers of encoder layers and decoder layers with the ResNet-50 backbone COCO va12017. We use the 12-epoch setting and 100 DN queries without negative samples here.
Analysis on the Number Encoder and Decoder Layers: We also investigate the influence of varying numbers of encoder and decoder layers. As shown in Table 8 decreasing the number of decoder layers hurts the performance more significantly. For example, using the same encoder layers while decreasing the number of decoder layers from 6 to 2 leads to 3.0 AP drop. This per- formance drop is expected as the boxes are dynamically updated and refined through each decoder layer to get the final results. Moreover, like Dynamic DETR (Dai et al.) 2021a) whose performance drops by 13.8AP (29.1 vs 42.9) when decreasing the number of decoder layers 2, the performance drop of DINO is much smaller. This is because our mixed query selection approach feeds the selected boxes from the encoder to enhance the decoder queries. Therefore, the decoder queries are well initialized and deeply coupled with decoder layer refinement.
also observe that compared with other DETR- models
Denoising query
100 CDN
1000 DN
200 DN
100 DN
50 DN
DO
No DN
AP
47.9
47.6
47.4
47.4
46.7
46.0
45.1
Table 9: Ablation on number denoising queries with the ResNet-50 backbone on COCO valida- tion. Note that 100 CND query pairs contains 200 queries which are 100 positive and 100 negative queries.
Analysis on Query Denoising: We continue to investigate the influence of query denoising by vary- ing the number of denoising queries. We use the optimized dynamic denoising group (detailed in Appendix F.I). As shown in Table 9 when we use less than 100 denoising queries, increasing the number lead significant performance improvement. However, continuing increase the DN number after 100 yields only a small additional or even worse performance improvement. We also analysis the effect of the number of encoder and decoder Layers in Appendix
14

---

Under review as a conference paper at ICLR 2023
F MORE IMPLEMENTATION DETAILS
F.1 ADAPTIVE DN GROUPS
In DN-DETR, all the GT objects (label+box) in one image are collected one GT group for de- noising. To improve the DN training efficiency, multiple noised versions of the GT group in an image are used during training. In DN-DETR, the number of groups is set five ten according to different model sizes. As DETR-like models adopt mini-batch training, the total number DN queries for each image in one batch is padded the largest one in the batch. Considering that the number of objects in one image in COCO dataset ranges from 1 to 80, this design is inefficient and results in excessive memory consumption. To address this problem, we propose to fix the number of DN queries and dynamically adjust the number of groups for each image according to its number of objects.
F.2
MODEL PRE-TRIANING
Objects365 (Shao et al. for training and 80, 000 annotated images for validation. To use the data more efficiently, We select the first 5, 000 out of 80, 000 validation images as validation set and add the others to training. We pre-train DINO on Objects365 for 26 epochs using 64 Nvidia A100 GPUs and fine-tune the model COCO for 18 epochs using 16 Nvidia A100 GPUS. Each GPU has a local batch size of 1 image only. In the fine-tuning stage, we enlarge the image size to 1.5x (i.e., with max size 1200 x 2000). This adds around 0.5 AP to the final result. To reduce the GPU memory usage, we leverage checkpointing (Chen et al. 2016 and mixed precision (Micikevicius et al. 2018) during training. Moreover, we use 1000 DN queries for this large model.
2019) is large-scale detection data set with over 1.7M annotated images
F.3 OTHER IMPLEMENTATION DETAILS
F.3.1 BASIC HYPER-PARAMETERS
For hyper-parameters, as DN-DETR, we use a 6-layer Transformer encoder and 6-layer Trans- former decoder and 256 as the hidden feature dimension. We set the initial learning rate (Ir) as 1 10-4 and adopt a simple
scheduler, which drops Ir at the 11-th, 20-th, and 30-th epoch by multiplying 0. for the 12, 24, and 36 epoch settings with RestNet50, respectively. We use the AdamW (Kingma& & Ba 2014; Loshchilov & Hutter 2017) optimizer with weight decay of 1 x 10-4 and train our model on Nvidia A100 GPUs with batch size 16. Since DN-DETR (Li et al., 2022) adopts 300 decoder queries and patterns (Wang et al. (021), we use 300 x 3 900 decoder queries with the same computation cost. Learning schedules of our DINO with Swinl. are available in the appendix.
F.3.2 LOSS FUNCTION.
We use the L1 loss and GIOU (Rezatofighi et al., 2019) loss for box regression and focal loss (Lin et al. 2020) with = 0.25, 7 = 2 for classification. As in DETR (Carion et al. 2020), we add auxiliary losses after each decoder layer. Similar to Deformable DETR (Zhu et al. 2021). we add extra intermediate losses after the query selection module, with the same components as for each
decoder layer. We use the same loss coefficients as in DAB-DETR (Liu et al.) 2022) and DN-
DETR (Li al. 2022), that is, 1.0 for classification loss, 5.0 for L1 loss, and 2.0 for GIOU loss.
F.3.3 DETAILED MODEL COMPONENTS.
We also optimize the detection pipeline used in DAB-DETR (Liu et al.) 2022 and DN-DETR (Li cet al., 2022). Following DN- ON-Deformmale-DETT (Li et al. 2022), we use the same multi-scale ap- proach in Deformable DETR (Zhu et al. 2021 and adopt the deformable attention. DN-DETR uses different prediction heads with unshared parameters in different decoder layers. In addition, we introduce dynamic denoising group to increase denoising training efficiency and alleviate memory overhead (see Appendix F.I. In this work, we find that using a shared prediction head will add ad- ditional performance improvement. This also leads to reduction of about one million parameters.
15

---

Under review as a conference paper at ICLR 2023
Item
Value
0.0001
Ir backbone
1e-05
weight decay
0.0001
clip.max norm
0.1
pe_temperature
20
enc_layers
6
dec_layers
dim feedforwwdr
2048
hidden dim
256
dropout
0.0
nheads
8
num queries
900
enc_n.points
dec_n_points
4
transformer.activve activation
"relu"
batch_norm_type
"FrozemBatchNorm222
cost class
2.0
set cost bbox
5.0
set_cost_giou
2.0
cls_loss_coe'
1.0
Bbox_loss, _cosf
5.0
giou.loss.coef
2.0
focal_alpha
0.25
dn box noise noale
0.4
dn labelbee ratio
0.5
Table 10: paper-pprraettrs used in our models.
In addition, we find the conditional queries (Meng al. 2021) used in DAB-DETR does not suit our model and we do not include them in our final model.
F.3.4 TRAINING AUGMENTATION.
We use the same random crop and scale augmentation during training following DETR (Carion al. 2020). For example, we randomly resize input image with its shorter side between 480 and 800 pixels and its longer side at most 1333. For DINO with SwinL, we pre-train the model using the default setting, but finetune using 1.5x larger scale (shorter side between 720 and 1200 pixels and longer side at most 2000 pixels) to compare with models on the leaderboard (pap). Without using
any other tricks, achieve the result of 63.1 on va12017 and 63.2 on test-dev dev without test time augmentation (TTA) (see Appendix C). outperforming the previous state-of tfute-of-the-art result 63.1 achieved by SwinV2 (Liu al. 2021a) with a much neater solution.
F.3.5 MULTI-SCALE SETTING.
For our 4-scale models, we extract features from stages 2, 3, and 4 of the backbone and add an extra feature by down-sampling the output the stage 4. An additional feature map of the backbone stage 1 is used for our 5-scale models. For CDN pairs which contain 100 positive queries and 100 negative queries.
set X 1.0 and λ 2.0 and use 100
F.4 DETAILED HYPER-PARAMETERS
We list the hyper-parameters for those who want to reproduce our results in Table 10
16

---

Under review as a conference paper at ICLR 2023
#parameters
GFLOPs
FPs
HINO-Swin-L-4Serie DINO-Swin-L-5Scatt
217.6
1284.5
8.1
217.2
703.5
12.8
Table 11: The inference speed and computation cost of our laege model.
# epochs
AP
AP50
AP75
APg
APM M
AP,
DINO-LFT
12
49.0
66.6
53.5
32.0
52.3
63.0
DINO-LF3
12
48.3
63.5
52.7
31.2
51.7
62.5
DINO-LF4
12
48.2
65,4
52.9
30.4
51.8
62.9
Table 12: The experiments of Look Forward Three (LF3) and Four times (LF4).
G INFERENCE SPEED AND GFLOPS
We list the inference cost our 4-scale and 5-scale model with Swin-L backbones in Table that our model for Table 4 is 5-scale model.
Note
H WHY DINO IMPROVES AP ON SMALL OBJECTS BY LARGE
There are several reasons for the large AP improvement on small objects (AP,).
1. In Table 5] our optimized DN-Deformable DETR has AP, of 28.2 which is +3.4 higher than that of the original DN DN-Deformable DETR in Table 2 The original one uses dense attention in the decoder while the optimized uses deformable attention which is better for local attention and therefore improves AP,- In addition, we fixed a problem in the original DN-Deformable DETR's Transformer encoder-their deformable attention is not properly initialized using the initialization method in Deformable DETR. The Transformer encoder is a critical component that processes multi-scale image features and multi-scale image features are critical for small objects. Therefore, the optimized one has higher AP,-
2. In Table 55, we can see that CDN improves AP, by 1.2. By introducing negative noised queries, CDN encourages the model to pick up the anchor nearest to the center a GT box to make predictions and explicitly suppresses farther anchors. Since small object detection is more sensitive to the quality of anchors, DINO with high-quality anchors can achieve better AP..
3. Query selection improves AP, by 1.9. Query selection provides high-quality anchor ini- tialization which especially beneficial to small objects. The reason similar to reason that small object detection is more sensitive the quality of anchors.
MORE DETAILS ABOUT LOOK FORWARD TWICE (LFT)
We propose LFT because the original Look Forward Once scheme for box refinement is greedy, which will lead to sub-optimal results. We propose LFT make the model far-sighted. But there is a trade-off. When we increase the number of layers to "look forward". the model becomes harder converge. We conduct experiments of Look Forward Three and Four times as shown in Table T2 The results become worse when we continue to increase the number of layers to "look forward"..
Following is detailed explanation of why LFT is worse than LFO in layers 0 to 2 but outperforms LFO in layers 3 to 6 as shown in Fig.
There are actually three factors affecting the performance of layer
1. The performance of layer i 1. Since the predictions of layer are based predictions of layer 1, better predictions in layer
lead to better predictions in layer i.
2. Whether allows gradients to backpropagate from layer to layer i i- Allowing the gradient to propagate to layer i helps performance in layer i.
17

---

Under review as a conference paper at ICLR 2023
3. Whether allows gradients backpropagate from layer + 1 to layer propagate from layer + to layer jeopardizes performance layer i.
i. Allowing gradient to
For layer there is no layer i 1, so factors 1 and 2 do not affect the performance. According to factor 3, L.FO is better than LFT.
For layers 1 to 5, LFO an advantage factor and LFT has an advantage factor Because factor 2 affects the performance more than factor 3, The gap between LFT is narrowed down from layer 2 and LFT exceeds LFO in layer
In the last layer (when i 6). there is no layer i + 1 (factor 3 does not affect the result) and LFT has the advantage both factors and Therefore, LFT exceeds LFO the last layer.
VISUALIZATIONS
90 DETR
100 DINO
66
Figure 7: Visualization of cases DINO outperforms DN-DETR.
We present a comparison of visualizations in Fig. 7 predictions than DN-DETR.
The results show that our DINO has better
K LVIS RESULTS
To evaluate DINO's s performance on other detection datasets, we conducted experiments on more challenging LVIS (Gupta et al. 2019) dataset.
18

---

Under review as a conference paper at ICLR 2023
Val
vT.0
Model
Backbone
AP
AP,
AP.
AP/ f
Supervised-RF'S RFS Gupta et al. 2019)
R50
25.4
12.3
24.3
32,4
MaskRCNN-LOCE (He et al. 2017)
R50
27.4
GLIP al. 202
Swin-L
26.9
17.1
23.3
35,4
DINO
R50
31.2
24.5
29.8
35.7
Table 13: The results on LVIS val v1.0. * denotes zero-shot results. 12 epochs and is not fully conveged.
denotes DINO is trained for
19


Reviews:
[{"review_id": "wsxJKXcXTP", "reviewer": "Reviewer_eqZC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an end-to-end Transformer detector named DINO that archives impressive performance on the COCO benchmark. The proposed DINO model improves the training efficiency and the detection performance by utilizing (i) contrastive denoising training, (ii) look forward twice, and (iii) mixed query selection strategies. The experimental results show that  DINO outperforms all previous ResNet-50-based models on COCO val2017 in the 12-epoch and the 36-epoch settings using multi-scale features. Further, training DINO with a stronger backbone on a larger dataset leads to an impressive result of 63.3 AP on COCO 2017 test-dev which is comparable with even CNN-based state-of-the-art methods on mainstream detection task. ", "summary_of_the_review": "The paper has some incremental contributions. But those contributions are highly effective in achieving state-of-the-art performance on DETR framework. So, I am in favor of accepting this paper. ", "strength_and_weaknesses": "Strengths:\n\n1.\tImpressive results. Although previous DETR-like models were having end-to-end object detection pipelines, their results are inferior to classical CNN-based object detectors.  Unlike the previous DETR-like models, the paper reports state-of-the-art object detection results on the COCO benchmark. \n\n2.\tThe model requires relatively less pre-training data and a smaller model size compared to many state-of-the-art methods in the benchmark. \n\n3.\tThe proposed changes are simple, but effective in improving accuracy. \n\n4.\tThe paper reports results on a large Swin-L backbone with large-scale pre-training. \n\n5.\tThorough ablation experiments. \n\n\nWeakness:\n\n1.    Incremental novelty:  The key idea behind the proposed contributions such as contrastive denoising training, and look forward twice are well-known in the literature. I will not consider this as a major problem since the authors effectively adapted those ideas for improving performance on DETR framework. \n\n2.   Please explain why the improvement on small objects is very high (+7.2% in Tab. 2) compared to medium and large-sized objects. Which of the proposed contribution helps in achieving this?  Why?  \n\n3.   Table 2 shows that the proposed method is faster (FPS 24) than existing DETR-based object detectors, and its speed is even comparable with deformable DETR. But it is not clear to me how the proposed model with many components can achieve a speed, comparable to deformable DETR, and faster than DN-DETR?\n\n4.  In page 8 it is mentioned that “Note that the results of our models with ResNet-50 backbone are higher than those in the first version of our paper due to engineering techniques”. Please clarify which “first version” the authors are referring to”? \n\n5. The paper utilizes many engineering tricks to achieve state-of-the-art results (Appendix F). \n\n6.  In tables 2 and 4, what are the results of the proposed model without using additional engineering techniques mentioned in appendix F?\n\n7. Table 4 shows that the proposed method achoves Impressive SOTA results. It uses smaller pre-training, it does not need mask annotation and requires less paramets. Still it will achieve results comparable with SOTA methods. For complete understanding of the model performance, it will be beneficial to introduce a comparison on  the inference time and FLOPS in Table 4.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qk7ko1lrzy", "reviewer": "Reviewer_nKUz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose three different methods to improve the framework of DETR: a contrastive way for centre-surrounding, twice forward prediction for refining results and a mixed query selection. The experiments demonstrate the necessity of these methods and significant improvement in classical benchmarks compared with other methods.", "summary_of_the_review": "Despite the lack of sufficient discussion on the speed of this article, I am still inclined to accept this paper.", "strength_and_weaknesses": "Strength:\n1. This paper investigates the DETR framework very well and gives a comprehensive survey of related works.\n2. The performance is impressive. The advantages are mainly faster convergence, better performance and the ability to have smaller models.\n3. The motivations of three different components are reasonable. And ablation experiments show the expected target has been achieved.\n\nWeakness:\nAlthough the model is only 1/10 of the swinT size, there is nothing result about the inference speed. So I wonder if the so-called small models don't actually have any advantage in terms of computational power.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ZUJau0H8_v", "reviewer": "Reviewer_dnJ7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes some improvements on the DETR family of Detectors.\nNamely, the improvements are as follows:\n- Add some negatives in the denoising objective\n- Refine the design of the iterative refinement of boxes\n- Propose a slightly different way to initialize object queries\n\nWith these improvements, the resulting model improves across the board on the Coco dataset, and in particular sets a new SOTA result for the setting with extra-data", "summary_of_the_review": "The paper is technically sound and the results are strong, I recommend acceptance.", "strength_and_weaknesses": "The paper shows strong results on the Coco benchmark, surpassing current SOTA even with less training data, which is an impressive feat.\nI find all the comparisons to be fair with previous approaches.\n\nOn the presentation side, one minor weakness is the lack of visualization of any kind. For example, I would have liked to see some head-to-head inference comparison with DN-DETR on the same images, as well as some failure cases.\n\nFinally, although it is not required for acceptance, I think the paper could be strengthen by showing results on datasets beyond Coco, perhaps LVIS or CrowdHuman. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "HSyH9Kaenl", "reviewer": "Reviewer_7vU5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an improved DETR-like algorithm (both accuracy and speed) for object detection. Three techniques are proposed in the paper, including contrastive denoising training, look forward twice, and mixed query selection. The idea of denoising anchor boxes in previous work has shown effectiveness in stabilizing training and accelerating convergence. Based on the previous work, the authors introduce the additional hard negative examples to further improve the performance. The experimental results of the 12-epoch setting show that the new contrastive denoising technique leads to a faster convergence. Furthermore, the ablation study shows that this new technique has achieved the gain of about 1% in detection accuracy. The Look Forward Twice technique for iterative box refinement is the extension of the look forward once in previous work Deformable DETR. The backpropagation of the gradient is performed in two consecutive decoder layers instead of one layer. The ablation study shows that this technique boosts the performance by about 0.5%. The Mixed Query Selection technique is a combination of previous works DN-DETR and Deformable DETR, with selected positions anchors and learnable the content queries. The ablation study also shows that this technique boosts the performance by about 0.5%.\n\nFurthermore, the experimental results show that the performance of the model with about 1 billion parameters trained by the new approach outperforms the larger models trained from even more pre-training data. \n", "summary_of_the_review": "I would recommend a marginally below acceptance. The novelty of this work is not significant as the three techniques are quite incremental without theoretical insights. The experimental results and reported performances are the state-of-the-arts and may have a good impact to the field of objection detection. ", "strength_and_weaknesses": "Strength:\n\nThe proposed method is simple yet effective.\nThe paper is well organized. The illustration provides the great details of the new techniques, which are helpful for the readers to understand.\nThe work has achieved strong experimental results with the improvement of about 2% over the state-of-the-art baselines.\nThe paper presents extensive details of the experiments.\n\n\nWeaknesses\n\nThe motivation of the look forward twice scheme isn't very convincing. The paper has shown that the choice of twice is better than once, but is it better than three times and four times? If there is no good theoretical analysis to support this choice, it would be better to have some empirical results to justify it.\n\nThe second last sentence in ABSTRACT, “…, DINO significantly reduces its model size and pre-training data size while achieving better results.”  The claim is confusing. To my understanding, with the training techniques of DINO, the model with small size can achieve better performance than the large models. But the approach of DINO itself does not reduce the model size.\n\nIn section 3.3, a typo in “They are are expected to predict “no object”.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5QDkEFiFai", "reviewer": "Reviewer_tCi7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In order to speed up the training convergence and highly optimize the Transformer-based detector (DETR-like models), a contrastive way for denoising training, a look forward twice scheme for box prediction, and a mixed query selection method for anchor initialization are proposed. The extensive ablation studies demonstrate the effectiveness of proposed methos over current DETR-like models.", "summary_of_the_review": "I would tend to accept this paper as it is practical DETR-like model and the proposed modules are supported by empirical experiments.", "strength_and_weaknesses": "Strengths:\n1) DINO points out the reason why newly developed DETR-like models are inferior to the classical detectors and propose the corresponding solutions, eg., denoising training, query initialization and box prediction to highly optimize DETR-series detectors.\n2) The contrastive denoising training helps the model to predict more precise boxes and avoid duplicate outputs of the same target compared with DN-DETR.\n3) The look forward twice scheme has the ability to correct the updated parameters with gradients from later layers to overcome the weakness of refining boxes in each decoder layer, while keeping the advantages of fast convergence.\n4) Authors report the performance of DETR-like model on large backbone with large-scale pre-training data.\n\nWeaknesses:\n\nPlease see the following part of `Unclarity and Questions'.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster