Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment
Authors: Qiang Chen
Xiaokang Chen
Jian Wang
Shan Zhang
Haocheng Feng
Junyu Han
Errui Ding
Gang Zeng
Jingdong Wang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Detection Transformer (DETR) relies on one-to-one assignment for end-to-end object detection and lacks the capability of exploiting multiple positive object queries. We present a novel DETR training approach, named {\em Group DETR}, to support one-to-many assignment in a group-wise manner. To achieve it, we make simple modifications during training: (i) adopt $K$ groups of object queries; (ii) conduct decoder self-attention on each group of object queries with the same parameters; (iii) perform one-to-one assignment for each group, leading to $K$ positive object queries for each ground-truth object. In inference, we only use one group of object queries, making no modifications to model architectures and inference processes. Group DETR is a versatile training method and is applicable to various DETR variants. Our experiments show that Group DETR significantly speeds up the training convergences and improves the performances of various DETR-based methods.

Under review as a conference paper at ICLR 2023
GROUP DETR: FAST DETR TRAINING WITH GROUP- WISE
ASSIGNMENT
Anonymous authors
Paper under double-blind review
ABSTRACT
Detection Transformer (DETR) relies one-to-one assignment for end-to-end object detection and lacks the capability of exploiting multiple positive object queries. We present novel DETR training approach, named Group DETR, support one-to-many assignment in a group-wise manner. To achieve it, we make simple modifications during training: (i) adopt K groups of object queries; (ii) conduct decoder self-attention on each group of object queries with the same pa- rameters; (iii) perform one-to-one assignment for each group, leading to K pos- itive object queries for each ground-truth object. In inference, we only use one group of object queries, making no modifications to model architectures and in- ference processes. Group DETR is versatile training method and is applicable to various DETR variants. Our experiments show that Group DETR sigmificantly speeds up the training convergences and improves the performances of various DETR-based methods.
INTRODUCTION
Detection Transformer (DETR) (Carion et al., 2020) achieves end-to-end detection without the need
of con-maximum suppression (NMS) (Hosang et al., 2017). There are several designs: (i) adopt an encoder-decoder architecture based on transformer layers (Vaswani et al., 2017), (ii) introduce object queries, and (iii) perform one-to-one assignment' by conducting bipartite matching (Kuhn, 1955) between object predictions and ground- truth objects.
The original DETR suffers from the slow convergence issue and needs 500 training epochs to achieve good performance. Various solutions have been developed to accelerate the training from different aspects. For example, sparse transformers (Zhu al., 2020b; Gao al., 2021: Chen al., 2022c; Roh et al., 2022) are adopted to replace dense transformers. Additional spatial modulations are introduced into object queries (Zhu et al., 2020b; Meng et al., 2021; Wang et al., 2022b; Yao al., 2021; Liu et al., 2022a; Gao et al., 2022). Denoising modules are presented for stabilizing the object query and group-truth matching the assignment process (Li et al., 2022; Zhang et al., 2022b).
In this paper, we propose novel training approach Group DETR to accelerate DETR training con- vergence. Group DETR introduces group-wise one-to-many assignment. It assigns each ground- truth object to many positive object queries (one-to-many assignment"), and separate them into multiple independent groups, keeping only one positive object query per object (one-to-one assign- ment) in each group. To achieve it, make simple modifications during training:
(i) adopt K groups of object queries; (ii) conduct decoder self-attention on each group of object queries with the same parameters; (iii) perform one-to-one assignment in each group, leading to K positive object queries for each ground-trun truth object. The design achieves fast training convergence, maintaining the key DETR property: enabling end-to-end object detection without NMS. We only use one group of object queries in inference, and we do not modify either architectures or processes, bringing no extra cost compared with the original method.
Group DETR is versatile training method and can be applied to various DETR- models. Extensive experiments prove that our method is effective in achieving fast training convergence
'Onc-to-one assignment: one ground-truth object is only assigned to one object query. OOnetoo-man Cone-to-many assignment: each ground-truth object he assigned more positive object queries.

---

Under review as a conference paper at ICLR 2023
58.1
49.7
45.1
mAp
DETR-DCS
Conditional DETR-DCS
DCS-DETR-DCS
DAB-Deformabte-DET
DINO-Delomabii-DETR
500
Epochs
Figure 1: Comparisons training convergence curves. We show the training convergence curves of various DETR-based models. All experiments are conducted on MS COCO (Lin et al., 2014) with ResNet-50 (He et al., 2016) as the backbone. More results and comparisons can be found in Table and Table 2. Here, we use different colors to distinguish different models in the figure and apply dashed curves and bold curves to highlight the comparisons between baseline models and their Group DETR counterparts. Best view in color.
(convergence curves are shown in Figure 1). Group DETR obtains consistent improvements on vari- ous DETR-based methods (Meng et al., 2021; Liu et al., 2022a; Li et al., 2022; Zhang et al., 2022b). With a 12-epoch (1x) training schedule on MS COCO (Lin et al., 2014), Group DETR significantly improves Conditional DETR-C5 by 5.0 mAP. The non-trivial improvements hold when we adopt longer training schedules (e.g., 36 epochs or 50 epochs). Moreover, Group DETR can easily out- perform baseline models when applied to multi-view 3D object detection (Liu et al., 2022b;e) and instance segmentation (Cheng et al., 2021).
RELATED WORKS
Acceleraing DETR training convergence. The success of DETR (Carion al., 2020) in object detection validates the potential to achieve elegant designs with transformers in computer vision. Since DETR (Carion et al., 2020) was proposed, its slow convergence issue has been critical problem that many researchers (Bar et al., 2022; Wang et al., 2022a; Song al., 2022; Roh et al., 2022) try to address.
Many works provide their solutions and achieve 10x speed up for DETR. They mainly focus on proposing better transformer layers (Zhu et al., 2020b; Gao et al., 2021; Meng et al., 2021; Dai al., 2021; Roh et al., 2022; Cao et al., 2022: Zhang et al., 2022a; Chen al., 2022d) and designing new types of object queries (Zhu et al., 2020b; Meng et al., 2021; Wang et al., 2022b; Yao al., 2021; Liu et al., 2022a; Gao al., 2022). DN-DETR (Li al., 2022) and DINO (Zhang et al., 2022b) attribute the slow convergence issue to the instability of bipartite matching (Kuhn, 1955).They present auxiliary query denoising tasks speed up the DETR training convergence. Unlike previous approaches, we show that assignment methods are critical to fast DETR training convergence. We propose Group DETR to support one-to- assignment in group- wise manner, which can be achieved by simple modifications during training compared with DETR.
One-to-many assignment and one-to-one assignment. One-to-many assignment is widely adopted in modern detectors (Redmon et al., 2016; Ren et al., 2015; Liu et al., 2016; He et al., 2017: Lin et al., 2017; Cai & Vasconcelos, 2018; Chen et al., 2019; Tian et al., 2019; Zhang et al., 2020: Zhu et al., 2020a; Kim & Lee, 2020; Bochkovskiy et al., 2020; Chen et al., 2021; Ge al., 2021). It produces duplicate predictions and needs NMS (Hosang et al., 2017: Bodla et al., 2017) for post-processing. DETR (Carion et al., 2020) explores alternative way (one-to-one assign-
2

---

Under review as a conference paper at ICLR 2023
ment) and achieves end-to-end detection, removing the need for NMS. Recent studies (Wang al., 2021: Sun et al., 2021b;a) show that one-to-one assignment a key factor in achieving end end-to-end detection. Differently. DETR-hased methods (Carion et al., 2020) and we focus on exploiting assignment methods to speed up DETR training in this paper.
find that the one-to-one assignment impacts the training convergence of
3
GROUP DETR
ground cruta
ground trush
ground truth
ansignment
assignment
arsignment
predictions
predictions
predictions
detection bead
detertion head
Docoder
Decoder
MX
.Mx
FFN
FEN
norm
K
K
add & corm
add & norm
imagge features
image features
relf-attention
self-artention
self- self-artention
K
abject querios
object quuerics group
object queries
group
Original DETR
Our Group DETR
Figure 2: Decoder architectures of DETR and Group DETR training. The key differences from DETR include: Group DETR feeds K groups of queries to the decoder, conducts self-attention on each group of object queries with the shared parameters, and makes one-to-one assignment for each group.
Group DETR is training approach for accelerating the DETR training convergence. We make simple modifications during training and adopt the same architecture and inference process for in- ference. Figure 2 illustrates the decoder parts of DETR (Carion et al., 2020) and our Group DETR for training.
3.1 DETR
DETR has three key designs: (i) adopt transformer encoder-decoder architecture (Vaswani al., 2017), (ii) introduce object queries, and (iii) perform one-to-one assignment by conducting bipartite matching between object predictions and ground-truth objects.
DETR architecture. The DETR architecture consists of backbone (e.g., ResNet (He et al., 2016), Swin Transformer (Liu et al., 2021), or others (Dosovitskiy et al., 2021; Liu et al., 2022d; Chen et al., 2022a; He et al., 2022: Chen et al., 2022b)), transformer encoder, transformer de- coder, and object class and box position predictors (Carion et al., 2020). Figure 2 (a) shows the architecture of the transformer decoder in DETR. The image features extracted by the backbone and the transformer encoder layers. The transformer decoder takes N object queries {qi...., qN input. It performs self-attention object queries, aggregates the image features refine the
3

---

Under review as a conference paper at ICLR 2023
query embeddings by conducting rros--attention, and adds an FFN get the output query embed- dings. The output query embeddings are fed into detection heads to produce N object predictions.
One-to-one assignment. In model training, DETR perform one-to-one assignment to find the learning target for each object prediction. It employs the Hungarian algorithm (Kuhn, 1955) to find an optimal bipartite matching ē between the predictions and the ground-truth objects:
arg min Cuutch (Wis Bo(i));
OEEN
where EN the set of permutations N elements and Cmatch (yts Bo(s)) the matching cost (Carion al., 2020) between the ground truth y; and the prediction with index (ii).
3.2
GROUP DETR
Group DETR makes simple modifications during training compared with DETR (Figure 2 (b)): (i) adopt K groups of object queries; (ii) conduct decoder self-attention on each group of object queries with the same parameters; (iii) perform one-to-one assignment in each group (group-wise one-to- many assignment), leading to positive object queries for each ground-truth object.
K groups of object queries. There are K groups of queries the pro- posed Group DETR:
Algorithm Pseudocode of one Group DETR decoder layer in python-style
G¹ = (ql..... I
(2)
self sel felf-tttentoon in decader pross_atten: aroas-attention ffns PPN in decodar
decoder
X: output image features of the ancoder
GK = [q
(3)
Q: object queries, with size (KNN, B, C)
N, K, B, e: object query aunber, group number, batch size, feature dimension
The total X object queries are con- catenated and fed to transformer de- coder layers (as shown in Figure 2 (b)).
Group DETR
if training:
split object queries to K groups 0_iist Q.split (N, dim=01 1ist tersors group_0 cat lisss,, dim=i) (N, KxB, C)
Group-wise decoder self-attention. We perform group-wise self-attention in the transformer decoder layers. This leads to that the object queries do not interact with the queries across groups. The pseudocode of group-wise self- attention is in Algorithm 1.
group-wise sel celf-attention out self (gzoup_0) + (N, ExB, concat all groups: (KxM, B, C) out - cat sputtsplit(B. dimml), dim=0)
arosa-atzention and ffa elses ffn(cross_atten (out, X))
in inference, only one group is kept Q[:N] f (N, B, C)
Group-wise one-to-many assign- ment. We apply one-to-one assign- ment to each group independently and could get K matching results:
self-attent croas-actention, and ffn out sulf_atters (Q)
out rín atntcrrossattteecout. X))
al arg min oEEN
Σ Cmatch (yir, (o())):
(4)
8* arg min
Σ Cmaath ()
(5)
oEEN
where &K and i are the optimal matching result and the prediction of the K-th group, respec- tively. During training, each group will calculate the loss (Carion al., 2020) independently in Group DETR. The final training loss the average of K groups.
Model inference. ference. According to our experiments, every group can achieve similar results in Group DETR (Table 5). We simply the first group of object queries.
We adopt the same architectures and processes the baseline models in in-
4

---

Under review as a conference paper at ICLR 2023
Model
w/ Group DETR
mAP
AP,
APm
APi
Conditional DETR-C5
32.6
14.7
35.0
48.3
Conditional DETR-C3
37.6 (+5.0)
18.2
40.7
55.9
Conditional DETR-DC5
36.4
18.0
39.6
52.5
Conditional DETR-DC5
41.2 (+4.8)
21.4
45.0
58.7
DAB-DETR-C5
35.2
16.7
38.6
51.6
DAB-DETR-C5
39.1 (+3.9)
19.7
42.5
56.8
DAB-DETR-DOE
37.5
19.4
40.6
53.2
DAB-DETR-DC5
41.9 (+4.4)
23.3
45.6
58.4
DN-DETR-C5
38.6
17.9
41.6
57.7
DN-DETR-C5
40.6 (+2.0)
19.8
43.9
59.4
DN-DETR-DC5
41.9
22.2
45.1
59.8
DN-DETR-DOD
44.5 (42.66
25.9
48.2
62.2
DAB- Deformmble--DEr
44.2
27.5
47.1
58.6
HaD--eformmall-DETTR
/
45.7 (41.5)
28.1
49.0
60.6
DINO- Dew---eeoooeee-eee
49.4
32.3
52.5
63.2
Tiri--DefomabbeDDeee DETR
50.1 (+0.7)
32.4
53.2
64.7
Table 1: Results with 12-epoch training schedule on MS COCO. All experiments adopt ResNet- 50 (He al., 2016) as the backbone. We highlight the improvements brought by Group DETR on various DETR-based methods. Note that we do not use multiple patterns (Wang et al., 2022b) in our experiments.
DN-DETR in the table, we use the improved version of DN (dynamic DN groups (Zhang et al., 2022b)) and set the DN number to 100 (more results about the DN number can be found in Appendix B Table 8). Thus, the baseline results DN-DETR are slightly different from the ones (with 3 patterns) reported in the original paper (Li et al., 2022). For DINO-Deformable- DETR, we adopt the 4scale version (Zhang et al., 2022b).
4 EXPERIMENTS
We demonstrate the effectiveness of our Group DETR object detection, instance segmentation and multi-view 3D object detection. We adopt the training settings and hyper-parametert same as the baseline models, including learning rate, optimizer, pre-trained model, initialization methods, and data augmentations The number of queries in each group is the same as the baselines.
4.1 OBJECT DETECTION
We verify our approach over object detection on MS COCO (Lin et al., 2014) and make comparison with representative DETR-bases based methods, including Conditional DETR (Meng et al., 2021), DAB- DETR (Liu et al., 2022a), DN-DETR (Liu et al., 2022a; Zhu et al., 2020b), and HINO-Deformable- DETR (Zhang et al., 2022b; Zhu et al., 2020b). We report the results with 12-epoch (1x) and 50-epoch training schedules, as well as the comparison in terms of convergence curve.
Results with standard 1x schedule. Table report the results. Group DETR gives consistent improvements over all baseline models. In comparison to query design methods, Group DETR significantly boosts detection performance when applied to Conditional DETR (+5.0 mAP for C5 and +4.8 mAP for DC5) (Meng et al., 2021), DAB-DE DAB-DETR (+3.9 mAP for C5 and +4.4 mAp for DC5) (Liu al., 2022a), and DAB-Deformable DETR (+1.5 mAP for multiple levels of feature maps) (Liu et al., 2022a; Zhu et al., 2020b).
In comparison the matching stabilization methods, Group DETR can also give non-trivial gains over DN-DETR (+2.0 mAP for C5 and +2.6 mAP for DC5) (Li et al., 2022) and DINO- Deformable-Dee DETR (+0.7 mAP for multiple levels of feature maps) (Zhang et al., 2022b; Zhu al.,
We may adjust the batch size according the GPU memory. Note that will retrain the baseline model with the same hatch size make fair comparisons when conducting experiments.
5

---

Under review as a conference paper at ICLR 2023
Model
w/ Group DETR
mAP
AP,
AP
AP;
Conditional DETR-C5
40.9
20.5
44.2
59.6
Conditional DETR-C5
43.4 (+2.55
23.0
47.3
62.3
Conditional DETR-DCS
43.7
23.9
47.6
60.1
Conditional DETR-DC5
45.8 (4-2.1)
26.8
49.7
63.1
DAB LAB-DETR-C5
42.2
21.5
45.7
60.3
DAB-DETR-C5
44.5 (42.3)
24.2
48.5
63.2
DAB-DETR-DC5
44.5
25.3
48.2
62.3
DAB-DETR-DC5
/
46.7 (42.2)
27.6
50.9
64.0
DN-DETR-C5
44.0
23.9
47.7
62.9
DN-DETR-C5
(41.4)
25.1
49.3
63.8
DN-DETR-DC5
47.5
27.9
50.7
65.9
DN-DETR-DC5
(4-0.)
29.3
52.1
65.4
DAB-Det H.B-Defomabbe-DEET
48.1
31.4
51.4
63.4
MAB--eeforaabe--DERR
49.7 (411.6)
31.4
52.5
65.6
DINO-Deformann DETR DINO-Deformahl--DEIW
50.9
34.6
54.1
64.6
51.3 (4-0.4)
34.7
54.5
65.3
DINO-Defommble-DETR-.. DETR-SWi-LL
58.4
41.0
62.5
73.9
Table 2: Results with a 50-epoch training schedule on MS COCO. We adopt the training schedule of 50 epochs in the table, while for DINO-Deformable-DEN 2020b), we train 36 epochs by following the settings in the original paper (Zhang et al., 2022b). Except that we use Swin-Large (Liu et al., 2021) in the last row of the table, we use ResNet-50 (He et al., 2016) as the backbone. As in Table 1, we do not use multiple patterns and apply dynamic DN groups DN-DETR. For MINO-DeformableDEETe 2022b).
(Zhang al., 2022b; Zhu et al.,
we adopt the 4scale version (Zhang al.,
2020b), even though they already achieve strong results by introducing auxiliary query denoising tasks in model training.
Results with a 50-epoch training schedule. Compared with the original DETR model, the above DETR-base methods achieve a 10x speed up on training. They provide good results with 50 epochs training schedule. We show the effectiveness of Group DETR under this setting. sented in Table 2, Group DETR can also outperform baseline models by large margins. When we adopt stronger backbone, Swin-Large (Liu et al., 2021), we can achieve 58.4 mAP (0.4 mAP higher than its baseline MiNo--eformable-De generalization ability of our Group DETR.
pre-
(58.0 mAP with Swin-Large)), which verifies the
Convergence curves. We report the convergence curves Figure We give each method two convergence curves of the baseline model and the baseline with Group DETR using dashed curves and bold curves. The comparisons Figure support that Group DETR gives a further speed up on DETR training convergence.
4.2 MULTI-VIEW 3D OBJECT DETECTION AND INSTANCE SEGMENTATION
Multi-view 3D object detection. We adopt PETR (Liu et al., 2022b) and PETR v2 (Liu et al., 2022c) as our baseline models. Table 3 shows that Group DETR brings significant gains to PETR and PETR v2. When train PETR v2 with longer training schedule (36 epochs), obtain more improvements on both nuScenes Detection Score (NDS) and mAP on the nuScenes va set (Caesar al., 2020).
Instance segmentation. We adopt Mask2Former (Cheng al., 2021) the baseline and apply Group DETR to it. Table 4, we provide comparisons different training schedules. Group DETR achieves non-trivial improvements Former.

---

Under review as a conference paper at ICLR 2023
Model
w/ Group DETR
Schedule
NDS
mAP
PETR
24e
42.0
37.4
PETR
24e
45.0 (+3.0)
(+1.4)
PETR v2
21e
50.3
40.7
PETR v2
/
24e
51.3 (+1.0)
41.9 (+11.)
PETR v2
36c
50.8
41.3
PETR v2
36c
52.3 (41.55)
42.7 (41.4)
Table 3: Results on multi-view 3D object detection. All experiments are conducted on the
nuScenes val lal set (Caesar et al., 2020). We train these experiments with VoVNetV2 (Lee & Park, 2020) as the backbone and with the image size of 800 x 320. We follow all the settings and hyper- parameters of PETR and PETR v2.
Group DETR
Schedule
mApm
Apm
Apm
APm
12c
38.5
17.6
41.4
60.4
12c
39.7 (+1.2)
18.7
42.8
60.8
50e
43.7
23.4
47.2
64.8
50e
44.0 (+-0.3)
23.8
47.1
65.1
Table 4: Results on instance segmentation. We adopt Mask2Former (Cheng et al., 2021) as the baseline and report the mask mAP (mAP")) on the MS COCO val split for instance segmentation.
4.3 ABLATION STUDIES
We conduct ablation studies object detection by using Conditional DETR-C5 (Meng al., 2021) as our baseline model. The studies include: the influence of group number, the performance each group, the assignment scheme, and the group design.
Influence of group number. Figure shows the influence of the number of groups K in Group DETR. The detection performance improves when increasing the number of groups, and saturates when the group number (K) is greater than 11. Thus, we adopt K 11 in Group DETR in our experiments.
38
36.7
37
37.2
37.6
37.6
37.7
mAP
35.3
35
32.6
32
3
5
7
9
11
13
15
Figure 3: Influence group number (K) in Group DETR. As the number of groups increases, continuous improvement could be achieved compared to the baseline model.
Performance in each group. Table 5 gives the performance of all groups in Group DETR. Each group can achieve similar results, which is consistent with the design of independent groups. In other experiments, we simply report the result of the first group.
Assignment. Figure and Table 6 study the training convergence and performance by keeping the same number (3300) of total object queries about three cases: single-group one-to-one assignment (K = 1), single-group one-to-many assignment (K = with 11 positive object queries for each ground-truth object), and group- sioup one-to-many assignment with K 11 groups (Group DETR). Figure shows that Group DETR and single-gen grgue one-to- one-to-many assignment give significantly faster convergence speeds than single-gun group one-to-one assignment.
Different from single-group one-to-one assignment in DETR (Carion et al., 2020), single dingle-group one- to-many assignment highly depends the post-processing step NMS (Table 6). Our Group DETR

---

Under review as a conference paper at ICLR 2023
group-1
group-2
group-3 group-1 group-5 group-6 group-7 group-8
group-9
group-10 group-11
37.6
37.5
37.4
37.5
37.6
37.5
37.6
37.4
37.5
37.6
37.5
Performance in each group. We show the mAP on the MS COCO val split of different Table 5:
groups in Group DETR. The results obtained by different groups are similar.
mAP vs. epochs
Assignment
NMS
mAP
40
one-to-one
w/o
41.5
one-to-one
w/
41.4
30
one-to-man to-many
w/o
7.2
20
assignment
one-to-many
W/
43.6
assignment
10
group-wise ona-to-many assignment
group-wise one-to-many
w/o 43.4
group-wise one-to-many
w/
43.3
0
5
10
45 50
Figure 4: Training convergence curves of dif- ferent assignment methods. Multiple positive object queries converge faster than positive object query per object.
Table 6: Comparisons of different assignment methods with and without NMS. Results are obtained with a 50-epoch training schedule. We set threshold of 0.7 in NMS following DETR.
w/o NMS achieves almost the same performance as single-group one-to-many assignment w/ NMS, and the inference of our Group DETR is more efficient than it. It is as expected that our Group DETR performs better than single-group one-to-one assignment.
Group design. DN-DETR and our Group DETR both adopt the group design and focus on differ- ent aspects: stabilizing the prediction and group-truth matching (DN-DETR), and exploiting multi- ple positive predictions for one ground-tree -tuth object (ours), respectively. The performance compari- son for DN-DETR and Group DETR in Table shows that Group DETR is superior than DN-DETR. The result of the combination of DN-DETR and Group DETR further improves the performance to 40.6 mAP, implying that the two approaches are complementary.
Group Design
None
DN-DETR*
Group DETR DN-DETR Group DETR
mAP
35.2
38.8
39.1
40.6
Table 7: Comparisons on group designs. We adopt the DAB-DETR (Liu et al., 2022a) as the baseline model (the one with 'None' group design the table) and train all models with a 12-epoch schedule on MS COCO. represents that we report the best results of DN-DETR among different numbers of denoising queries (more results are provided in Appendix B Table 8).
4.4 ANALYSES
The object queries are distributed similarly for all groups. We study the distributions of object queries in each group with conditional DETR as example. Figure 5 depicts 2D ref- erence points (positions) corresponding to the object queries, with one color for one group. We can see that the reference point distributes similarly for all the groups. This provides an explanation to that each group of object queries gives similar detection performances in Table
Visualization of positive object queries. We visualize the positions of positive object queries in all groups in Figure 6. The visualization
Figure 5: Distribution of all groups of object queries with different colors.
8

---

Under review as a conference paper at ICLR 2023
Figure 6: Visualization of positive object queries. To give a neat visualization, we only show one object per image. Each ground-truth box (green bounding box) is assigned to multiple positive queries (red points). We train Group DETR based on Conditional-DETR- number of groups is 11 here. In the figure, the positive queries (red points) may overlap. Best view in color and zoom in.
for 50 epochs. The
shows that the positions of positive object queries are distributed in a certain region on the ob- ject instance. This region is learned by the model and is somehow different from manually selecting center region within the bounding box. This is reasonable, since the center of the bounding box may not have enough information about the object instance. According to the visualization, the posi- tions of all positive object queries are considered good ones by the model to predict the ground-truth objects. the one-to-one assignment, only one of these object queries can be set as positive for the object, while others negatives. The model needs to distinguish the differences among these object queries during training, which impact model learning, leading to slow training convergence. With Group DETR, all these object queries are positives, which gives stronger supervision signals in training, thereby improving training efficiency and speeding up training convergence.
Training memory and training time. Compared with the baseline models, Group DETR adopts more object queries in the transformer decoder during training. It is expected that Group DETR will increase training memory and training time. Thanks to the parallel computation with K groups (shown in Algorithm 1), we only observe ~15% increase in training time. For instance, Group DETR increases the time for training one epoch from 47 minutes (96 minutes) to 51 minutes (108 minutes) on Conditional DETR-C5 (-DC5), from 61 minutes (121 minutes) to 70 minutes (135 minutes) DN-DETR-C5 (-DC5). Group DETR augments the training memory of Conditional DETR-C5 (-DC5) from 7.1 G (13.7 G) to 16.8 (23.0 G), the training memory of DN-DETR-C? (-DC5) from 6.8 G (10.1 G) to 19.8 G (20.4 G). All the above training time and training memory are measured with Tesla A100 GPUs.
5 CONCLUSION
In this paper, we present simple yet effective approach. Group DETR, to accelerate DETR training convergence, We study different assignment methods and propose a novel group-wise one-to-man assignment. It has shown positive results in variety of DbaRe methods and vision tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Amir Bar, Xin Wang, Vadim Kantorov, Colorado Reed, Roci Herzig, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Detreg: Unsupervised pretraining with region priors for object detection. In CVPR, pp. 14605-1461,, 2022.
Alexey Bochkovskiy, Yao Wang, and Hong- Hong-Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004. 109:44444.0094, 2020.
Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S. Davis. Soft-nms improving object detection with one line of code. ICCV, 2017.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, pp. 11621-11631, 2020.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR, June 2018
Xipeng Cao, Peng Yuan, Bailan Feng, and Kun Niu. Cf-detr: Coarse-to-fine transformers for end- to-end object detection. 2022.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey 7agoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. Hybrid task cascade for instance segmentation. In CVPR, June 2019.
Qiang Chen, Yingming Wang, Tong Yang. Xiangyu Zhang, Jian Cheng, and Jian Sun. You only look one-level feature. In CVPR, pp. 13039-13048, 2021.
Qiang Chen, Qiman Wu, Jian Wang, Qinghao Hu, Tao Hu, Errui Ding, Jian Cheng, and Jingdong Wang. Mixformer: Mixing features across windows and dimensions. In CVPR, pp. 5249-5259. June 2022a.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang. Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised represen- tation learning. CoRR, abs/2202.03026, 2022b. URL httph:tt////aaaa 03026.
Xiaokang Chen, Fangyun Wei, Gang Zeng, and Jingdong Wang. Conditional detr v2: Efficient detection transformer with box queries. arXiv preprint arXiv:2207.089144 2022c.
Zhe Chen, Jing Zhang, and Dacheng Tao. Recurrent glimpse-based decoder for detection with transformer. In CVPR, pp. 5260-5269, 2022d.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. arXiv preprint 12.10:21 2021.
Xiyang Dai, Yinpeng Chen, Jianwei Yang, Pengchuan Zhang, Lu Yuan, and Lei Zhang. Dynamic detr: End-to-end object detection with dynamic attention. In ICCV, pp. 2988-2997, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x words: Transformers for image recognition at scale. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:: httpss////ppeeeeeeeeeeee id-YichF FNNTT.
forum??
Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Fast convergence of detr with spatially modulated co-attention. In ICCV, pp. 3621-3630 2021.
10

---

Under review as a conference paper at ICLR 2023
Ziteng Gao, Limin Wang. Bing Han, and Sheng Guo. Adamixer: A fast-converging query query-based object detector. In CVPR, pp. -663,, 2022.
Zheng Ge, Songtao Liu, Zeming Li, Osamu Yoshie, and Jian Sun. Ota: Optimal transport assignment for object detection. In CVPR. pp. 303-312, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. ICCV, 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In CVPR, pp. 16000-16009, June 2022.
J Hosang, R Benenson, and B Schiele. Learning non maximum suppression. PAMI, 2017.
Kang Kim and Hee Seok Lee. Probabilistic anchor assignment with iou prediction for object detec- tion. In ECCV, pp. 355-375111 Springer, 2020.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 1955.
Youngwan Lee and Jongyoul Park. Centermask: Real-time anchor-free instance segmentation. In CVPR, pp. 13906-13915, 2020.
Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In CVPR, 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi. Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint 1rXv:22001.12399, 2022a.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng Yang Fu, and Alexander C Berg. SSD: Single shot multibox detector. In ECCV, 2016.
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. arXiv preprint arXiv:2203.05625, 2022b.
Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Qi Gao, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petrv2: A unified framework for 3d perception from multi-camera images. arXiv preprint arXiv:2206.012 012256 2022c.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv:2103.14030 2021.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin transformer v2: Scaling up capacity and resolution. In CVPR, pp. 12009-12019, June 2022d.
Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li. Yuhui Yuan, Lei Sun, and Jing- dong Wang. Conditional detr for fast training convergence. In ICCV, pp. 3651-3660, 2021.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurIPS, 28, 2015.

---

Under review as a conference paper at ICLR 2023
Byungseok Roh, Jae Woong Shin, Wuhyun Shin, and Sachoon Kim. Sparse detr: Efficient end-to- end object detection with learnable sparsity. ICLR, 2022.
Hwanjun Song, Deqing Sun, Sanghyuk Chun, Varun Jampani, Dongyoon Han, Byeongho Heo, Wonjae Kim, and Ming-Hsuan Yang. Vidt: An efficient and effective fully transformer-based object detector. In ICLR, 2022.
Peize Sun, Yi Jiang, Enze Xie, Wenqi Shao, Zehuan Yuan, Changhu Wang, and Ping Luo. What makes for end-to-end object detection? ICML, pp. 9934 9944. PMLR, 2021a.
Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, al. Sparse r-cnn: End-to-end object detection with learnable proposals. In CVPR, pp. 14454-14463, 2021b.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In ICCV, pp. 9627-9636, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Jianfeng Wang, Lin Song, Zeming Li, Hongbin Sun, Jian Sun, and Nanning Zheng. End-to-end object detection with fully convolutional network. CVPR, pp. 15849-15858, 2021.
Wen Wang, Yang Cao, Jing Zhang, and Dacheng Tao. Fp-detr: Detection transformer advanced by fully pre-training. In ICLR, 2022a.
Yingming Wang, Xiangyu Zhang, Tong Yang. and Jian Sun. Anchor detr: Query design for transformer-based detector. In AAAI, 2022b.
Zhuyu Yao, Jiangbo Ai, Boxun Li, and Chi Zhang. Efficient detr: improving end-to-end object detector with dense prior. arXiv preprint arXiv:2104.01318 2021.
Gongjie Zhang, Zhipeng Luo, Yingchen Yu, Kaiwen Cui, and Shijian Lu. Accelerating detr conver- gence via semantic-aligned matching. In CVPR, pp. 949-958, 2022a.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.0366 03605, 2022b.
Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z Li. Bridging the gap between anchor-base based and anchor-free detection via adaptive training sample selection. In CVPR, pp- 9759-9760 2020.
Benjin Zhu, Jianfeng Wang, Zhengkai Jiang, Fuhang Zong, Songtao Liu, Zeming Li, and Jian Sun. Autoassign: Differentiable label assignment for dense object detection. arXiv preprint arXiv:2007.03496, 2020a.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. CoRR, abs/2010.04159, 2020b. URL org//bo// /2010, 04159.
https://aat /taps:://axivv /ttts://arxtv.orrgbbb//////
12

---

Under review as a conference paper at ICLR 2023
APPENDIX
A DATASETS AND EVALUATION METRICS
We perform the object detection and instance segmentation experiments on the COCO 2017 (Lin al., 2014) dataset, which contains about 118K training (train) images and 5k validation (val) images. Following the common practice, we report the standard mean average precision (mAP) result (box mAP for object detection and mask mAP for instance segmentation) the COCO vali- dation dataset under different IoU thresholds and object scales.
We perform multi-view 3D object detection experiments the nuScenes (Caesar al., 2020) dataset, which contains 1000 driving sequences. There are 700 for train set, 150 for val set and 150 for test set. We report the standard nuScenes Detection Score (NDS) and mean Average Precision (mAP) result on the nuScenes val set.
B ADDITIONAL RESULTS
Results of DN-DETR with different number of denoising queries. We conduct experiments with different numbers of denoising queries in DN-DETR (Li et al., 2022). The results Table suggest that increasing the number of denoising queries can not achieve further improvements and show unstable performances. The effects denoising queries differ from the ones of Group DETR (Figure 3). The denoising queries mainly aim to solve the instability in the matching process, while our Group DETR aims to exploit multiple positive queries for one ground-tree truth object. We choose to use 100 denoising queries in experiments in Table I and Table 2 by following the setting in the original paper (Zhang et al., 2022b).
#Denoising Queries 100
300
600
900
1200
1500
1800
2100
2400
2700
3000
3300
mAP
38.6
38.8
37.8
38.7
38.5
38.1
38.7
37.9
38.1
38.7
38.1
38.7
Table 8: Results of DN-DETR with different number of denoising queries. We show the detec- tion performances (mAP) MS COCO (Lin et al., 2014) of adopting different number of denoising queries in DN-DETR.
13


Reviews:
[{"review_id": "fqGv3y_yAW5", "reviewer": "Reviewer_MZFA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a training approach to improve the performance of DETR-like object detectors. Namely, it add more query groups as the decoder inputs, and each group of queries is completely independent of each other during training. It means that the original one-to-one label assignment is adopted for each group. At inference, only the first group of queries is retained and the rest are discarded. The paper conducts experiments with a few DETR-like detectors and show various improvements over its original counterpart.", "summary_of_the_review": "As stated in the strengths and weakness, this paper shows some improvements over its counterparts, however, the proposed group queries seems more like an engineering trick rather than a novelty. Furthermore, it lacks some important experiments.", "strength_and_weaknesses": "Strengths\n++ The proposed training method is conceptually simple and can be adopted by different DETR-like methods.\n++ There are adequate experiments, including 2D and 3D object detection, instance segmentation. The proposed method can bring benefits to different extend.\n++ The paper is easy to follow.\n\nWeakness\n-- The technical novelty is limited. Similar to DN-DETR, the motivation of group DETR is simply adding more training to the decoder. Moreover, the choice of retaining the first group seems very hand-designed and lack a theoretical explanation.\n-- There are some key experiments missing. First, for a baseline (a standard DN-DETR, for example), what is the performance using K*N queries, comparing to group DETR using K groups of N query/group?  Second, although the total training epochs are the same between group DETR and its counterparts, the training time increases for group DETR (~15% increase). So, what is the performance of training a baseline with 15% more training epochs? In this case, the total training time is comparable, rather than the training epochs.\n-- The increase in GPU memory (i.e. almost double) is significant.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SCR4Y49e6K", "reviewer": "Reviewer_5yMc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple and effective method to accelerate the convergence speed and improve the performance of detection transformers, which only need to randomly initialize different groups of queries and use them to train the decoder separately. The method can be applied to all existing detection transformers and exhibits consistent improvements.", "summary_of_the_review": "Overall, this paper adopts a simple yet effective strategy to train detection transformers and brings consistent improvements in both performance and convergence speed on multiple kinds of detection transformers.\nHowever, the paper does not give a sufficient explanation or insights about how and why it works, which limits its contribution and makes the paper more like a technical report rather than a paper for this venue.\nFurthermore, some implementation details are missing.", "strength_and_weaknesses": "## Strength\n1. The paper is well-written and easy to follow\n2. The idea is simple yet effective, which only use multiple group of queries to train the decoder.\n3. The experimental results show that the idea can be applied to and improve many kinds of detection transformers.\n\n## Weakness\n1. The explanation about why this method could work is not sufficient, i.e., the reason why the authors do this modification and why it works need further explanation. For now, we only know that it could work, but providing more about why could bring more insights and values to the community.\n2. The explanation of the method increases the number and diversity of positive samples seem can only to explain it can help the model coverage faster. But the paper shows that it still works when training 50 epochs. Will the method still work for a longer training schedule (, e.g., by 150 or 300 epochs)? If yes, does the explanation claimed in the paper still make sense?\n3. How to apply the method to some dense version DETRs (such as DINO) is not described in detail in the paper. The paper only provides experiment results for the combination of DINO and Group DETR. Furthermore, given the minor improvements over DINO,  can the explanation that increasing the number and diversity of positive samples still holds for these dense version DETRs?\n4. There might be some contradiction in the explanation in the last several sentences in Sec 4.4. For example, although all similar queries in different groups are positive samples in Group DETR, the positive query in one group may also be negative samples in another group (or in other groups, there might be some queries that are very similar to the positive query in the current group), which is not shown in Figure 6.\n5. Typos: group-truth matching -> ground-truth matching in the last sentence of the 2nd paragraph in Sec.1.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6Gc1Vou0r0", "reviewer": "Reviewer_ySfN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new training strategy to improve the performance of DETR-based object detectors. The authors indicate the main reason for the low convergence rate of DETR-based object detectors is in the one-to-one matching and propose a simple strategy of group-wise one-to-many assignment to address. It is a simple plug-and-play strategy that can be applied to any DETR-based object detectors in the training phase. Extensive experiments have been conducted on 2D object detection (MS COCO), instance segmentation (MS COCO), and multi-view 3D object detection (Nuscenes) to show the advantage of the proposed approach. ", "summary_of_the_review": "+ The proposed method is simple and achieved impressive results on several DETR-based detectors and multiple datasets. However, there is a trade-off between performance versus runtime and memory consumption.\n+ The idea of group-wise one-to-many assignments is quite similar to H-DETR. I am happy to increase this paper’s score if the authors provide experiments with the suggested setting above.", "strength_and_weaknesses": "*Strengths*\n+ The paper is clearly written and easy to follow. \n+ The proposed method is simple.\n+ Extensive experiments have been conducted to verify the performance of the proposed method.\n\n*Weaknesses*\n+ This is not mandatory but it would be better to cite the Hybrid DETR (H-DETR) paper: https://arxiv.org/abs/2207.13080 and have a discussion to show to clear difference between the two. \n+ The proposed method is orthogonal to the previous improvements of DETR so that when adding group-wise one-to-many assignments to others the performance is improved with a cost in training memory and time. There is no clear significant improvement over prior improvement. For example in Tab. 1, comparing the DN-DETR on top of the DAB-DETR with the denoising part and the Group-DETR on top of the DAB-DETR, the DN-DETR and Group-DETR are comparable (+3.4 vs +3.9 for C5 backbone and +4.4 vs +4.4 for DC5 backbone). Also, when adding Group-DETR on top of DINO, the improvement becomes less significant. In other words, adding Group-DETR on top of similar improvements results in less effectiveness. \n+ The experiments are quite not fair. As Group-DETR increases the number of queries K times (K is the number of groups), it takes a longer time to train an epoch. Therefore, it would be better to compare the results of the proposed method with the previous methods on the same training hours, not the same number of epochs.\n+ It is not clear why adding new groups improve the performance since there are more negative queries forcing in-object queries to be negative as well. The authors should have a better explanation.\n+ Failure cases and limitations of the proposed method should be added since the proposed approach is not perfect.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject