Learning Dynamic Query Combinations for Transformer-based Object Detection and Segmentation
Authors: Yiming Cui
Linjie Yang
Haichao Yu
Venue: Submitted to ICLR 2023 (2023)

Abstract: Transformer-based detection and segmentation methods use a list of learned detection queries to retrieve information from the transformer network and learn to predict the location and category of one specific object from each query. We empirically find that random convex combinations of the learned queries are still good queries for the corresponding models. We then propose to learn a convex combination with dynamic coefficients based on the high-level semantics of the image. The generated dynamic queries better capture the prior of object locations and categories in the different images. Equipped with our dynamic queries, a wide range of DETR-based models achieve consistent and superior performance across multiple tasks (object detection, instance segmentation, panoptic segmentation) and on different benchmarks (MS COCO, CityScapes, YoutubeVIS).

Under review as a conference paper at ICLR 2023
LEARNING DYNAMIC QUERY COMBINATIONS FOR TRANSFORMER-BASER OBJECT DETECTION AND SEGMENTATION
Anonymous authors Paper under double-bling review
ABSTRACT
Transformer-based detection and segmentation methods use a list of learned de- tection queries to retrieve information from the transformer network and learn to predict the location and category of one specific object from each query. We empirically find that random convex combinations of the learned queries are still good queries for the corresponding models. We then propose to learn a convex combination with dynamic coefficients based the high-level semantics of the image. The generated dynamic queries better capture the prior of object locations and categories in the different images. Equipped with our dynamic queries, wide range of DETR-based models achieve consistent and superior performance across multiple tasks (object detection, instance segmentation, panoptic segmentation) and on different benchmarks (MS COCO, CityScapes, YoutubeVIS;
INTRODUCTION
Object detection is fundamental yet challenging task in computer vision, which aims to localize and categorize objects of interest in the images simultaneously. Traditional detection models (Ren al., 2015; Cai & Vasconcelos, 2019; Duan et al., 2019; Lin et al., 2b;ab;a) complicated an- chor designs and heavy post- poost-rrocesiing. steps such as Non-Maximum Maxximum-Supprrssionn (NMS) to remove duplicated detections. Recently, Transformer-bar based object detectors such DETR (Carion al., 2020) have been introduced to simplify the process. In detail, DETR combines convolutional neu- ral networks (CNNs) with Transformer (Vaswani et al., 2017) by introducing an encoder-decoder framework to generate a series of predictions from list of object queries. Following works improve the efficiency and convergence speed of DETR with modifications to the attention module (Zhu et al., 2021; Roh et al., 2021), and divide queries into positional and content queries (Liu et al., 2022; Meng et al., 2021). This paradigm is also adopted for instance/panoptic segmentation, where each query is associated with specific object mask in the decoding stage of the segmentation model (Cheng et al., 2021a).
The existing DETR-based detection models always use list of fixed queries, regardless of the input image. The queries will attend to different objects in the image through a multi-stage attention process. Here, the queries are served global priors for the location and semantics of target objects in the image. In this paper, we would like to associate the detection queries with the content of the image, i.e., adjusting detection queries based on the high-level semantics of the image in order to capture the distribution of object locations and categories in this specific scene. For example, when the highlevel semantics show the image is a group photo, we know that there will be group of people (category) inside the image and they are more likely to be close to the center of the image (location).
Since the detection queries are implicit features that do not directly relate to specific locations and object categories the DETR framework, it is hard to design a mechanism to change the queries while keeping them within a meaningful "query" subspace to the model. Through empirical study, we notice that convex combinations of learned queries are still good queries to different DETR-based models, achieving similar performance the originally learned queries (See Section 3.2). Motivated by this, we propose a method to generate dynamic detection queries based on the high-level semantics of the image in DETR-base methods while constraining the generated queries

---

Under review as a conference paper at ICLR 2023
50.5
120 Quarten
18.5
-8.5
42.5
DAB Defemsste DEM
Cc.04..ufeemsse.cmm CET
Delemabre DETR, 00 Ro-Belamatre BETA
Figure 1: Comparison of DETR -eated detection models integrated with and without our methods on MS COCO (Lin et al., 2014) val benchmark. ResNet-50 is used as the backhone.
in a sequence of convex hulls spanned by the static queries. Therefore, the generated detection queries are more related the target objects in the image and stay in meaningful subspace. Exten- sive experiments on MS COCO (Lin et al., 2014), CityScapes (Cordts et al., 2016) and YouTube- VIS (Yang al., 2019) benchmarks with multiple tasks, including object detection, instance segmenta- tion, and panoptic segmentation show superior performance of our approach combined with wide range of DETR-based models. Figure 1, we show the performance of our method on object detection combined with two baseline models. When integrated with proposed method, the mAP recent detection models DAB--eformable-DETT (Liu et al., 2022) can be increased by 1.6%. With fewer dynamic detection queries and less computation in the transformer decoder, method can still achieve better performance than baseline models on both Deformable-DETR and DAB- Se--Deformable-Derr
2 RELATED WORKS
Transformers for object detection. Traditional CNN-based object detectors require manually de- signed components such as anchors (Ren et al., 2015; Tian et al., 2019) or post-processing steps such NMS(Neubeck & Van Gool, 2006; Hosang al., 2017). Transformer-based detectors di- rectly generate predictions for a list target objects with a series of learnable queries. Among them, DETR (Carion et al., 2020) first combines the serpuecce-t-seqqeeeee able queries and CNN features for object detection. Following DETR, multiple works were proposed improve its convergence speed and accuracy. Deformable-DEs DETR (Zhu et al., 2021) and Sparse- DETR (Roh et al., 2021) replace the self-attention modules with more efficient attention operations where only small set of key-value pairs are used for calculation. Conditional-DEN DETR (Tian al., 2020) changes the queries in DETR to be conditional spatial queries, which speeds up the conver- gence process. Anchor-DETR (Wang et al., 2021 generates the object queries using anchor points rather than set of learnable embeddings. DAB-DETR (Liu et al., 2022) directly box coordinates as queries which can refined in the Transformer decoder layers. DINO (Zhang al., 2022) and DN-DETR Li al. (2022) introduce strategy train models with noisy ground truths to help the model learn the representation of the positive samples more efficiently. Recently, Group-DETR Chen et al. (2022) and HDETR Jia al. (2022) both added auxiliary queries and one-to-many matching loss to improve the convergence of the DETR-bas based models. They still use static queries which does not change the general architecture of DETR. All these Transformer- based detection methods use fixed initial detection queries learned on the whole dataset. In contrast, we propose modulate the queries based on the image's content, which generates more effective queries for the current image.
framework with learn-
learnable
Transformers for object segmentation. Besides object detection, Transformer-basee models are also proposed for object segmentation tasks including image instance segmentation, panoptic seg- mentation (Kirillov et al., 2019; Wang et al., 2021a; Zhang et al., 2021) and video instance seg- mentation (VIS) (Yang et al., 2019). In DETR (Carion et al., 2020), a mask head is introduced on top of the decoder outputs to generate the predictions for panoptic segmentation. Following DETR, ISTR (Hu al., 2021) generates low-dimensional mask embeddings, which are matched with the ground truth mask embeddings using Hungarian Algorithm for instance segmentation. SOLQ (Dong al., 2021) unified query representation for class, location, and object mask. Fask2Former
2

---

Under review as a conference paper at ICLR 2023
DAB-DETR
Deformale-DETR
Mask2Formerr
Model
r=
r=
Convex Combination
37.9(#0.11)
30.4(=0.20)
35.0(++.220)
242(+00005
41.2(=0.10)
Non convex Combination
37.0(
29.5(10.10)
32.6(10.22)
24.0110.10)
0..75=0.45)
Averaged Comhination
37.0
28.4
32.9
22.5
40.9
Queries sampled randomly
(9.7(=:0.05)
33.9(10.15)
39.8(+0.30)
28.1(20.30)
41.7(=0.10)
Table 1: Comparison of pretrained detection models DAB-DETR (Liu et al., 2022) and Deformable-
DETR and segmentation model Mask2Former (Cheng al., 2021a) with different queries. The shown metrics are box mAP for detection and mask mAP for segmentation. ResNet-50 is used as the backbone and models are evaluated on MS COCO val.
(Cheng et al., 2021a) introduces masked attention extract localized features, and predict output for panoptic, instance and semantic segmentation a unified framework. SeqFormer (Wu et al., 2021) utilizes video-level instance queries where each query attends to a specific object across frames in the video. These Transformer-based models follow the general paradigm of DETR and use fixed queries regardless of the input.
Dynamic deep neural networks. Dynamic deep neural networks (Han et al., 2021) aim at adjusting the computation procedure of neural network adaptively in order to reduce the overall computation cost or enhance the model capacity. Slimmable networks (Yu et al., 2018; Yu & Huang, 2019; Li al., 2021) introduce strategy adapt multiple devices by simply changing channel numbers without the need for retraining. Dynamic Convolution (Chen et al., 2020) proposes dynamic perceptron that uses dynamic attention weights to aggregate multiple convolution kernels based on the input features. On object detection, Dynamic R-CNN (Zhang et al., 2020) proposes new training strategy to dynamically adjust the label assignment for two-stage object detectors based on the statics of proposals. Cui et al. (2022) proposes to train a single detection model which can adjust the number proposals based on the complexity of the input image. Wang et al. (202 1c) introduces Dynamic Transformer to determine the number of tokens according to the input image for efficient image recognition, by stacking multiple Transformer layers with increasing numbers of tokens. In contrast the existing work, explore generating dynamic queries for wide range of DETR-based models using the same framework. Our focus is not to reduce the computation cost of DETR- based models, but improve the model performances with queries more related to the content of each individual image.
3 METHODOLOGY
3.1 PRELIMINARY
We first summarize the inference process the existing Transformer-basee models for a series of tasks, including object detection, instance segmentation, and panoptic segmentation, the following Equation:
Ni (Nace (Nenc (F), Q)).
(1)
For the object detection task, given the input image I, multi-scale features F are extracted from the backbone network and then fed into a Transformer encoder Nene- After processing the features with multiple encoder layers, the output features fed into Transformer decoder Ndec together with n randomly initialized query vectors Q E "xf, where n and ff denote the number of queries and length of each query respectively. Each query be a feature Vector (Carion et al., 2020; Zhu et al., 2021), )), or a learned anchor box (Liu et al., 2022). The outputs of Noeo is then fed into a task head M, to generate the final predictions Y {(b,, c.) bounding boxes and their corresponding categories of the detected objects. Then, the predictions are matched with the ground truths Y using the Hungarian Algorithm (Carion al., 2020) generate a bipartite matching. Then, the final loss is computed based on this bipartite matching:
1,2..... n). where b, C; represent the
C = CHungarian (Y,
(2)
For the segmentation tasks, the final predictions are updated to Y = {(b,, c; m;) i 1, .......... n). where m, denotes the predicted masks for different object instances. Since there is direct cor-
3

---

Under review as a conference paper at ICLR 2023
respondence of the predictions with the ground truth annotations, bipartite matching is also com- puted to find the correspondence of the predictions and the ground truths Y*. The final loss is then computed based the matching. In some models such Mask2Former (Cheng et al., 2021a), there will be no Transformer encoder Nene to enhance the feature representations, while the other computational components follow the same paradigm.
3.2 FIXED QUERY COMBINATIONS
Though some existing works analyze the contents of the queries for the decoder, such as Conditional- DETR (Tian et al., 2020) and Enchor-DETR (Wang al., 2021b), they always exam each query individually, To the best of our knowledge, there is no work studying the interaction between the queries in Q. Here, we would like to explore what kind of transformations conducted between the learned queries still generate "good" queries. If we compute the average of few queries, is it still an effective query? we use different types of linear transformations, which would be better to produce good queries?
We conduct experiments to analyze the results of queries generated by different perturbations from the original queries. The procedure of the experiments is follows: given well- trained Transformer-basee
model, the initial queries for the decoder are denoted as Q" {9f q v."........ E Raxs, The first type of perturbation uses linear combinations of the origi- nal queries. We first separate the n queries into m groups, where each group has r a queries and generates one new query. Then, we initialize the combination coefficients W E Rm where Wij E W is the coefficient used for the i-th group, j-th queries, denoted as z.. to generate group of new queries Q = {gf of .............................. am E gmxf, The process can be summarized as:
qi
wig wiy
(3)
We use three settings to evaluate the impact of different coefficients in Equation 3. namely Convex Combination, Non-convex Combination, and Averaged Combination. In Convex Combination, qf is within the convex hull of j = 1,2,
r. The combination coefficients Wij are randomly initialized using uniform distribution in [-1.1] 1] and then passed through softmax function to sat- isfy the criteria: Wij- 2 O, Σn= Wiy = For Non-convex Combination, Waj are initialized in the same way as those in the convex combination, and the sum of wy is forced to be 1. However, there is no guarantee on its range and Wij can be negative values. For Averaged Combination, we generate of by averaging pffn.j = 1,2, 1,2,..., As a baseline, we evaluate the model on m queries ran- domly sampled from Q". The experiments are conduct on MS COCO benchmark (Lin et al., 2014) for object detection, and instance segmentation, using DAB-DETR (Liu et al., 2022), Deformable- DETR (Zhu et al., 2021) and Mask2Former (Cheng et al., 2021a), with ResNet-50 as the backbone. The results are summarized in Table 1. From Table 1, we notice that Convex Combination achieves the best results among all the compared settings except the baseline. Convex Combination only de- generate slightly compared with learned queries on DAB-DETR and Mask2Former. In addition, the performance of Convex Combination only has very small variances across different models, proving that convex combinations of the group-wiss learned queries are naturally high-quality object queries for different Transformer- based models on both detection and segmentation tasks. n is to 300 for detection models and 100 for Fask2Former. We run each setting 6 times to compute the variance.
3.3 DYNAMIC QUERY COMBINATIONS
From the previous section, we learn that fixed convex combinations of learned queries are still able to produce reasonable accuracy compared to the learned queries. In this section, we propose strategy to learn dynamic query combinations for the Transformer-bass models instead of randomly generating the coefficients Why for query combinations. Our model predicts their values according to the high-level content of the input images. Therefore, each input image will have a distinct set of object queries fed into the Transformer decoder.
To generate dynamic queries, a naive idea is to generate the modulated queries directly from the in- put features F. This method will increase the number of parameters dramatically, causing it difficult
4

---

where A is global average pooling to generate global feature from the feature map F, 0 is an MLP, a a softmax function to guarantee the elements of W satisfy the convex constraints. simple possible show the potential of using dynamic Under review as a conference paper at ICLR 2023
Global Pooling
Clobal
MLP
Image Testame
Comvitatior Collieees
Ben Quaries
Mocilladl Quaniix
Only
Class
Decodes
Combiration Coefficients
Baric Queries
Modullned Qurvies
The framework of the proposed method. (a) Model pipeline with dynamic query combi- nations. The step with dashed line is only used in training. (b) Illustration of generating modulated queries from basic queries given combination coefficients.
Figure 2:
to optimize and inevitably computationally inefficient. To verify this, we conduct an experiment on Deformable-DETR (Zhu et al., 2021) with ResNer-50 as the backbone. We replace the original ran- domly initialized queries with those generated by a multi-layer perceptron (MLP), which transforms the image feature F to Q. With 50 epochs, the model only achieves 45.1% mAP, which is lower than the original model with 46.2%.
Inspired by the dynamic convolution (Chen al., 2020), which aggregates the features with multiple kernels in each convolutional layer, we propose a query modulation method. We first introduce two types of queries: basic queries Q E RRax// and modulated queries Qu Rmxx, where n, m the number of queries and n = rm. Equation is updated as:
M q:
Wi Yz3
j=1
where W Rmxr is the combination coefficient matrix and w W is the coefficient for the i-th group, j-th query in Q denoted as d. To guarantee our query combinations to be convex, we add extra constraints to the coefficients as w 2 0, Σj= 1.
In our dynamic query combination module, the coefficient matrix WD is learned based the input feature F through mini-networt as:
WD = a (A
(5)
Here try to make the mini-networ queries. This attention-style structure happens to simple and effective design choice.
During the training process, feed both QM and Q to the decoder generate the corre- sponding predictions YM and B as follows,
YM N₁ (Nacc (Nenc (F), (F)
B = No (Natc (Nenc (F), "))
(6)
The final training loss is then updated to
C Llungarrin (YM Y+) + 3 BL (Y
B Y*)
where 3 is a hyperparameter. During the inference, only QM is used to generate the final predictions
M while the basic queries Q are not used. Therefore, the computational complexity increases
5

---

Under review as a conference paper at ICLR 2023
Backbone
Method
mAP
APo.5
AP0.5
Conditional-DENN (Tian et al., 2020)
40.9
61.7
43.3
DQ--onditiinal--EETt
42.0
63.3
44.2
SMCA-DETR (Gao et al., 2021)
41.0
61.5
43.5
DQ-SMCA-DETR
42. 42.1
63.3
44.971.4
DAB-DETR (Liu et al., 2022)
42.1
63.1
44.6
ResNet-50
DQ-DAB-DETR
43.7
64.4
46.6y2.0
Deformable-DETR (Zhu et al., 2021)
46.2
65.0
49,9
DQ-Defformable-DETTR
47.0po.8
10.5
50.9
DA
DAAB-Deformaabe-DDERR (Liu et al., 2022)
48.1
66.4
52.0
Do--DAB-Deffomabbe--DDrTt
49.7
68. 11.7
54.2
Deformable-DETTR (Zhu et al., 2021)
50.9
70.5
55.3
DQ-Deformable-DEET
53.2
72.8f2.3
57.7
Swin-Base
DAB-Defommabl--EERR Liu et al. (2022)
52.7
71.8
57.4
DQ-DAB--eformable--D
53.8
72.8
58.611.2
Table 2: Comparison of existing DETR-based object detectors with/without proposed methods integrated on MS COCO val split. ResNet-50 is used as the backbone.
for our models are negligible compared the original DETR-based models. The only difference in the computation is that we have an additional MLP and convex combination to generate the modulated queries. Since the role modulated queries in our model is exactly the same as the fixed object queries in the original models, we use modulated queries and queries interchangeably to refer to the modulated queries.
EXPERIMENTS
To evaluate the effectiveness of our proposed methods, we first conduct experiments on a series of tasks, including object detection, instance segmentation, panoptic segmentation, and video instance segmentation with different DETR-based models. Then we conduct several ablation studies to in- vestigate the impact of different hyperparameters in model. Detailed experiment setups and visualization examples are provided in the supplementary materials.
4.1 EXPERIMENT SETUP
Datasets. For the object detection task, MS COCO benchmark (Lin et al., 2014) for eval- uation, which contains 118, 287 images for training and 5, 000 for validation. For instance and panoptic segmentation, besides the MS COCO benchmark (80 "things" and 53 "stuff" categories), we also conduct experiments on the CityScapes (Cordts et al., 2016) benchmark (8 "things" and 11 "stuff" categories) to validate the effectiveness of our proposed method. For the VIS task, YouTube VIS-2019 (Yang et al., 2019) is used for evaluation. For experiments on VIS, we pretrain our models on MS COCO and finetune them on the training set of YouTube- VIS-
Evaluation metrics. For panoptic segmentation, the standard PQ (panoptic quality) metric (Kirillov al., 2019) used for evaluation. For instance segmentation (image or video) and object detection, we use the standard mAP (mean average precision) metric for evaluation. For VIS, mAp and AR (average recall) on video instances are evaluation metrics.
Implementation details. The query ratio used to generate the combination coefficients is set to 4 by default. is set to be is implemented as a two-layer MLP with ReLU as nonlinear activations. The output size of its first layer is 512, and that of the second layer is the length of D in corresponding models. For detection models, we use 300 queries if not specified otherwise. For segmentation models, queries for Formerormer on VIS, and 300 queries for SeqFormer on VIS. ResNet50 is used as the backbone for different models.
50 queries for Mask2Former image segmentation tasks, 100

---

Under review as a conference paper at ICLR 2023
Methods
mAP
APo.5
APa.75
Mask R-CNN (He et al., 2017)
35,4
56.4
37.9
QueryInst (Fang al., 2021)
39.8
61.8
43.1
Mask2Former (Cheng al., 2021a) (50 queries)
42,4
64.3
45.7
DQ-Mask2Former (50 queries)
43.2
65.2
46.7
Mask2Former (100 queries)
43.3
65.5
46,9
DQ-Mask2Former (100 queries)
44.1
66.5
47.4po.5
Table 3: Comparison of existing instance segmentation approaches and DQ-Mask2Former on MS COCO val split. All models use ResNet-50 as the backbone.
Methods
PQ
RQ
PQth
RQth
UPSnet (Xiong et al., 2019)
42.5
52.5
48.6
59.6
DETR (Carion et al., 2020)
43.4
53.8
48.2
59.5
Mask2Former (Cheng al., 2021a)
50.4
59.9
55.8
65.9
FQ-Mer
1
60.6
56.9
67.1
Table Comparison of existing panoptic segmentation approaches with DQ-Mask2Former on MS COCO val 1 split with ResNet-50 as the backbone.
Panoptic
Instance
Method
PQ
RQ
PQts
RQth
mAP
APo.s
Mask2Former
60.3
73.2
50.6
63.0
36.7
60.9
DQ-Marr2Former
61.6 1.3
74.3
53.5
65.7
37.5
62.2
Table 5: Comparison of Mask2Former and DQ-Mask2Former on panoptic and instance segmenta- tion tasks on CityScapes val split with ResNet-50 as the backbone.
4.2 MAIN RESULTS
Object detection. We evaluate our proposed methods with the DETR-based models Deformable- DETR (Zhu et al., 2021), SMCA-DETR (Gao et al., 2021), Conditional-DETR (Tian et al., 2020), DAB-DETR and DAB--eformable-DEETR (Liu et al., 2022) for object detection on the MS COCO benchmark. For fair comparison, we run the original model integrated with and without our proposed dynamic queries using the same experimental settings, including the number of queries and epochs. The models equipped with our dynamic query combinations are denoted as DQ- Deformable-DETR, DQ-SMCA-DETR, DQ-Conditional-D DEERR, DQ-DAB-DETR, and DQ-DAB- Deformable- DETR, respectively. The results are shown in Table 2. From Table 2, when integrated with our proposed method, mAP can be improved consistently by at least 0.8% for all the models listed in the table. For DAB-Deformable-DDi backbone and 1.1% with Swin-B backbone. For Deformable-DETTe the mAP can be improved significantly by 2.3% with Swin-B backbone. This proves the benefit of our method with different backbones. Note that models with dynamic queries only have negligible increased computation cost compared to the original models.
the mAP can be improved by 1.6% with ResNet50
Instance/panoptic segmentation. Mask2FormerrCheng et al., 2021a) recent state-of-the-art model that can be used for different segmentation tasks with unified model architecture. We com- pare Mask2Former with/withoun our dynamic queries for image instance and panoptic segmentation tasks on the MS COCO (Lin al., 2014) and City (Cordts al., 2016) benchmarks. The model plugged with dynamic queries is named DQ 2o0rrmrr.Frrmer. The results are shown as in Table 3, and 5. We use 50 queries for all the listed settings. For instance segmentation (Table and 5), our model DQ-Mask2Former achieves consistent improvement across different metrics compared to the original Mask2Former. For example, the performance on mAP improved by around 0.8% on both MS COCO and CityScapes. For panoptic segmentation, shown in Table 4 and 5, DQ- Mask2Former again significantly outperforms Mask2Former (Cheng et al., 2021a) across all the evaluation metries both MS COCO and CityScapes.

---

Under review as a conference paper at ICLR 2023
Method
mAP
APo.s
AP
AR
MaskTrack R-CNN (Yang et al., 2019)
30.3
51.1
32.6
35.5
IFC (Hwang et al., 2021)
42.8
65.8
46.8
51.2
Mask2Former (Cheng al., 2021a)
45.2
65.8
48.9
55,4
DQ-Mask Former
46.3
68.6 68.6y2.8
50.6,1.7
56.4
SeqFormer (Wu et al., 2021)
46.0
68.5
50.4
53.6
DQ-SeqFormer
47.5
70.3
52.0f1.6
55.0
Table 6: Comparison of existing video instance segmentation approaches with DQ-Mask2Former and DQ-SeqFormer on YouTube- YouTube-VIS-2019 val split with ResNet-50 as the backbone.
3
mAP
APo.5
AP
APs
APM
API.
0.0
45.6
64.1
49.4
27.2
49.1
60.5
0.5
46.4
65.0
50.3
28.1
49.2
62.6
1.0
47.0
65.5
50.9
28.8
50.1
62.2
Table 7: Analysis of 3 using DQ- De-D--oombbb-DD-er (ResNet-50 as the backbone) on the MS COCO benchmark with different settings.
Video instance segmentation. Besides image tasks, we also evaluate our method on the video instance segmentation task. We evaluated our method on two state-of-the-art VIS methods Mask2Former (Cheng et al., 2021b) and SeqFormer (Wu et al., 2021). Results are shown in Ta- ble 6. It can be seen from Table 6 that when integrated with dynamic queries, mAP and AR of Mask2Former are improved by at around 1.0%. The mAP SeqFormer is significantly boosted by 1.5%. Note the additional computation cost are negligible for these two models.
4.3 MODEL ANALYSIS
Analysis of number of queries. We use Deformable-DETR and DAB-Defommabi-DETTs as base- line models to study the effects of number queries on the performance of object detection. We compare the baseline models with D--Deformable-EETT and DQ-DAB-Deformale Deformable-DeDTT inte- grated with different numbers of queries in Figure Note that we include the additional compo- nents of our models into the FLOPs computation of the decoder. When integrated with our method, even by reducing the number of queries from 300 to 100, the mAP of DQ-Deformable-DFT and DQ-DAB Dew.--eebee-eeeeeeeeeee
still better than the bascline models with 300 queries. We also able to reduce the computation cost of the decoders of Deformable-DEr -EDR and DAB-Deformable- DETR by about 14% and 24% by using our method with 100 queries, respectively. However, do not observe significant speedup using our method with fewer queries mainly because the main computation costs are from the backbones and the transformer encoders.
Analysis of number of training epochs. In Figure 3 (a), we show the impact of the number of training epochs sample model DO-Deforrmable-DErR together with the original Deformable- DETR. From the figure, the mAP of D0-Defformalle-EET is always better than that of the orig- inal Deformable-DEr DETR at different epochs on the MS COCO benchmark. At early epochs around 30, DO-Deformable-DETR achieves an even more significant performance gain compared then Deformable-DEr DETR compared with later epochs.
Analysis of ,3. We analyze the impact of the scale of 3 on models equipped with our dynamic queries. We conduct experiments using Reformable-DETR with ResNet-50 as the backbone on the MS COCO benchmark with different values of B. Results are shown in Table 7. As shown in the table, when set to be O, where no loss is directly computed with the prediction from the basic queries, the performance drops by 2.4% compared to the original setting. In this case, the basic queries are not necessarily proper queries for the detection model, which will affect the quality of the modulated queries produced by them. The performance can be improved by increasing the value of B to 0.5. Empirically we find | = is a good choice to balance the scale of losses between the basic and modulated queries.
Analysis of query ratio. We use DQ- Do-Deeormmble-EETT (Zhu et al., 2021) to analyze the perfor- mance of our proposed methods with different query ratios 4, and 8, as in Figure 3 (b). From
8

---

Under review as a conference paper at ICLR 2023
Andyy Number Spocss
47.5
Analeti en Ousry Ratlo
Celernare
Querles)
Oeformable 1302
47.0
$4.0
$2.0
Spcera
48.0
Model
(b)
Figure 3: Analysis of the impact of number of epochs and query ratios the performance.
Figure 4: t-SNE visualization of W on 200 images from MS COCO val. Zoom in to see details.
the figure, using 4 as the query ratio achieves the best performance for DQ-Deformable-DETT with 300 queries. However, other query ratio choices still generate better accuracies than the original Deformable-DETR, which validates the effectiveness and robustness of our method.
Visualization of W Since W is conditioned on the high-level content of the image, jecture that images with similar scenes or object categories may have similar WD parameters. We choose 200 images from the validation set of MS COCO and compute their WD from DQ-DAB- DETR with 300 queries. The resulting WD are first flattened into vectors, and then projected onto two-dimensional space using t-SNE Van der Maaten Hinton, 2008). We visualize the projected D parameters along with their corresponding input images as Figure 4. We can see that some object categories tend to be clustered. For example, can see lot of transportation vehicles in the top right corner of the figure. and wild animals tend to be in the lower part of the figure, which indicates that the model uses some high-level semantics of the image to produce the combination coefficients.
CONCLUSION
In this paper, we propose to use dynamic queries depending on the input image to enhance DETR- based detection and segmentation models. We find that convex combinations of learned queries are naturally high-quality object queries for the corresponding models. Based on this observation, we design a pipeline to learn dynamic convex combinations of the basic queries, adapting object queries according to the high level semanties of the input images. This approach consistently improves the performance of a wide range of DETR-basee based models on object detection and segmentation tasks. The gain of our model is agnostic to the different designs of the Transformer decoders and different types object queries. We believe this approach opens the door to designing dynamic queries and creates new perspective for Transformer-hase models.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Zhaowei Cai and Nuno Vasconcelos, Cascade r-cmn: High quality object detection and instance seg- mentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. pp. 1-1, 2019. ISSN 1939-3539. doi: 10.1 10000//am.22119929555555 tpami , 2019.2956515
URL http: //t doi. 10..//11/1/1////////...111111111
Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, In ECCV, 2020.
Qiang Chen, Xiaokang Chen, Gang Zeng, and Jingdong Wang. Group detr: Fast training conver- gence with decoupled one-to-many label assignment. arXiv preprint arXiv:2207. 13085, 2022.
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kemels.. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11030-11039, 2020.
Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexan- der Schwing. Mask2former for video instance segmentation. arXiv preprint arXiv:21 10764, 2021a.
Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexan- der G. Schwing. Mask2former for video instance segmentation. CoRR, abs/2112.10764 2021b. /ttp//bb///111111.1./ 10764.
URL https::/aaxiv..ttt
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Yiming Cui, Linjie Yang, and Ding Liu. Dynamic proposals for efficient object detection. arXiv preprint arXiv:2207.00 220.005220 2022.
Bin Dong, Fangao Zeng, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Solq: Segmenting objects 2021.
by learning queries. Advances in Neural Information Processing Systems,
Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 6578. 2019.
Yuxin Fang, Shusheng Yang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Instances queries. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 6910-6919, October 2021.
Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Fast convergence of detr with spatially modulated co-attention, 2021.
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey, IEEE Transactions Puttern Analysis and Machine Intelligence, 2021.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cmn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Jan Hosang, Rodrigo Benenson, and Bernt Schiele. Learning non-maximum suppression. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 4507-4515, 2017.
Jie Hu, Liujuan Cao, Yao Lu, ShengChuan Zhang, Yan Wang, Ke Li, Feiyue Huang, Ling Shao, and Rongrong Ji. Istr: End-to-end instance segmentation with transformers. arXiv preprint 10Xiv:2/05.00637 2021.
Sukjun Hwang, Miran Heo, Scoung Wug Oh, and Scon Joo Kim. Video instance segmentation using inter- frame- communication transformers. Advances in Neural Information Processing Systems, 34:13352-13363 2021.
10

---

Under review as a conference paper at ICLR 2023
Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. arXiv preprint orXiv:2207.33080 2022.
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmen- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9404-9413, 2019.
Changlin Li, Guangrun Wang, Bing Wang. Xiaodan Liang, Zhihui Li, and Xiaojun Chang. Dynamic slimmable network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8607-8617, 2021.
Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 13619-13627, 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017a.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017b.
Shilong Liu, Feng Li, Hao Zhang, Xiao Yang. Xianbiao Qi. Hang Su, Jun Zhu, and Lei Zhang. DAB-DETR: Dynamic anchor boxes are better queries for DETR. In International Confer- ence on Learning Representations, 2022. URL https:ttpp/cccc..t. httts:://pperrvvtte oMI9P Job99J1
net forum?id-
Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jing- dong Wang. Conditional detr for fast training convergence. In Proceedings of the IEEE Interna- lional Conference on Computer Vision (ICCV), 2021.
Alexander Neubeck and Luc Van Gool. Efficient non-maximum suppression. In 18th International Conference on Pattern Recognition (ICPR'06) volume pp. 850-855. IEEE, 2006.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Byungseok Roh, Joono Shin, Wuhyun Shin, and Sachoon Kim. Sparse detr: Efficient end-to- end object detection with learnable sparsity. arXiv preprint 1:X1v:2111.14330. 2021.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. FCOS: Fully convolutional one-stage object detection. In Proc. Int. Conf. Computer Vision (ICCV), 2019.
Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In Proc. Eur. Conf. Computer Vision (ECCV), 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Max-deeplab: End- to-end panoptic segmentation with mask transformers. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 5463-5474, 2021a.

---

Under review as a conference paper at ICLR 2023
Yingming Wang, Xiangyu Zhang. Tong Yang. and Jian Sun. Anchor detr: Query design for transformer- based object detection. arXiv preprint arXiv:2109.07007, 2021b.
Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, and Gao Huang, Not all images are worth 16x 16 words: Dynamic transformers for efficient image recognition. Advances in Neural Information Processing Systems, 34:11960-11 1960-19973 2021c.
Junfeng Wu, Yi Jiang, Wenqing Zhang, Xiang Bai, and Song Bai. Seqformer: frustratingly simple model for video instance segmentation. arXiv preprint arXiv: 12.0:211..00825, 2021.
Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun. Upsnet: A unified panoptic segmentation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 88188882, 2019.
Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5188-5197, 2019.
Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers, 2019.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks, 2018.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung- Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection, 2022.
Hongkai Zhang, Hong Chang, Bingpeng Ma, Naiyan Wang, and Xilin Chen. Dynamic r-cnm: To- wards high quality object detection via dynamic training. In European conference on computer vision, pp. 260-275. Springer, 2020.
Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. Advances in Neural Information Processing Systems, 34: 34:10326-1033 10338, 2021.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: De- formable transformers for end-to-end object detection. In International Conference on Learning Representations, 2021. URL https: //opp/eeeeeew....////e net weeerrummiiggggghoee 6ke.
12


Reviews:
[{"review_id": "oNWnW8crfPb", "reviewer": "Reviewer_ykWm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an interesting approach to improving DETR-based models across multiple tasks. The proposed approach mainly consists of the following three steps: \n- learn image-adaptive dynamic coefficients for each of the original grouped learned queries,\n- perform convex combination, i.e., aggregate, each group of learned queries according to coefficients, thus producing a small set of clustered queries, and\n- apply Hungarian matching over the clustered queries and the original queries.\n\nLast, the proposed approach also achieves relatively WEAK improvements across multiple benchmarks.\nIn general, the proposed approach seems to be capable of\n\n- modeling the co-occurring relations between different objects, thus I guess the proposed approach\nmainly benefits from explicitly modeling the interactions between different objects. However, the authors still use the original single-object distribution to\nact as the training signal instead of constructing a set of multi-object distributions, which makes the proposed approach less reasonable.\n\n- increasing more positive samples during matching. If my understanding is correct,\nanother key is that **the proposed approach also benefits from adding more positive samples as each matched modulated query essentially corresponds to r=4 original queries, thus bringing 4x more positive samples**, which is also verified in the recent approaches[1][2].\n\n[1] DETRs with Hybrid Matching, arXiv:2207.13080\n\n[2] Group-DETR: Fast DETR Training with Group-Wise One-to-Many Assignment, arXiv:2207.13085", "summary_of_the_review": "The authors' efforts on pushing the frontier of object detection and segmentation performance along the DETR path are highly encouraging.\nThe overall idea is interesting and the results are encouraging but relatively WEAK.\nPlease carefully address the above-listed weaknesses. I will increase the ratings if the authors could well address these concerns, especially to verify the proposed method over strong backbones under longer training schedules following Mask2Former and SeqFormer.", "strength_and_weaknesses": "> Strengths\n\n✅ The presented idea is simple and easy to follow.\n\n✅ The overall writing is satisfying.\n\n✅ The proposed method can be used to improve performance across multiple detection and segmentation tasks.\n\n> Weaknesses\n\n❎ According to Table 2, the proposed method only slightly (+0.8) improves Deformable-DETR but improves DAB-Deformable-DETR significantly (+1.6), which is very STRANGE.\nThe authors explain why. Besides, the authors fail to compare their approach with a very important baseline, i.e., DN-DETR[1]. In fact, according to\nthe official implementation[2] of DN-DETR, DN-DAB-Deformable-DETR already achieves 49.5 while the proposed approach achieves 49.7 based on DAB-Deformable-DETR.\nAlthough the proposed approach seems relatively different from DN-DETR/Group-DETR/HDETR, the authors are still encouraged to discuss the differences between the proposed approach with DN-DETR,\nGroup-DETR[3], and HDETR[4], which already verify that the key insight is to increase more positive samples during matching. If my understanding is correct,\nthe key is that **the proposed approach also benefits from adding more positive samples as each matched modulated query essentially corresponds to r=4 original queries, thus bringing 4x more positive samples**.\n\n[1] DN-DETR: Accelerate detr training by introducing query denoising, CVPR2022\n\n[2] https://github.com/IDEA-Research/DN-DETR\n\n[3] Group-DETR: Fast DETR Training with Group-Wise One-to-Many Assignment, arXiv:2207.13085\n\n[4] DETRs with Hybrid Matching, arXiv:2207.13080\n\n❎ The experimental results are slightly WEAK and far from convincing as the authors only report improvements over the VERY weak results of Mask2Former and SeqFormer.\nFor example, the Mask2Former reports mAP=43.7/PQ=51.9 with R50 as the backbone while the authors only conduct experiments with a baseline of mAP=42.4/PQ=50.4 under a\nshort learning schedule. According to my experience, the proposed approach might become less useful under a longer training schedule. In summary, the experiments are WEAK and the authors should include more comparisons with results under the original learning schedule of Mask2Former or\nSeqFormer. Besides, the authors should also report the comparison results with stronger backbones such as Swin-L to verify whether the proposed method generalizes well. \n\n❎ Figure 1 is misleading to some degree considering the overall GFLOPs of the transformer decoder are much smaller compared to the transformer encoder\nand backbone. In fact, the GFLOPs of the transformer encoder might be more than 10x larger if we take Deformable-DETR as an example. Therefore, the proposed\napproach only brings VERY slight efficiency improvements.\n\n❎ Some important details are missing:\n\n- The authors should justify how to divide the basic queries into multiple groups. Especially, how to divide the basic queries seems non-trivial for a two-stage approach that selects the most confident queries to initialize the basic queries.\n\n- **The authors are required to clarify how many queries are used during training indeed to avoid misunderstanding.**   As mentioned by the other reviewers, the authors seem to try to hide some very important details.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "e8NKM6V8MPm", "reviewer": "Reviewer_tEAE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a learnable convex combination of the learned object queries. High-level semantics are used to generate dynamic coefficients for query combination. Experiments are performed on a wide range of detr-based methods. Consistent improvements are observed on coco, cityscapes and ytvis.", "summary_of_the_review": "Exploring the queries properties is meaningful; the experiment improvement shown on various datasets; the analysis can be further improved (see weakness).", "strength_and_weaknesses": "strength:\n1. extensive experiments on various of tasks, showing effect of the proposed method.\n2. the idea of convex combination of object queries is interesting, and the paper is with good motivation.\n3. paper writing is good with clear structure. \n\n\nweakness:\n\n1. When dividing n queries into m groups, is the dividing process fully random? Is there better grouping strategy to discuss?\n2. What kind of objects will the modulated queries correspond to during training and testing? What kind of objects are being improved? Is there visualization on the dynamic attention weights to help explain the improvement? Is there explanation on why non-convex combination is better than convex one?\n3. The improvement in Table 3 and 4 is very limited.\n4. why mapping one query to one group is necessary? can one query correspond to multiple modulated queries?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YfeelSdyrWS", "reviewer": "Reviewer_LvX2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a general design improvement over models from the DETR family. Specifically, it introduces an additional set of *modulated queries* which are convex combinations of the regular queries. The weights used for the combination are predicted by a small MLP operating on the pooled representation of the image.\nThen:\n- At train time, both regular queries and modulated queries are trained, being fed independently to the decoder, and with a separate hungarian matching\n- At test time, only the modulated queries are used.\n\nThe paper demonstrates the applicability of the method on a wide range of DETR variants as well as tasks (detection, instance segmentation, panoptic segmentation)", "summary_of_the_review": "Without the result of the ablation I mentioned, it is not possible for me to conclude whether the reported improvements come from previously discovered phenomenon (ie using several groups of queries help performance) or from the claimed contribution of this paper.\nAs such, I recommend rejection for now, but depending on the result of the ablation I'm willing to increase my score.\n\n\nPOST REBUTTAL:\nAuthors have provided an ablation that better demonstrate their points. I'm increasing my score 5->8.", "strength_and_weaknesses": "Overall, the paper is clearly motivated, and shows improvements for a variety of models and tasks, suggesting the general applicability of the method.\n\n## Non-modulated ablation\n\nTo me, the main weakness of the paper is that the ablations don't prove the usefulness of the *modulation* in itself.\nIn my opinion, here is the ablation that would demonstrate it convincingly:\nInstead of adding $n/r$ *modulated* queries to the model (as is currently done), one could add the same number of *basic* queries. The rest of the design would be the same: this additional set of queries would be fed separately to the decoder and a separate matching would be computed for it.\n\nThe reason why this ablation is important is because it is known (see [1]) that adding extra groups of queries at training time with a separate matching helps with performance. Thus, the ablation above is required to demonstrate that the gains indeed come from the modulation component as claimed, and not from the same effect uncovered by [1].\n\n\n\n[1] \"Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment\", Chen et al", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DxMA5y8kuM", "reviewer": "Reviewer_QkjX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an attentioinal plug-and-play module, namely, Dynamic Query (DQ), to better generate object queries for set-based transformer detector and segmenter. The proposed method can consistently help improve different kinds of frameworks for both object detection, instance segmentation and panoptic segmentation.", "summary_of_the_review": "Most of my concerns lie in the novelty issue and also the non-impressive performance. So I am inclined to reject this paper.", "strength_and_weaknesses": "### Strength\n\n**1. The idea is simple and sweet, which should be easy to follow.** The proposed method is essentially an attentional plug-in, which should be able to be implemented in just a few lines of code. Such simplicity will attract people to follow.\n\n**2. The experiments are extensive and comprehensive.** The authors evaluate the proposed component into about 10 different frameworks for set-based object detection, instance and panoptic segmentation. \n\n### Weaknesses\n\n**1. The novelty is limited.** The proposed method is too similar to other attentional modules proposed in previous works [1, 2, 3]. The group attention design seems to be related to ResNeSt [4] but it is not discussed in the paper. Although these works did not evaluate their performance on object detection and instance segmentation, the overall structures between these modules and the one that this paper proposed are pretty similar.\n\n**2. Though the improvement is consistent for different frameworks and tasks, the relative gains are not very strong.** For most of the baselines, the proposed methods can only achieve just about 1% gain on a relative small backbone ResNet-50. As the proposed method introduces global pooling into its structure, it might be easy to improve a relatively small backbone since it is with a smaller receptive field. I suspect whether the proposed method still works well on large backbone models like Swin-B or Swin-L.\n\n**3. Some of the baseline results do not matched with their original paper.** I roughly checked the original Mask2former paper but the performance reported in this paper is much lower than the one reported in the original Mask2former paper. For example, for panoptic segmentation, Mask2former reported 51.9 but in this paper it's 50.4, and the AP for instance segmentation reported in the original paper is 43.7 but here what reported is 42.4.\n\nMeanwhile, there are some missing references about panoptic segmentation that should be included in this paper [5, 6].\n\n### Reference\n\n[1] Chen, Yunpeng, et al. \"A^ 2-nets: Double attention networks.\" NeurIPS 2018.\n\n[2] Cao, Yue, et al. \"Gcnet: Non-local networks meet squeeze-excitation networks and beyond.\" T-PAMI 2020\n\n[3] Yinpeng Chen, et al. Dynamic convolution: Attention over convolution kernels. CVPR 2020.\n\n[4] Zhang, Hang, et al. \"Resnest: Split-attention networks.\" CVPR workshop 2022.\n\n[5] Zhang, Wenwei, et al. \"K-net: Towards unified image segmentation.\" Advances in Neural Information Processing Systems 34 (2021): 10326-10338.\n\n[6] Wang, Huiyu, et al. \"Max-deeplab: End-to-end panoptic segmentation with mask transformers.\" CVPR 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject