Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a learnable convex combination of the learned object queries. High-level semantics are used to generate dynamic coefficients for query combination. Experiments are performed on a wide range of detr-based methods. Consistent improvements are observed on coco, cityscapes and ytvis.
strength_and_weaknesses: strength:
1. extensive experiments on various of tasks, showing effect of the proposed method.
2. the idea of convex combination of object queries is interesting, and the paper is with good motivation.
3. paper writing is good with clear structure. 


weakness:

1. When dividing n queries into m groups, is the dividing process fully random? Is there better grouping strategy to discuss?
2. What kind of objects will the modulated queries correspond to during training and testing? What kind of objects are being improved? Is there visualization on the dynamic attention weights to help explain the improvement? Is there explanation on why non-convex combination is better than convex one?
3. The improvement in Table 3 and 4 is very limited.
4. why mapping one query to one group is necessary? can one query correspond to multiple modulated queries?
clarity,_quality,_novelty_and_reproducibility: Good writing and interesting idea.
summary_of_the_review: Exploring the queries properties is meaningful; the experiment improvement shown on various datasets; the analysis can be further improved (see weakness).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a general design improvement over models from the DETR family. Specifically, it introduces an additional set of *modulated queries* which are convex combinations of the regular queries. The weights used for the combination are predicted by a small MLP operating on the pooled representation of the image.
Then:
- At train time, both regular queries and modulated queries are trained, being fed independently to the decoder, and with a separate hungarian matching
- At test time, only the modulated queries are used.

The paper demonstrates the applicability of the method on a wide range of DETR variants as well as tasks (detection, instance segmentation, panoptic segmentation)
strength_and_weaknesses: Overall, the paper is clearly motivated, and shows improvements for a variety of models and tasks, suggesting the general applicability of the method.

## Non-modulated ablation

To me, the main weakness of the paper is that the ablations don't prove the usefulness of the *modulation* in itself.
In my opinion, here is the ablation that would demonstrate it convincingly:
Instead of adding $n/r$ *modulated* queries to the model (as is currently done), one could add the same number of *basic* queries. The rest of the design would be the same: this additional set of queries would be fed separately to the decoder and a separate matching would be computed for it.

The reason why this ablation is important is because it is known (see [1]) that adding extra groups of queries at training time with a separate matching helps with performance. Thus, the ablation above is required to demonstrate that the gains indeed come from the modulation component as claimed, and not from the same effect uncovered by [1].



[1] "Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment", Chen et al
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow. The idea is novel as far as I know. I believe enough details are provided to allow reproduction.


Questions:
## Inference speed

Let's take an example to make sure I understand correctly. Let's say we start from a DeformableDETR model with 300 base queries, and we use r=4 as is done in the experiments. This will create 300/4 = 75 modulated queries. At train time, there are two forwards through the decoder, which naturally incurs a slightly higher cost. However, at inference time, only the modulated queries are used. So effectively the run-time is equivalent to a DeformableDETR with 75 queries plus the cost of computing the modulation coefficient. I would expect it to be faster because of the reduced computation in the decoder (quadratic self attention). However the paper seems to indicate that it's not really the case? Is it because computing the modulation coefficient is costly? Reporting some inference speeds would help here (eg Deformable DETR 300q vs Deformable DETR 75q vs DQ Deformable DETR 300q)

Typos:
- "visualization examples are proviede" (section 4)


summary_of_the_review: Without the result of the ablation I mentioned, it is not possible for me to conclude whether the reported improvements come from previously discovered phenomenon (ie using several groups of queries help performance) or from the claimed contribution of this paper.
As such, I recommend rejection for now, but depending on the result of the ablation I'm willing to increase my score.


POST REBUTTAL:
Authors have provided an ablation that better demonstrate their points. I'm increasing my score 5->8.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an attentioinal plug-and-play module, namely, Dynamic Query (DQ), to better generate object queries for set-based transformer detector and segmenter. The proposed method can consistently help improve different kinds of frameworks for both object detection, instance segmentation and panoptic segmentation.
strength_and_weaknesses: ### Strength

**1. The idea is simple and sweet, which should be easy to follow.** The proposed method is essentially an attentional plug-in, which should be able to be implemented in just a few lines of code. Such simplicity will attract people to follow.

**2. The experiments are extensive and comprehensive.** The authors evaluate the proposed component into about 10 different frameworks for set-based object detection, instance and panoptic segmentation. 

### Weaknesses

**1. The novelty is limited.** The proposed method is too similar to other attentional modules proposed in previous works [1, 2, 3]. The group attention design seems to be related to ResNeSt [4] but it is not discussed in the paper. Although these works did not evaluate their performance on object detection and instance segmentation, the overall structures between these modules and the one that this paper proposed are pretty similar.

**2. Though the improvement is consistent for different frameworks and tasks, the relative gains are not very strong.** For most of the baselines, the proposed methods can only achieve just about 1% gain on a relative small backbone ResNet-50. As the proposed method introduces global pooling into its structure, it might be easy to improve a relatively small backbone since it is with a smaller receptive field. I suspect whether the proposed method still works well on large backbone models like Swin-B or Swin-L.

**3. Some of the baseline results do not matched with their original paper.** I roughly checked the original Mask2former paper but the performance reported in this paper is much lower than the one reported in the original Mask2former paper. For example, for panoptic segmentation, Mask2former reported 51.9 but in this paper it's 50.4, and the AP for instance segmentation reported in the original paper is 43.7 but here what reported is 42.4.

Meanwhile, there are some missing references about panoptic segmentation that should be included in this paper [5, 6].

### Reference

[1] Chen, Yunpeng, et al. "A^ 2-nets: Double attention networks." NeurIPS 2018.

[2] Cao, Yue, et al. "Gcnet: Non-local networks meet squeeze-excitation networks and beyond." T-PAMI 2020

[3] Yinpeng Chen, et al. Dynamic convolution: Attention over convolution kernels. CVPR 2020.

[4] Zhang, Hang, et al. "Resnest: Split-attention networks." CVPR workshop 2022.

[5] Zhang, Wenwei, et al. "K-net: Towards unified image segmentation." Advances in Neural Information Processing Systems 34 (2021): 10326-10338.

[6] Wang, Huiyu, et al. "Max-deeplab: End-to-end panoptic segmentation with mask transformers." CVPR 2021
clarity,_quality,_novelty_and_reproducibility: As discussed above, the illustration of this paper is clear and the overall quality for writing is acceptable. However, the originality of this paper might be a bit limited and its reproducibility cannot be guaranteed since the authors do not promise to release the code publicly in the future.
summary_of_the_review: Most of my concerns lie in the novelty issue and also the non-impressive performance. So I am inclined to reject this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an interesting approach to improving DETR-based models across multiple tasks. The proposed approach mainly consists of the following three steps: 
- learn image-adaptive dynamic coefficients for each of the original grouped learned queries,
- perform convex combination, i.e., aggregate, each group of learned queries according to coefficients, thus producing a small set of clustered queries, and
- apply Hungarian matching over the clustered queries and the original queries.

Last, the proposed approach also achieves relatively WEAK improvements across multiple benchmarks.
In general, the proposed approach seems to be capable of

- modeling the co-occurring relations between different objects, thus I guess the proposed approach
mainly benefits from explicitly modeling the interactions between different objects. However, the authors still use the original single-object distribution to
act as the training signal instead of constructing a set of multi-object distributions, which makes the proposed approach less reasonable.

- increasing more positive samples during matching. If my understanding is correct,
another key is that **the proposed approach also benefits from adding more positive samples as each matched modulated query essentially corresponds to r=4 original queries, thus bringing 4x more positive samples**, which is also verified in the recent approaches[1][2].

[1] DETRs with Hybrid Matching, arXiv:2207.13080

[2] Group-DETR: Fast DETR Training with Group-Wise One-to-Many Assignment, arXiv:2207.13085
strength_and_weaknesses: > Strengths

✅ The presented idea is simple and easy to follow.

✅ The overall writing is satisfying.

✅ The proposed method can be used to improve performance across multiple detection and segmentation tasks.

> Weaknesses

❎ According to Table 2, the proposed method only slightly (+0.8) improves Deformable-DETR but improves DAB-Deformable-DETR significantly (+1.6), which is very STRANGE.
The authors explain why. Besides, the authors fail to compare their approach with a very important baseline, i.e., DN-DETR[1]. In fact, according to
the official implementation[2] of DN-DETR, DN-DAB-Deformable-DETR already achieves 49.5 while the proposed approach achieves 49.7 based on DAB-Deformable-DETR.
Although the proposed approach seems relatively different from DN-DETR/Group-DETR/HDETR, the authors are still encouraged to discuss the differences between the proposed approach with DN-DETR,
Group-DETR[3], and HDETR[4], which already verify that the key insight is to increase more positive samples during matching. If my understanding is correct,
the key is that **the proposed approach also benefits from adding more positive samples as each matched modulated query essentially corresponds to r=4 original queries, thus bringing 4x more positive samples**.

[1] DN-DETR: Accelerate detr training by introducing query denoising, CVPR2022

[2] https://github.com/IDEA-Research/DN-DETR

[3] Group-DETR: Fast DETR Training with Group-Wise One-to-Many Assignment, arXiv:2207.13085

[4] DETRs with Hybrid Matching, arXiv:2207.13080

❎ The experimental results are slightly WEAK and far from convincing as the authors only report improvements over the VERY weak results of Mask2Former and SeqFormer.
For example, the Mask2Former reports mAP=43.7/PQ=51.9 with R50 as the backbone while the authors only conduct experiments with a baseline of mAP=42.4/PQ=50.4 under a
short learning schedule. According to my experience, the proposed approach might become less useful under a longer training schedule. In summary, the experiments are WEAK and the authors should include more comparisons with results under the original learning schedule of Mask2Former or
SeqFormer. Besides, the authors should also report the comparison results with stronger backbones such as Swin-L to verify whether the proposed method generalizes well. 

❎ Figure 1 is misleading to some degree considering the overall GFLOPs of the transformer decoder are much smaller compared to the transformer encoder
and backbone. In fact, the GFLOPs of the transformer encoder might be more than 10x larger if we take Deformable-DETR as an example. Therefore, the proposed
approach only brings VERY slight efficiency improvements.

❎ Some important details are missing:

- The authors should justify how to divide the basic queries into multiple groups. Especially, how to divide the basic queries seems non-trivial for a two-stage approach that selects the most confident queries to initialize the basic queries.

- **The authors are required to clarify how many queries are used during training indeed to avoid misunderstanding.**   As mentioned by the other reviewers, the authors seem to try to hide some very important details.
clarity,_quality,_novelty_and_reproducibility: > Clarity

Good.

> Quality

Good.

> Novelty

Relatively weak considering the recent HDETR and Group-DETR.

> Reproducibility

No code is available.
summary_of_the_review: The authors' efforts on pushing the frontier of object detection and segmentation performance along the DETR path are highly encouraging.
The overall idea is interesting and the results are encouraging but relatively WEAK.
Please carefully address the above-listed weaknesses. I will increase the ratings if the authors could well address these concerns, especially to verify the proposed method over strong backbones under longer training schedules following Mask2Former and SeqFormer.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an idea of learned convex combination queries for DETR architectures. The coefficients of these convex combination queries is conditioned on the individual images. The paper was reviewed by four reviewers and initially received more positive ratings, but after rebuttal the ratings have came down to:

* 1 x accept, good paper
* 3 x marginally below the acceptance threshold

Reviewers have raised a number of concerns with the work in rebuttal: (1) limited improvements [tEAE], (2) limited novelty [QkjX], and (3) lack of clear comparison to other methods with respect to the number of queries used. The last point is very serious, as nearly all reviewers initially believed that the number of queries in the proposed method were significantly lower than in reality. When this came to light, this raised serious concerns of multiple reviewers about the fairness of experiments and validity of results. If the number of queries is larger, then it is not at all clear that the proposed convex query combination is what is driving the improved performance. 

AC has carefully looked at the reviews, rebuttal, discussion and the paper itself, and ultimately agrees with reviewers that this is problematic and, even if not done intentionally, is misleading. Notably, even after revisions this was not clarified in the revised manuscript. No revision mentions that authors "use 1200 basic queries and 300 modulated queries in most experiments", the text still reads that they use "300 queries". As such and in the current form the paper cannot be accepted for publication in ICLR. 

Authors are encouraged to test more apples-to-apples (fairer) settings proposed by the reviewers, as well as to adjust tables (e.g., Table 2 and others) to include the number of queries and kinds used in each case, before resubmission to a future venue. 

justification_for_why_not_higher_score: The paper appears to have miss-characterized the results. The current comparisons are not completely fair and need to be redone in more comparable manner.
Decision: Reject