Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose three different methods to improve the framework of DETR: a contrastive way for centre-surrounding, twice forward prediction for refining results and a mixed query selection. The experiments demonstrate the necessity of these methods and significant improvement in classical benchmarks compared with other methods.
strength_and_weaknesses: Strength:
1. This paper investigates the DETR framework very well and gives a comprehensive survey of related works.
2. The performance is impressive. The advantages are mainly faster convergence, better performance and the ability to have smaller models.
3. The motivations of three different components are reasonable. And ablation experiments show the expected target has been achieved.

Weakness:
Although the model is only 1/10 of the swinT size, there is nothing result about the inference speed. So I wonder if the so-called small models don't actually have any advantage in terms of computational power.
clarity,_quality,_novelty_and_reproducibility: Overall I thought the quality of the article was good and clear. It has good originality and a significant solution to a hot topic. 
summary_of_the_review: Despite the lack of sufficient discussion on the speed of this article, I am still inclined to accept this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an end-to-end Transformer detector named DINO that archives impressive performance on the COCO benchmark. The proposed DINO model improves the training efficiency and the detection performance by utilizing (i) contrastive denoising training, (ii) look forward twice, and (iii) mixed query selection strategies. The experimental results show that  DINO outperforms all previous ResNet-50-based models on COCO val2017 in the 12-epoch and the 36-epoch settings using multi-scale features. Further, training DINO with a stronger backbone on a larger dataset leads to an impressive result of 63.3 AP on COCO 2017 test-dev which is comparable with even CNN-based state-of-the-art methods on mainstream detection task. 
strength_and_weaknesses: Strengths:

1.	Impressive results. Although previous DETR-like models were having end-to-end object detection pipelines, their results are inferior to classical CNN-based object detectors.  Unlike the previous DETR-like models, the paper reports state-of-the-art object detection results on the COCO benchmark. 

2.	The model requires relatively less pre-training data and a smaller model size compared to many state-of-the-art methods in the benchmark. 

3.	The proposed changes are simple, but effective in improving accuracy. 

4.	The paper reports results on a large Swin-L backbone with large-scale pre-training. 

5.	Thorough ablation experiments. 


Weakness:

1.    Incremental novelty:  The key idea behind the proposed contributions such as contrastive denoising training, and look forward twice are well-known in the literature. I will not consider this as a major problem since the authors effectively adapted those ideas for improving performance on DETR framework. 

2.   Please explain why the improvement on small objects is very high (+7.2% in Tab. 2) compared to medium and large-sized objects. Which of the proposed contribution helps in achieving this?  Why?  

3.   Table 2 shows that the proposed method is faster (FPS 24) than existing DETR-based object detectors, and its speed is even comparable with deformable DETR. But it is not clear to me how the proposed model with many components can achieve a speed, comparable to deformable DETR, and faster than DN-DETR?

4.  In page 8 it is mentioned that “Note that the results of our models with ResNet-50 backbone are higher than those in the first version of our paper due to engineering techniques”. Please clarify which “first version” the authors are referring to”? 

5. The paper utilizes many engineering tricks to achieve state-of-the-art results (Appendix F). 

6.  In tables 2 and 4, what are the results of the proposed model without using additional engineering techniques mentioned in appendix F?

7. Table 4 shows that the proposed method achoves Impressive SOTA results. It uses smaller pre-training, it does not need mask annotation and requires less paramets. Still it will achieve results comparable with SOTA methods. For complete understanding of the model performance, it will be beneficial to introduce a comparison on  the inference time and FLOPS in Table 4.


clarity,_quality,_novelty_and_reproducibility: It is easy to follow the core ideas of the paper and the authors provided detailed explanation of experimental setup for effectively reproducing the results. Further, the authors promised to release their codebase. The paper has some simple and incrimental contributions. But those contributions are highly effective in achieving state-of-the-art performance. 
summary_of_the_review: The paper has some incremental contributions. But those contributions are highly effective in achieving state-of-the-art performance on DETR framework. So, I am in favor of accepting this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an improved DETR-like algorithm (both accuracy and speed) for object detection. Three techniques are proposed in the paper, including contrastive denoising training, look forward twice, and mixed query selection. The idea of denoising anchor boxes in previous work has shown effectiveness in stabilizing training and accelerating convergence. Based on the previous work, the authors introduce the additional hard negative examples to further improve the performance. The experimental results of the 12-epoch setting show that the new contrastive denoising technique leads to a faster convergence. Furthermore, the ablation study shows that this new technique has achieved the gain of about 1% in detection accuracy. The Look Forward Twice technique for iterative box refinement is the extension of the look forward once in previous work Deformable DETR. The backpropagation of the gradient is performed in two consecutive decoder layers instead of one layer. The ablation study shows that this technique boosts the performance by about 0.5%. The Mixed Query Selection technique is a combination of previous works DN-DETR and Deformable DETR, with selected positions anchors and learnable the content queries. The ablation study also shows that this technique boosts the performance by about 0.5%.

Furthermore, the experimental results show that the performance of the model with about 1 billion parameters trained by the new approach outperforms the larger models trained from even more pre-training data. 

strength_and_weaknesses: Strength:

The proposed method is simple yet effective.
The paper is well organized. The illustration provides the great details of the new techniques, which are helpful for the readers to understand.
The work has achieved strong experimental results with the improvement of about 2% over the state-of-the-art baselines.
The paper presents extensive details of the experiments.


Weaknesses

The motivation of the look forward twice scheme isn't very convincing. The paper has shown that the choice of twice is better than once, but is it better than three times and four times? If there is no good theoretical analysis to support this choice, it would be better to have some empirical results to justify it.

The second last sentence in ABSTRACT, “…, DINO significantly reduces its model size and pre-training data size while achieving better results.”  The claim is confusing. To my understanding, with the training techniques of DINO, the model with small size can achieve better performance than the large models. But the approach of DINO itself does not reduce the model size.

In section 3.3, a typo in “They are are expected to predict “no object”.
clarity,_quality,_novelty_and_reproducibility: The quality of the paper is good: it studies the fundamental task, object detection in computer vision, and has achieved the new state-of-the-art results.

The clarity of the paper is good: introduction of the method, experiments and claims are clear enough to read and understand.

The novelty of the paper is *not* significant, since the three new techniques are the extensions or modifications of existing works.

Source code isn't provided but the authors state the code will be made available for reproducibility.

summary_of_the_review: I would recommend a marginally below acceptance. The novelty of this work is not significant as the three techniques are quite incremental without theoretical insights. The experimental results and reported performances are the state-of-the-arts and may have a good impact to the field of objection detection.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes some improvements on the DETR family of Detectors.
Namely, the improvements are as follows:
- Add some negatives in the denoising objective
- Refine the design of the iterative refinement of boxes
- Propose a slightly different way to initialize object queries

With these improvements, the resulting model improves across the board on the Coco dataset, and in particular sets a new SOTA result for the setting with extra-data
strength_and_weaknesses: The paper shows strong results on the Coco benchmark, surpassing current SOTA even with less training data, which is an impressive feat.
I find all the comparisons to be fair with previous approaches.

On the presentation side, one minor weakness is the lack of visualization of any kind. For example, I would have liked to see some head-to-head inference comparison with DN-DETR on the same images, as well as some failure cases.

Finally, although it is not required for acceptance, I think the paper could be strengthen by showing results on datasets beyond Coco, perhaps LVIS or CrowdHuman. 


clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, all the changes are well described, motivated, and empirically demonstrated.
I believe all the technical details provided, including hyper-parameters, are sufficient for reproduction.


Typo:
- "LFT scarifies performance" -> sacrifices 
summary_of_the_review: The paper is technically sound and the results are strong, I recommend acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In order to speed up the training convergence and highly optimize the Transformer-based detector (DETR-like models), a contrastive way for denoising training, a look forward twice scheme for box prediction, and a mixed query selection method for anchor initialization are proposed. The extensive ablation studies demonstrate the effectiveness of proposed methos over current DETR-like models.
strength_and_weaknesses: Strengths:
1) DINO points out the reason why newly developed DETR-like models are inferior to the classical detectors and propose the corresponding solutions, eg., denoising training, query initialization and box prediction to highly optimize DETR-series detectors.
2) The contrastive denoising training helps the model to predict more precise boxes and avoid duplicate outputs of the same target compared with DN-DETR.
3) The look forward twice scheme has the ability to correct the updated parameters with gradients from later layers to overcome the weakness of refining boxes in each decoder layer, while keeping the advantages of fast convergence.
4) Authors report the performance of DETR-like model on large backbone with large-scale pre-training data.

Weaknesses:

Please see the following part of `Unclarity and Questions'.



clarity,_quality,_novelty_and_reproducibility: Unclarity and Questions:
1. The denoising step is actually to add many-to-one matching (many queries are assigned to one ground true bounding box) in the vanilla DETR-like detector training scheme that normally is a one-to-one on bipartite matching. Such a many-to-one matching strategy with contrastive training is commonly used in classical detectors, and NMS is needed in those detectors due to duplicated detections. Why author claim that explicitly introduces negative examples in denoising step for DETR-like models can reduce duplicate predictions, any differences between classical and DETR-like detectors in many-to-one matching and contrastive training?

2. Why making the content queries learnable can force the first decoder layer to focus on the spatial prior? As I though the behind reason of learnable scheme is to align with queries in CDN part. Please provide more explanations.  

3. For a look forward twice technic, why LFO performs better than LFT in layer 0 to 2, while preforms pool in layer 3 to 6? And how the variants of looking forward trice or quartic perform in DETR-like models？

summary_of_the_review: I would tend to accept this paper as it is practical DETR-like model and the proposed modules are supported by empirical experiments.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper received high scores and maintained them after rebuttal which warrant a solid accept. Nonetheless, there were concerns about the incremental novelty, e.g., 'the key idea behind the proposed contributions such as contrastive denoising training, and look forward twice are well-known in the literature' and concerns about lack of strong theoretical analysis and motivation regarding the look forward twice scheme. On a less technical note, there exists DINO ViT model: perhaps it would make sense for authors to find a more unique to this work title. For these reasons, AC recommends to accept the paper as a poster.
justification_for_why_not_higher_score: In general, the paper got solid scores but reviewers listed a fair amount of concerns about the incremental novelty, e.g., 'the key idea behind the proposed contributions such as contrastive denoising training, and look forward twice are well-known in the literature' and concerns about lack of strong theoretical analysis and motivation regarding the look forward twice scheme. For these reasons, the paper is an interesting combination of existing techniques but the level of innovation and novelty could be greatly improved.
justification_for_why_not_lower_score: Reviewers seem to agree that the proposed method is simple yet effective, the paper well organized, and the approach achieved strong experimental results over the state-of-the-art baselines.
Decision: Accept: poster