QFree-Det: Query-Free Detector with Transformer and Sequential Matching

Authors: Guiping Cao, Xiangyuan Lan, Wenjian Huang, Jianguo Zhang, Dongmei Jiang, Yaowei Wang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: free-object prediction, query-free, detecting ambiguity, location-deduplication decoder

Abstract:
Transformer-based detectors, such as DETR and DINO, often struggle with a specific limitation: they can detect only a fixed number of objects based on the predefined number of queries set. This limitation leads to missed detections when the scene exceeds the model’s capacity and increases false positives when the scene contains fewer objects. In addition, existing approaches often combine one-to-one and one-to-many matching label assignment methods in the decoder for accelerating the model training and convergence. However, this operation can introduce a new detecting ambiguity issue, which is often overlooked by those methods. To address these challenges, we propose QFree-Det, a novel query-free detector capable of dynamically detecting a variable number of objects across different input images. In particular, we present an Adaptive Free Query Selection (AFQS) algorithm to dynamically select queries from the encoder tokens, which resolves the issue of fixed capacity. Then, we propose a sequential matching method that decouples the one-to-one and one-to-many processes into separating sequential steps, effectively addressing the issue of detecting ambiguity. To achieve the sequential matching, we design a new Location-Deduplication Decoder (LDD) by rethinking the role of cross-attention (CA) and self-attention (SA) within the decoder. LDD first regresses the location of multiple boxes with CA in a one-to-many manner and then performs object classification to recognize and eliminate duplicate boxes with SA in a one-to-one manner. Finally, to improve the detection ability on small objects, we design a unified PoCoo loss that leverages prior knowledge of box size to encourage the model to pay more attention to small objects. Extensive experiments on COCO2017 and WiderPerson datasets demonstrate the effectiveness of our QFreeDet. For instance, QFree-Det achieves consistent and remarkable improvements over DINO across five different backbones. Notably, QFree-Det obtains a new state-of-the-art of 54.4% AP and 38.8% APs on val2017 of COCO with the backbone of VMamba-T under 1× training schedule (12 epochs), higher than DINO-VMamba-T by +0.9% AP and +2.2% APs. The source codes will be released upon acceptance.

Decision: reject
Reviewer scores: 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730474076823

thread_type:
official_review

<thread_object>
creator: reviewer_eexb

<content>
summary:
Transformer-based detectors like DETR and DINO have a fixed capacity for detecting objects, leading to missed detections and false positives. To overcome these limitations, QFree-Det is proposed as a dynamic, query-free detector that adapts to varying object counts in input images. It features an Adaptive Free Query Selection (AFQS) algorithm for flexible query selection and a Location-Deduplication Decoder (LDD) that separates one-to-one and one-to-many detection processes to reduce ambiguities. Additionally, a unified PoCoo loss enhances small object detection.

strengths:
1. The idea of producing a flexible number of queries to predict a flexible number of detections, is interesting.
2. The paper is easy to follow, and the presentation is good.
3. The performance of the proposed QFree-Det compared to SOTAs looks good.

weaknesses:
1. The contribution of AFQS is not strong enough. The main idea of the proposed Adaptive Free Query Selection (AFQS) algorithm: The fixed content query (CQ) is replaced by the proposed flexible self-adaptive decoder query (SADQ), where SADQ is obtained by sorting the classification scores of all N encoder tokens and selecting the scores above a certain threshold as M SADQ. In short, the AFQS algorithm is a simple strategy to reduce N queries into M queries by sorting the classification scores. I don't think it is enough to be a main contribution.
2. The novelty of sequential matching is incremental, where sequential matching combines Box Locating Part (One-to-many matching), and Deduplication Part (One-to-one matching). One-to-many matching and One-to-one matching are the existing methods.
3. The framework of Box Locating Part (One-to-many matching) and Deduplication Part (One-to-one matching), looks complex. Deduplication Part (One-to-one matching) can be replaced by the parameter-free NMS. The Deduplication Part (One-to-one matching) looks more complex than NMS.
4. PoCoo loss is useful for small object detection. However, the connection between PoCoo loss and the motivation of designing a query-free detector, is very weak.
5. Dynamic query selection and the dual-step decoding process may require more computational resources, potentially impacting processing speed in real-time detection applications.
6. According to Table 5, free-query with AFQS only is worse than fixed-query.

questions:
see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730613008028

thread_type:
official_review

<thread_object>
creator: reviewer_rmnn

<content>
summary:
This paper propose QFree-Det, a query-free detector that select dynamic queries from encoders and decouples one-to-one and one-to-many matching in sequential way. It achieves remarkable improvements compared to the DINO baseline.

strengths:
1. The paper analyze the ambiguity problem overlooked in current one-to-many detectors, and propose a new sequential matching method to tackle this problem, which is not explored before.
2. The paper is well-written and the experiments are comprehensive.

weaknesses:
The dynamic query selection mechanism is similar to the distinct query selection process in DQS[1]. And the sequential matching process is similar to the hybrid layers in H-DETR[2] and MS-DETR[3]. It is better to add citation and discussion with them.

questions:
See the weakness part
</content>
</thread_object>

</thread>
<thread>
creation date:
1730819499081

thread_type:
official_review

<thread_object>
creator: reviewer_tzb2

<content>
summary:
This paper proposes QFree-Det, a novel query-free detector based on transformers, aiming to address the limitations of fixed query numbers and detection ambiguity in existing transformer-based detectors. The main contributions include the Adaptive Free Query Selection (AFQS) algorithm and the Location-Deduplication Decoder (LDD) framework.

strengths:
-The AFQS algorithm and LDD framework provide a novel solution to the challenges of fixed query limitations and detection ambiguities that are common in traditional transformer-based detectors like DETR and DINO.

-The model achieves state-of-the-art results on the COCO2017 and WiderPerson datasets, particularly excelling in small object detection. This demonstrates the effectiveness of the proposed method over existing leading techniques.

-By enabling  dynamic query selection, QFree-Det effectively adapts to varying scene complexities, providing a robust framework for diverse detection scenarios.

-The introduction of the PoCoo loss function significantly enhances the detection capabilities for small objects, addressing a common challenge in object detection.

weaknesses:
-The authors propose AFQS algorithms and LDD frameworks, but lack in-depth theoretical analysis.  There is no detailed explanation of why removing location queries (PQ) does not significantly affect model performance, nor is there a theoretical proof of how LDD can effectively solve the detection ambiguity problem. 

-Insufficient comparison with state-of-the-art methods, In Table 3, the authors mainly compare with older methods such as DINO, but lack detailed comparisons with more recently published methods (e.g. DAC-DETR, Co-DETR, etc.). This makes it difficult to assess the position of QFree-Det in the current research frontier. 


-The author mentions that AFQS can reduce redundant queries, but does not provide a detailed computational complexity analysis or efficiency comparison with other methods. Lack of specific inference time and memory usage data.

-Although the improvement of the PoCoo loss function for small object detection is shown in Table 7, there is no in-depth analysis of why this loss function can be particularly effective in improving small object detection performance.

questions:
-How does QFree-Det perform on other challenging datasets beyond COCO and WiderPerson? Are there specific scenarios where QFree-Det might underperform?

-Can you provide a more detailed analysis of the trade-offs between performance and computational cost in QFree-Det?

-How does the AFQS algorithm scale with increasing image complexity and object density?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730998731082

thread_type:
official_review

<thread_object>
creator: reviewer_pjik

<content>
summary:
The paper introduces a novel query-free detector, QFree-Det, aimed at addressing the limitation of a fixed number of object queries in Transformer-based detectors such as DETR and DINO. The authors propose an Adaptive Free Query Selection (AFQS) algorithm and a sequential matching method, which significantly enhance the model's capability to dynamically detect a variable number of objects across different input images. Additionally, a new loss function, PoCoo, is designed to improve detection capabilities. The experimental results demonstrate that QFree-Det achieves remarkable performance on multiple datasets and various backbone networks.

strengths:
> 1. Motivation:

The paper introduces the QFree-Det model, addressing the variable number of instances in different images, which is a pertinent issue in object detection. The proposal for a dynamic query selection approach is innovative and could significantly enhance the flexibility and accuracy of detection models.

> 2. Performance Improvement:

The paper demonstrates some performance improvements on the WiderPerson dataset, particularly in detecting small objects, which is a challenging aspect of computer vision.

weaknesses:
> Comparison with DETA and DDQ-DETR:

The discussions in Section 3.1.3 regarding the role of Self-Attention (SA) bear similarities to the disscussions taken in "NMS Strikes Back" (DETA) [1] and "Dense Distinct Query for End-to-End Object Detection" (DDQ-DETR) [2]. Both papers explore the effectiveness of traditional non-maximum suppression (NMS) and the role of SA integrated with transformer-based detectors, which is a relevant area of comparison for QFree-Det.

[1] https://arxiv.org/pdf/2212.06137

[2] Dense Distinct Query for End-to-End Object Detection (CVPR2023)

> Comparative Experiments in Table 3:

While the paper provides comparisons with models like DINO, it lacks a comprehensive set of comparative experiments with other state-of-the-art Transformer detectors, such as Stable DINO and CO-DINO. Including these comparisons would provide a more holistic view of QFree-Det's performance relative to the current landscape of object detection models.

> Performance Comparison with Fixed Detectors:

The paper compares the performance of QFree-Det (AP 50.5) with DINO (AP 49.0). However, it is noted that improved versions of DINO, utilizing focal loss, have achieved an AP of 50.0, as demonstrated in the MMDETECTION repository. This raises the question of whether other state-of-the-art fixed detectors, when combined with loss functions like PoCoo or IA-BCE, could potentially outperform QFree-Det. Further experimentation and comparison with these models would be beneficial to assert the superiority or uniqueness of QFree-Det's approach.

questions:
> The number of decoder layers.

It seems decoder only have one layer of CA and 6 layers of SA?

> Speed of inference.

How fast can the QFree-Det be for some sparse scenario, i.e., only a instance in a image.

> Performances in line 476

Is the same experiments (47.1) as that of line 248?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732090811667

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Responses About: inference speed tests and analysis

comment:
Dear Reviewers,

We sincerely thank you for your thorough and valuable reviews. We believe your suggestions have improved our paper. In this section, we present the General Responses about the *inference speed tests and analysis*. 

In **Sec.A.5** and **Sec.B.2** of the Appendix in the main paper, we have conducted experiments on COCO and WiderPerson, along with analyses to explore the relationship between the adaptive number of queries, computational complexity, and model accuracy. The results highlight the advantages of our method in both aspects.

Following suggestions, we conduct additional tests on the inference speed of QFree-Det. For a fair comparison of our model with the baseline model DINO, the test was applied on the same codebase (official published code of DINO), backbone model (ResNet50), input image size (1280 $\times$ 800), GPU device (RTX 3090), PyTorch lib (pytorch 2.2), and CUDA version (Driver Version: 535.183.01, CUDA Version: 12.2). We tested the DINO (900 query) model and QFreeDet (query varies from *10 to 1800*) model's inference speed, and the results are shown as below: 


| Model | DINO | QFree-Det | QFree-Det | QFree-Det | QFree-Det | QFree-Det |
| --- | --- | --- | --- | --- | --- | --- |
| Query Number |  900 | 1800 | 900 | 500 | 100 | 10 |
| Backbone (ms) | 15.2 | 15.2 | 15.2 | 15.2 | 15.2 | 15.2 | 
| Encoder (ms) |  30.6 | 30.6 | 30.6 | 30.6 | 30.6 | 30.6 |
| Query Selection (ms) | 7.3 | 7.8 | 7.5 | 7.4 | 7.4 | 7.4 |
| Decoder (ms) | 14.1 | 11.0 | 9.2 | 8.9 | 8.9 | 8.8 |
| Inference Time (ms) | 67.2 | 64.6 | 62.5 | 62.1 | 62.1 | 62.0 |
| FPS (frame/s) | 14.9 | 15.5 | 16.0 | 16.1 | 16.1 | 16.1 |

The inference time of both models is composed of four components: *backbone time, encoder time, query selection time, and decoder time*. Since the backbone and encoder components are identical in both models, their inference speeds are also the same. The inference time of query selection process in both models is similar, with most of the duration spent on the model classification head in predicting scores (about 7ms) among all encoder tokens for subsequent selection. For our AFQS algorithm, it then retrieves queries based on these scores within **1 ms**, which is faster in query selection process of transforming the fixed-query into a free-query for the DETR detector.

Additionally, we observed that under the same query conditions, such as using *900 queries*, our LDD framework significantly enhances the inference speed of decoder, achieving a **+34.8\%** improvement compared to DINO. This result clearly validates the effectiveness of our designed novel LDD decoder framework. When the number of queries is further reduced, the decoder's inference speed remains stable, primarily due to the parallel computation performed by CA and SA in the decoder. 

It is noted that to improve the model's inference speed is not the primary objective of our method. The above table indicates that the backbone and encoder models account for **73\%** of the inference time. This observation provides insights for further optimizing these components to accelerate the inference speed of DETR models. Notably, our efficient LDD decoder framework has successfully increased the inference speed of the model's decoder by **34.8\%**. Thus, this framework can be integrated with other model architectures, such as the YOLO series, not only to further enhance overall inference speed but also achieve the detection of a free number of objects. 

Additionally, this query-free model, which adapts the number of queries based on the image itself, *holds significant potential for open-ended object detection task*. This task directly generates object names through a generative **large language model**, where the query count greatly influences the model's complexity. The *adaption characteristic* of our model enables reduced queries to feed into generative large language model, thereby decreasing the workload for subsequent category generation processing, which highlights its potential applications and importance for the future research in vision community.

Kind regards,

Authors of Paper ID: 4457
</content>
</thread_object>

</thread>
<thread>
creation date:
1732534715727

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision

comment:
Dear reviewers,

We sincerely appreciate your thorough and detailed reviews. We believe your suggestions have improved our paper. In the following, we present the changes made in the revised version, which are highlighted in blue. We have also provided individual responses to each of your concerns under respective reviews. If you have any additional comments or suggestions, we would be grateful for the opportunity to clarify further.

- We have updated the query type in Table 5 from ''free'' to ''free-conditioned query''.
- We have added comparisons of Stable-Dino, DDQ-DETR, DETA, MS-DETR in Table 3 to enhance the comparison with SOTA models. Additionally, we included experiments on CrowdHuman in Table 19 of the Appendix to further validate the effectiveness of our method.
 - We have added the ratio of query numbers to object numbers in Table 9 (row 5) of the Appendix to further illustrate the relation between the number of objects and queries, highlighting the advantages of the query-free method.
 - We have added a new Fig.6 (visualization of row 2 and row 4 of Table 9) to illustrate the trend between the number of queries selected by QFree-Det model and the number of objects.
 - We have added a new section titled ''*Inference Speed Tests and Analysis*'' (Sec.D) in the Appendix, which includes the ''Speed Tests'' and the ''Potential Effectiveness on Downstream Open-Ended Detection Tasks'' to show the potential applications of the query-free model in downstream tasks.
 - We have added a new ``Discussion`` section (Sec.E) in the Appendix, which primarily explores the differences and effectiveness of our method compared to existing detection methods, along with the advantages and potential applications of our QFree-Det model. This section includes the following topics: 
 
 (1) Sec.E.1: ''The Deduplication Roles of Self-Attention (SA) and NMS'' and ''Deduplication Part (DP) vs. NMS'';
 
 (2) Sec.E.2:  ''The Differences of Sequential Matching with Existing Methods'' and ''Differences between H-DETR, MS-DETR and QFree-Det on Sequential Matching'';
 
 (3) Sec.E.3: ''The impact of removing PQ'' and ''LDD framework for addressing Matching Ambiguity'';
 
 (4) Sec.E.4: ''The Computational Complexity Analysis'';
 
 (5) Sec.E.5: ''The Contributions of AFQS Algorithm'' ;
 
 (6) Sec.E.6: ''Advantages of the QFree-Det Model'';


Kind regards,

Authors of Paper ID: 4457
</content>
</thread_object>

</thread>
<thread>
creation date:
1732759457999

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response

comment:
Dear reviewers and ACs,

We sincerely appreciate your valuable time, diligent efforts, and thorough reviews.

We propose QFree-Det, a novel query-free model that achieves ''*free-object predictions*'' through the AFQS algorithm and a more efficient LDD architecture. This model effectively addresses the limitations of ''*fixed-query*'' and ''*detection ambiguity*'', achieving high-rate of effective query utilization, improved cost-efficiency, and enhanced performance. Additionally, it strengthens the detector's adaptive detection capabilities, accelerates the inference speed of the decoder, and shows promising potential for downstream open-ended detection tasks.

We are delighted to note that reviewers find that: 
* Cool motivation (``Pjik``); 
* Innovative/novel/interesting solution/idea for achieving free-query detection (``Pjik``, ``TZb2``, ``eExB``); Novel and the first method for addressing the ambiguity problem (``rMNn``);
* Demonstrated improved performance and effectiveness (``Pjik``, ``TZb2``), robust framework for diverse detection scenarios (``TZb2``), enhancing for small objects detection (``Pjik``, ``TZb2``), the comprehensive experiments (``rMNn``), good performance compared to SOTA (``TZb2``, ``eExB``); 
* Well-written/good presentation/easy to follow in writing style (``rMNn``, ``eExB``).

We deeply appreciate your constructive feedback on our manuscript. Your insights are highly valued, and we believe your suggestions have improved our paper. We have carefully revised the manuscript and addressed each of your concerns with additional tests, experiments, and detailed analyses. We hope our individual responses (accompanying the revised PDF) have carefully addressed each of your concerns. We are respectfully request that the revieweres to kindly reconsider the ratings based on our responses and the revised manuscript. If you have any additional comments or suggestions during the discussion period, please feel free to reach out, and we would greatly appreciate to provide further clarification.  

Thank you once again for your time and consideration.

Best Regards!

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) This paper proposes  a query-free detector based on transformers to address the limitations of fixed query numbers and detection ambiguity in existing transformer-based detectors. (b) The motivation is clear and the experimental results are convincing. (c) The main weakness lies in the technique contribution and the unclear advantage over fixed query based detector. (d) After carefully reading the paper and reviewers' comments, the AC agrees with the reviewers that the advantage over fixed query based detector is unclear.