Redundant Queries in DETR-Based 3D Detection Methods: Unnecessary and Prunable

Authors: Lizhen Xu, Shanmin Pang, WenZhao Qiu, Zehao Wu, Xiuxiu Bai
Primary area: applications to robotics, autonomy, planning
Keywords: DETR, 3D object detection, Query pruning

Abstract:
Query-based models are extensively used in 3D object detection tasks, with a wide range of pre-trained checkpoints readily available online. However, despite their popularity, these models often require an excessive number of object queries, far surpassing the actual number of objects to detect. The redundant queries result in unnecessary computational and memory costs. In this paper, we find that not all queries contribute equally --- a significant portion of queries have a much smaller impact compared to others. Based on this observation, we propose an embarrassingly simple approach that Gradually Prunes Queries (GPQ) according to classification scores that queries generated. Compared to existing pruning methods, our method introduces no additional learnable parameters. GPQ is easy to implement to any query-based method by integrating it in after-training fine-tune using an existing checkpoint. By using our method, one can easily generate several different models with fewer queries using an checkpoint has exicessive queries. Experiments on various advanced 3D detectors show that GPQ effectively reduces redundant queries while maintaining performance. Using our method, model inference on desktop GPUs can be accelerated by up to 1.31x. Moreover, after deployment on edge devices, it achieves up to a 67.86\% reduction in FLOPs and a 76.38\% decrease in inference time. The code will be available soon.

Decision: reject
Reviewer scores: 5, 3, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730544650884

thread_type:
official_review

<thread_object>
creator: reviewer_kmgm

<content>
summary:
This paper addresses the issue of redundancy in DETR-based 3D object detection models. The authors find that not all queries contribute equally to the detection task, and a significant portion of them have a minimal impact. They propose an approach called Gradually Prunes Queries (GPQ), which reduces the number of queries by pruning those with lower classification scores. The overall framework appears simple. However, my main concern lies in the discussion of related work. There is indeed a substantial amount of work aimed at speeding up query-based methods in 2D object detection. The authors did not provide a convincing and comprehensive review, nor did they offer a comparison with the most relevant methods.

strengths:
1. The paper is well-structured, with a clear abstract, introduction, methodology, experiments, and conclusion sections that logically flow from one to the next.
2. The proposed Gradually Prunes Queries method is simple and effective.
3. The experiment results, when compared to different standard detectors, show that the design is effective.

weaknesses:
1. There are some works aimed at speeding up query-based methods in 2D object detection, but this paper did not discuss. So, it is not clear if very similar methods exist in 2D object detection. 
[1] Yang, Chenhongyi, Zehao Huang, and Naiyan Wang. "QueryDet: Cascaded sparse query for accelerating high-resolution small object detection." Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 2022.
[2] Zhu, Yuan, Qingyuan Xia, and Wen Jin. "Srdd: a lightweight end-to-end object detection with transformer." Connection Science 34.1 (2022): 2448-2465.
2. It is not clear what the special challenges are in 3D space, and why the experiments are only conducted in 3D space.
3. Using fewer queries may lead to slower network convergence, so fewer queries might require more training iterations. However, since the same number of epochs were used in Table 1, it's difficult to conclude that the precision drops quickly solely due to fewer queries.

questions:
1.	Are there very similar methods proposed in the field of 2D object detection?
2.	Has there been a comparison with other methods that reduce the number of queries or speed up DETR? What challenges are there in transferring these methods to 3D object detection? How about the tested efficiency?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730677992956

thread_type:
official_review

<thread_object>
creator: reviewer_edff

<content>
summary:
The manuscript proposes to prune queries in DETR-based 3D detectors to speed up inference by almost 2x without loss of detection performance (in some cases even improving performance). Pruning is performed after training has finished without the need for any gradients to flow. The lowest scoring queries are incrementally removed one by one.

strengths:
- The paper is well written and and illustrated. The figures help support the text i.e. Fig 1 and 3.
- The approach is well motivated through experiment: Fig 2 shows that the imbalanced number of selections per query is prevalent across various DETR-based detectors. There indeed seem to be some queries that are used very often and others that are not.
- The experiments are thorough for the proposed approach (but lack deeper/broader investigations see weaknesses). The approach is evaluated on four current PETR-based architectures and show across the board that the pruning increases throughput and that detection performance is maintained or slightly improved. The last point is positively surprising.
- The proposed approach is indeed straight forward and it is great that it can be deployed on existing checkpoints after training. This makes it universally applicable to 3D DETR models. Simple and universally applicable enables impact.

weaknesses:
- The manuscript could have tried harder to investigate why training on a large set of queries and pruning them after is a better choice than training with fewer queries to begin with. The experiments show this (which is good), but the reader is left wondering about the why. 
    - Which are the queries that get pruned? Is it that they attend to similar areas in space but always loose out against more confident queries? 
    - If we took the top most confident queries (as per the proposed algorithm) and the least confident ones (invert the algorithm and prune the most confident ones) how would the detections from those two pruning strategies compare? Do the least confident queries still give good OBBs just not very confidently? Or are they only giving rubish?
- There are no pruning baselines evaluated leaving the reader to wonder about some of the algorithm choices and how close to optimal this approach performs.
    - How important is the incremental pruning vs pruning down to the desired number of queries in one go?
    - How well does an oracle-based pruner perform? Always prune the worst query when matched against GT bounding boxes. How does that pruned model perform relative to the proposed score based on? 
- I appreciate that the timings were collected on a Jetson Nano device which shows the clear motivation to deploy the algorithms in practice. Checking against related methods, though the timing information was typically performed on a desktop-grade GPU. Showing those timings in addition would allow more direct comparability. I think Table 3 and 4 could have been combined to make room for the aforementioned additional investigations.
- Showing reprojections of 3D bounding boxes into frames hides any depth errors. So I would have appreciated Fig 4 to include some top down 3D visualizations instead of just frame reprojections (which frankly all look very similar). 
- Algo 1: showing an algorithm is meant to help clarify the proposed method. I do not think it is achieving this aim as is. I think the algorithm depiction should be simplified more to where it actually captures the high level information. The details can certainly be found in the code once it is released.

questions:
Fig 2 renders poorly for me and the text is hard to read. Please fix.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730678377289

thread_type:
official_review

<thread_object>
creator: reviewer_xnmm

<content>
summary:
The paper proposes query pruning in typical query-based object detectors to improve their performance. Experiments on the nuscenes dataset show the effectiveness of the approach.

strengths:
+ The idea of using classification scores to prune queries is nice.
+ The paper is easy to understand.
+ The approach demonstrates promising results on the nuScenes validation set.

weaknesses:
- An alternate way to reduce the processing time of 3D detectors is to use Token Merge [A]. How does query prune quantitatively compare against Token Merge [A]?
- Another way to reduce processing time is to use model quantization. How does query pruning quantitatively compare against Model Quantization?
- It would be beneficial to quantitatively include results from the nuScenes leaderboard, particularly comparing against a strong camera baseline like SparseBEV with 640x1600 resolution.
- The experiments are conducted with super small backbone (ResNet50). It would be insightful to quantitatively evaluate the approach on higher resolutions (512x1508 and 640x1600) to assess its performance.

References:
- [A] Token Merging: Your ViT but faster, Bolya et al, ICLR 2023.

questions:
Please see the weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706604803

thread_type:
official_review

<thread_object>
creator: reviewer_xxxp

<content>
summary:
This work proposes Gradually Pruned Queries (GPQ), which removes redundant queries in transformer-based 3D object detection models. Queries are sorted by classification scores, and the lowest-score query is iteratively removed. Experiments on the nuScenes dataset validate the effectiveness of GPQ.

strengths:
- This work proposes a simple and reasonable pruning method for transformer-based 3D detection models.

weaknesses:
- Numerous typos are present, such as "3D objec detection" (L351), "each query as as the fundamental" (L264), "it is the queries" (L268), "The reason of why our method" (L300). The authors are encouraged to correct all typographical errors and improve their English writing to enhance readability.
- Lack of comparison with prior work. This paper does not provide comparisons with previous pruning methods or implemented baseline pruning techniques. The authors are encouraged to include comparisons with other pruning baselines to demonstrate the advantages of their approach.
- No ablation studies. This paper lacks ablation studies on the design aspects of their method. The authors are encouraged to conduct ablation experiments to illustrate the impact of their design choices.

questions:
- In Table 1, could the authors provide time and memory costs instead of accuracy? This would allow readers to more easily understand the relationship between the number of queries and model efficiency.
- In Figure 3, should the last column be labeled "Iteration 𝑛+1"? Or is Iteration 𝑛 run twice in this work?
- In Table 2, could the authors include model cost along with accuracy? This addition would help readers better understand the pruning effect of the proposed method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718287869

thread_type:
official_review

<thread_object>
creator: reviewer_nz4d

<content>
summary:
In query-based models, the number of queries significantly exceeds the actual number of objects for both 2d object detection and 3d object detection. The performance of models is positively correlated with the number of queries, and the performance consistently declines as the number of queries is reduced. In this paper, the authors propose that queries do not contribute equally and even there are queries are never selected as final results. The redundant queries result in unnecessary computational and memory costs. Therefore, the authors propose a query pruning method to gradually prune redundant queries in transformer.

strengths:
1. This paper is the first study to explore query pruning in query-based detectors. 
2. The experiments demonstrate the effectiveness of proposed method, which reduces redundant queries while maintaining performance.

weaknesses:
1. The experiment is incomplete and unpersuasive.

questions:
1. The font in Figure 1 is not consistent with other figures. The label of Y-axis is not clear.
2. In query-based models, there are content query and spatial query. Does the proposed method perform experiments on the detectors that use both content query and spatial query? If not, the author should add additional experiments to prove the effectiveness of the proposed method on this kind of detectors.
3. In StreamPETR and FocalPETR, two variants of the model are trained for 24 epochs and 60 epochs, respectively. In StreamPETR, the models in the ablation study are trained for 24 epochs, while trained for 60 epochs when compared with others. The authors should add additional comparison experiments which trained for 60 epochs. 
4. For model pruning methods, except for precision, FLOPs and time, the number of parameters and memory cost are also important evaluation metrics. The authors should provide the comparison experiments between detectors based on the number of parameters and memory cost.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732592964869

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
# To all reviewers

First, please allow us to express our gratitude to all reviewers. In this paper, we propose a pruning method for Query-based 3D object detection approaches, aiming to prune redundant queries in the model. 

We discussed the redundancy of queries in existing 3D object detection methods in the paper and proposed a gradual pruning approach that eliminates redundant queries while maintaining the performance of the original model.  

- **Reviewer nz4D** suggested that we should add results of applying GPQ to base models trained for more epochs.  
- **Reviewer xxXP** recommended including ablation studies and comparisons with other methods.  
- **Reviewer Xnmm** suggested comparing GPQ with the ToME method and model quantization approaches, and adding experimental results on models with larger backbones and image sizes.  
- **Reviewer EdfF** raised questions about query pruning strategies and metrics. Based on their guidance, we added a comparison of model runtime performance before and after pruning on desktop GPUs.  
- Following **Reviewer KmgM**'s suggestion, we extended GPQ to the 2D object detection domain and explored the integration of GPQ with training, allowing for pruning during training.


# List of changes

According to the comments of all reviewwers, we have made the following changes to our paper:

1. In **Table 1**, we have added the memory usage and FPS changes with varying query numbers (Page 2).  
2. In **Section 2.2**, we have explained why existing pruning methods for Transformers cannot or are difficult to transfer to 3D object detection models (Page 4).  
3. In **Section 3.4**, we have inserted a new **Figure 4** to more clearly demonstrate the distribution differences between queries pruned using GPQ and those obtained by training with fewer queries from scratch (Page 7).  
4. In **Table 2**, we have added more models and included backbone, image size, and model runtime (FPS) on RTX3090 (Page 8).  
5. In **Section 4.3.2**, we merged the original **Table 3** and **Table 4** (Page 9).  
6. We have added **Section 4.3.3** (Page 9), where we compare GPQ with previous methods.  
7. We have added **Section 4.4** (Page 10) to include ablation experiments.  
8. We moved the original **Figure 4** to the appendix (now **Figure 5**, Page 15) and added visualizations from a top-down view.  
9. In **Appendix A.2**, we have added the effects of GPQ in 2D object detection (Page 16).  
10. In **Appendix A.3**, we have added the resules of integrating GPQ with training from the beginning (Page 16).
11. In **Appendix A.4**, we have added the results of GPQ on fully converged models (Page 17).
11. Additionally, we made several minor adjustments, such as changes to the font in images, fixing typos, and increasing image resolutions.
</content>
</thread_object>

</thread>


Metareview: