Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection

Authors: Chuhan ZHANG, Chaoyang Zhu, Pingcheng Dong, Long Chen, Dong Zhang
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Contrastive Learning, Knowledge Transfer, Open-Vocabulary Object Detection

Abstract:
In pursuit of detecting unstinted objects that extend beyond predefined categories, prior arts of open-vocabulary object detection (OVD) typically resort to pretrained vision-language models (VLMs) for base-to-novel category generalization. However, to mitigate the misalignment between upstream image-text pretraining and downstream region-level perception, additional supervisions are indispensable, e.g., image-text pairs or pseudo annotations generated via self-training strategies. In this work, we propose CCKT-Det trained without any extra supervision. The proposed framework constructs a cyclic and dynamic knowledge transfer from language queries and visual region features extracted from VLMs, which forces the detector to closely align with the visual-semantic space of VLMs. Specifically, 1) we prefilter and inject semantic priors to guide the learning of queries, and 2) introduce a regional contrastive loss to improve the awareness of queries on novel objects. CCKT-Det can consistently improve performance as the scale of VLMs increases, all while requiring the detector at a moderate level of computation overhead. Comprehensive experimental results demonstrate that our method achieves performance gain of +2.9% and +10.2% AP_{50} over previous state-of-the-arts on the challenging COCO benchmark, both without and with a stronger teacher model.

Decision: accept (poster)
Reviewer scores: 8, 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730095633396

thread_type:
official_review

<thread_object>
creator: reviewer_tqtb

<content>
summary:
The paper introduces Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection (CCKT-Det), a method designed to enhance the detection of novel objects in open-vocabulary settings without relying on additional supervision like image-text pairs or pseudo annotations. It leverages pretrained vision-language models (VLMs) to transfer knowledge effectively from base categories to novel ones. The key contributions include using semantic priors to guide object queries and a regional contrastive knowledge distillation mechanism to align the visual features between a student and teacher model. CCKT-Det achieves state-of-the-art performance on COCO and LVIS benchmarks, improving novel object detection without extra data, and scales efficiently with stronger VLMs while maintaining moderate computational overhead.

strengths:
The paper is well-structured, with solid methodological rigor. The authors present comprehensive experiments and ablation studies that demonstrate the effectiveness of their proposed method. The proposed CCKT-Det framework has high significance in advancing the state of open-vocabulary object detection. By eliminating the need for additional supervision and showing competitive performance with existing methods that do require extra data, the paper addresses a key limitation in the field.

weaknesses:
- Although you claim and emphasized CCKT-Det did not rely on the extra data, in contrary to the previous method, you use the extra MLLM to be a discriminator and generate the prior semantic guidance.

questions:
- In the paper, you mention, “In contrast to previous works, where object queries are static and fixed for each image after training, we dynamically inject text embeddings {ti} as semantic priors into object queries to form the language queries for each image.” However, similar ideas have been explored in earlier works, such as GroundingDINO. It would be appropriate to supplement your references by citing these related works and discuss their similarities and differences.
- In Table 4, the paper states that “While semantic guidance demonstrates effectiveness to a degree, its efficacy is constrained in the absence of regional contrastive training.” However, contrary to this claim, regional contrastive appears to have a detrimental effect on semantic guidance, as the ablation experiment did not demonstrate any significant effectiveness of regional contrastive training. This seems to contradict the statement in the paper, “As indicated in the third row of Table 4, the absence of this loss results in a performance decline of 5.4% AP50 on novel classes.” Based on the results, there appears to be no significant relationship between Regional-Contrastive Loss and Semantic Guiding, while it works in conjunction with Similarity Classification. This seems to be a clerical error.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730467958193

thread_type:
official_review

<thread_object>
creator: reviewer_kmme

<content>
summary:
The paper introduces CCKT-Det, a novel framework for open-vocabulary object detection (OVD) that operates without any additional supervision beyond the base training set. The method leverages semantic priors and regional contrastive knowledge distillation to align the detector with the visual-semantic space of vision-language models (VLMs).  This paper presents semantic priors to boost the detector's capacity for novel object recognition and a unique loss function that heightens the detector's sensitivity to new objects by aligning region-level features. Based on these contributions, this work achieves start-of-the-art results on COCO benchmark.

strengths:
* The paper introduces a novel framework, CCKT-Det, for open-vocabulary object detection (OVD) that does not rely on any additional supervision beyond the base training set and achieves start-of-the-results on the COCO benchmark.
﻿
* Comprehensive experiments are conducted on multiple benchmarks (OV-COCO, LVIS, Objects365), demonstrating the method's effectiveness. The results show consistent performance improvements as the strength of the teacher model increases.  The paper includes thorough ablation studies to validate the effectiveness of each component of the proposed method.

* The paper demonstrates that high-performance OVD can be achieved without the need for additional supervision, such as image-text pairs or pseudo annotations. This is a significant advancement, especially in scenarios where such data may not be readily available.

weaknesses:
* It is better to add a more detailed result comparison in Tab.1 and Tab.2. For example, show the AP_Base result for COCO and AP_c and AP_f results for LVIS.
* This method has shown promising results in detecting novel objects, but it seems not very good for detecting base objects.

questions:
None
</content>
</thread_object>

</thread>
<thread>
creation date:
1730483265344

thread_type:
official_review

<thread_object>
creator: reviewer_s2q1

<content>
summary:
This paper presents CCKT-Det, a method for open-vocabulary object detection (OVD) that eliminates the need for extra supervision like image-text pairs or pseudo-labels. It leverages cyclic and dynamic knowledge transfer between language queries and visual region features from pretrained vision-language models (VLMs). The approach uses semantic priors for guiding query learning and a regional contrastive loss to enhance detection of novel objects. CCKT-Det achieves significant performance gains on the COCO benchmark, showing scalability with stronger VLMs while maintaining moderate computational requirements.

strengths:
1. The proposed method is quite reasonable and can effectively transfer the visual and language capabilities of VLMs.
2. The experiments on the OV-COCO benchmark are relatively comprehensive, showing a significant improvement in detecting novel categories.
3. The ablation experiments are relatively thorough and comprehensive.

weaknesses:
1. The introduction to the task setup is limited, and there is a lack of a focused, clear description and visualization of the workflow details of CCKT-Det during training and testing. Figure 2 is somewhat confusing.
2. The AP performance of CCKT-Det on the OV-COCO benchmark is relatively poor, and its detection capability on base categories is not as strong as other methods. This may be the cost of improving detection capabilities for novel categories.
3. The experimental results did not report the AP for the base category.
4. There is no comparison or analysis of time complexity.

questions:
1. Why does the Regional-Contrastive Loss in the ablation study cause drop of AP50 (from 32.6 to 31.7)?
2. There are relatively few comparative methods on the OV-LVIS benchmark.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730598516360

thread_type:
official_review

<thread_object>
creator: reviewer_b8e4

<content>
summary:
In order to solve the problem of relying on a large amount of additional data for open vocabulary object detection, the authors propose CCKT-Det method. This method guides the learning of queries by pre-filtering and injecting semantic priors, and introduces regional contrast loss to improve the perception of queries to novel objects.

strengths:
1. The frameworks are explicit and effectively illustrates the idea of the method.
2. This method has an improvement in the detection performance of novel classes.

weaknesses:
1. The problem of the authors to solve and the motivation are both ambiguous. The authors point out that some methods rely on a large amount of extra caption data, but many methods can achieve high performance without it, such as the quoted OV-DQUO.
2. The authors introduce semantic prior, but do not clearly define the concept.
3. Figure 3 shows that the semantic prior needs to go through the VLM's text encoder, what is the difference between the semantic prior and the caption? And what is the difference between semantic priors and the prompts that currently popular? Neither of these issues is addressed by the authors.
4. The authors do not present the detection performance of the base class in the experiment.

questions:
1. Clearly state the motivation for this paper.
2. It is necessary to give a clear definition of semantic prior and explain its difference from caption and prompts.
3. Please explain why the performance of the base class is not shown in the experiment.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730815442471

thread_type:
official_review

<thread_object>
creator: reviewer_6x5s

<content>
summary:
This paper introduces CCKT-Det++, an open-vocabulary object detection framework designed to detect novel object categories without relying on extensive additional supervision. The approach leverages semantic priors as guidance and a regional contrastive knowledge distillation loss to improve the model's detection capability for novel classes. CCKT-Det++ utilizes visual-semantic embeddings from both VLMs and MLLMs to dynamically transfer knowledge from both image and text encoders, aligning the detector’s latent space with meaningful, structured knowledge.

strengths:
This work is easy to follow. Most of the used techniques are correct.

weaknesses:
1. While CCKT-Det++ performs well by leveraging stronger teacher models, it is heavily reliant on the quality and alignment of these teacher models. If the teacher model has limitations or biases, these could propagate to CCKT-Det++, affecting performance and potentially introducing unintended biases.

2. Although the student model maintains moderate parameters, using stronger teacher models during training and inference can introduce significant computational costs. Scaling the teacher models effectively requires access to substantial computational resources, which could limit practical applications for researchers without high-performance infrastructure.

3. The approach relies on pseudo annotations, which are less accurate than human annotations. If the pseudo labels are of low quality, they could introduce noise into the training process, possibly harming model performance on more challenging or nuanced object classes.

4. While semantic priors improve performance on novel objects, there is a risk that the model might over-rely on these priors, leading to reduced flexibility in identifying objects that fall outside its learned semantic scope. This dependency may affect its generalization to truly unseen classes in new environments.

5. The method heavily relies on CLIP for both semantic priors and feature extraction, making it vulnerable to limitations inherent in the CLIP model. For instance, CLIP's biases in language and visual associations could impact detection accuracy and lead to incorrect classifications in culturally or contextually sensitive settings.

questions:
Most concerns are listed on above boxes. Hungarian matching based on CLIP embeddings is central to your contrastive knowledge transfer scheme, aligning regional embeddings with the teacher’s visual-semantic space. However, given that CLIP embeddings may not fully capture object-specific regional nuances, why would this Hungarian matching approach be expected to significantly improve performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732259505148

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We thank all reviewers for their valuable comments and constructive suggestions. We have made the revisions to the main paper accordingly, and the revised part is marked by blue. The main revisions are summarized as follows:

1. Added more explanations about ablation experiments; added more clarifications and descriptions when necessary.
2. Some minor modifications over clerical error.

The details of the revisions and other supplemented experiment results are referred to the following official comments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732496107519

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response 2

comment:
All authors would like to thank the reviewers again for their valuable efforts. As the discussion period is coming to an end, if you have any further comments or suggestions, please feel free to let us know. Thank you.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper was reviewed by five experts in the field. The paper received mixed review ratings of 8, 5, 6, 5, and 6. The AC read the paper, reviews, and rebuttal carefully. 

Reviewer B8e4 and kmme are negative about the paper. Both reviewers asked for results on base classes. Reviewer B8e4 also had some clarification questions. During the rebuttal, reviewer B8e4 said their concerns had been addressed, but they still kept the original rating without further explanation. In this case, the AC considered the clarification questions raised by reviewer B8e4 were addressed. The authors' response included detailed results on base classes, and the AC believes they sufficiently addressed the concerns. These results should be incorporated into the final paper.

Therefore, the decision is to recommend the paper for acceptance. The reviewers did raise some valuable suggestions in the discussion that should be incorporated in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.