Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection
Authors: Chuhan ZHANG
Chaoyang Zhu
Pingcheng Dong
Long Chen
Dong Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Contrastive Learning
Knowledge Transfer
Open-Vocabulary Object Detection
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: Our proposed framework constructs a cyclic and dynamic matching between semantic priors and visual region features extracted from VLMs, which forces the detector to closely align with the visual-semantic space of VLMs.

Abstract: In pursuit of detecting unstinted objects that extend beyond predefined categories, prior arts of open-vocabulary object detection (OVD) typically resort to pretrained vision-language models (VLMs) for base-to-novel category generalization. However, to mitigate the misalignment between upstream image-text pretraining and downstream region-level perception, additional supervisions are indispensable, e.g., image-text pairs or pseudo annotations generated via self-training strategies. In this work, we propose CCKT-Det trained without any extra supervision. The proposed framework constructs a cyclic and dynamic knowledge transfer from language queries and visual region features extracted from VLMs, which forces the detector to closely align with the visual-semantic space of VLMs. Specifically, 1) we prefilter and inject semantic priors to guide the learning of queries, and 2) introduce a regional contrastive loss to improve the awareness of queries on novel objects. CCKT-Det can consistently improve performance as the scale of VLMs increases, all while requiring the detector at a moderate level of computation overhead. Comprehensive experimental results demonstrate that our method achieves performance gain of +2.9% and +10.2% AP_{50} over previous state-of-the-arts on the challenging COCO benchmark, both without and with a stronger teacher model.

Published as a conference paper at ICLR 2025
CYCLIC CONTRASTIVE KNOWLEDGE TRANSFER FOR OBJECT DETECTION
Chuhan Zhang!? Chaoyang Zhu', Pingcheng Dong', Long Chen', Dong Zhang! 1.2+
'The Hong Kong University of Science and Technology
2 AT Chip Center for Emerging Smart Systems (ACCESS)
(chuhanzhang, Longchen, dongz eust .k; scan. zhuh 2huh&gmail com pingeheng. donge connect. ust. hk
ABSTRACT
In pursuit of detecting unstinted objects that extend beyond predefined categories, prior arts of open- conabuabuyl object detection (OVD) typically resort to pretrained vision-language models (VLMs) for base-to-novel category generalization. How- ever, mitigate the misalignment between upstream image-text pretraining and downstream region-level perception, additional supervisions are indispensable, e.g., image-text pairs or pseudo annotations generated via self-training strategies. In this work, we propose CCKT-Det trained without any extra supervision. The proposed framework constructs a cyclic and dynamic knowledge transfer from language queries and visual region features extracted from VLMs, which forces the detector to elosely align with the visual-semantic space of VLMMs. Specifi- cally, 1) we prefilter and inject semantic priors to guide the learning of queries, and 2) introduce a regional contrastive loss to improve the awareness of queries novel objects. CCKT
consistently improve performance the scale of VLMs increases, all while requiring the detector at a moderate level computa- tion overhead. Comprehensive experimental results demonstrate that our method achieves performance gain of +2.9% and +10.2% APso over previous state-of- the-arts on the challenging COCO benchmark, both without and with stronger teacher model. The code is provided at
INTRODUCTION
Object detection, fundamental perception task in computer vision, entails both localization and classification of objects on given images. The last decade has witnessed great success in object detection including advanced architectures (Ren, 2015; Carion et al., 2020), tailored loss objec- tives (Lin et al., 2017; Rezatofighi et al., 2019). and large-scale datasets (Gupta et al., 2019; Shao et al., 2019). However, canonical close-set detectors are constrained by small-scale and predefined categories, rendering them less capable of detecting new concepts (Bansal et al., 2018; Gu et al., 2021). To deploy detectors in real-world scenarios with countless concepts, many of which have no annotations in the training set, open-vocabulary object detection is formulated to facilitate the recognition of novel objects (Zareian et al., 2021).
Drawing the impressive image-level zero-shot learning capabilities of vasion-language models like CLIP (Radford et al., 2021), recent initiatives replace the learnable classifier weights in tradi- tional detectors with frozen text embeddings generated from VLMs text encoder that takes input template prompts filled with category names. To address the challenges posed by the misalignment between image-text pre-training and region-level perception, as shown in Figure 1, existing methods heavily rely on additional data such as: 1) caption datasets, e.g., CC3M (Sharma et al., 2018) and COCÓ Caption (Chen et al., 2015), or 2) meticulously designed self-training strategies generate pseudo annotations. These extra supervisions are typically coarse and noisy, leading to inaccurate region-word woodd alignment. This raises the question of whether moderate level of weak supervision is sufficient while effectively maximizing the knowledge transfer from VLMs and multi-modal large language models (MLLMs) (Wang et al., 2023).
Corresponding author (Dong Zhang).

---

Published as a conference paper at ICLR 2025
A RANDY'S DONUTS
"A photo of #
A RANDY'S
"A photo 17 Novel "dienu"
R RANOFA ''S
photo 0
"gassor
"car"
Pseudo annotations
DONUTS
DONUTS
Base
Know ledge Transfer
Image
captions
"car"
VLMS
MILLMs
"n husting scane of people gothered In orontof dount stop with giant darut sigr"
(b) Pseudo annotations & self- telf-trainigg techniques
Our proposed cyclic contrastive knowledge transfer
(a) Image captions
The evolution of extracting novel concepts in OVD models. Compared to existing methods by using extra captions (a) or pseudo annotations & self-training strategies in (b), we propose leveraging semantic priors to reveal novel concepts and employing contrastive knowledge distillation paradigm in (c) to align the enriched teacher space with the region-aware student space.
Figure 1:
In this paper, as illustrated in Figure 2, we introduce novel framework termed as CCKT-Det without any additional supervision to optimize the pipeline for efficient knowledge distillation in OVD. We argue that a cyclic and dynamic knowledge transfer from VLMs can efficiently mimick their well- aligned visual-semantic space and close the gap with methods that leverage extra data. Concretely, first construct language queries by integrating semantic priors into the input of the detector, i.e., object queries, which makes the model novel-concepts aware and accelerates model convergence (ref Sec. 3.2). To leverage the enriched semantics, further propose regional contrastive knowledge distillation loss that aligns the region-level visual features queries from the student model with that from the teacher model (ref Sec. 3.3). The semantic priors and regional contrastive knowledge distillation loss together form a cross-modal cycle with detector in between that can dynamically integrate semantic and visual knowledge from VLMs. The cycle forces detector to tightly align with VLMs as breaking either visual or semantic alignment with VLMs would fail this cyclic knowledge transfer evidenced by our ablation study. Unlike existing methods that enhance performance by utilizing a more complex student backbone, our method exhibits consistent performance improve- ments as the strength and reliability of the teacher model increase without introducing any additional computational overhead during inference. We validate the effectiveness of the proposed CCKT-Det across the COCO (Lin et al., 2014), LVIS (Gupta et al., 2019), and Objects365 (Shao et al., 2019) transfer benchmarks. On the OV-COCO (Zareian et al., 2021) benchmark, CCKT-Det enhances the APso for novel categories by 2.9% compared to previous methods. This enhancement consistently scales with stronger teacher model, culminating in on novel classes, all without the need for additional training data. On OV-LVIS (Gupta et al., 2019), CCKT-Det attains competitive average precision of 18.2% average AP on rare without the incor- poration of additional training data. Furthermore, when the model trained on the LVIS dataset is directly evaluated on the COCO and Objects365 datasets, CCKT-Det++ yields performance of 38.5% AP and 15.2% AP, narrowing the performance difference compared to supervised models on COCO (-17%) and Objects365 (-40%), respectively.
ffte-off-the-ar accuracy of 46.0% AP
The main contributions are summarized as the following:
We propose semantic priors as guidance and regional contrastive knowledge distillation loss to effectively align with the visual-semantic space of VLMs.
With the proposed two components, CCKT-Det constructs cross-modal cycle that the detector in between can dynamically choose to transfer knowledge from.
The performance steadily improves as the teacher VLMs and MLLMs scale, and the proposed method achieves state-of-the-art in OVD without relying any additional training data.
RELATED WORK
Contrastive Representation Learning. Recent developments in self-supervised pretraining, partic- ularly through contrastive learning, have highlighted its ability to reduce reliance on labeled datasets. image-level
These methods primarily focus the extraction of robust visual representations tasks (He et al., 2020; Chen et al., 2020). While effective at capturing general features, such global representations face significant challenges when applied to dense perception tasks. To mitigate,
2

---

Published as a conference paper at ICLR 2025
region-level contrastive learning has been introduced with different variants, such sliding win- dows (Xiao et al., 2021), object proposals (Wei et al., 2021; Hénaff et al., 2021), 1), and random query patches (Dai et al., 2021) to capture local visual clues. These strategies transform and adapt repre- sentations from image-level to pixel- or region-level, enhancing spatial reasoning capability in dense perception tasks. At the core of these methods is the definition of regional-level pre-text tasks. In our work, we leverage contrastive learning to distill rich semantic knowledge from a teacher model into the regional representations of student model, tightly integrated with Hungarian matching to estab- lish regional-level pairs. The proposed method captures higher-order correlations and dependencies in representation space, offering more diverse set of negative and positive samples for distillation, thereby alleviating the constraints of learning exclusively from positive pairs of base categories.
Vision-Language Models (VLMs) and Multimodal Large Language Models (MLLMs). VLMs and MLLMS have profoundly transformed the integration of linguistic information in visual recog- nition tasks. As a prominent example of VLMs, CLIP (Radford et al., 2021), successfully aligns which image goes with which text on billion-scale image-text pairs, demonstrating strong general- ization on novel categories. Concurrently, MLLMs, e.g.. BLIP- (Li et al., 2023b), MiniGPT-4 (Zhu et al., 2023), and Vicuna (Zheng et al., 2023) connect image encoders with large language models to interpret visual clues and respond to human instructions effectively. Pretraining on web-scale data has endowed VLMMs and MI.LMss with a remarkable capability across various downstream tasks and benchmarks (Li et al., 2023a; Liu et al., 2024; Zeng et al., 2024). In this study, we adapt VLMs and MLLMs as teachers to convey novel semantic knowledge to our proposed detector.
Open Vocabulary Object Detection (OVD). To transfer the rich image-level semantic knowledge in VLMs to region-level, many works (Zhu & Chen, 2024) seek to utilize additional supervisions, e-g-, region-word alignment on image-text pairs in a weakly supervised manner which noisy, or pseudo annotations generated via a self-training strategy which needs to know novel categories during training. Another methodology distills from teacher VLMs to student detectors for semantic knowledge transfer. The pioncering work ViLD (Gu et al., 2021) distills region embeddings from teacher VLMs in an element-wise manner, rendering the cross-modal similarity comparison toward the well-aligned sesual-semantic space of VLMs. Building upon ViLD, BARON (Wu et al., 2023a) distills from a single region to a bag-of-regions level, effectively mining the co-occurrence and compositional structure of visual concepts inherently captured by VLMs. In addition, the pretrained image encoder of VLMs can be directly leveraged as a detector backbone while adding detection heads on top of it. The backbone is either frozen (Kuo et al., 2022: Zhong et al., 2022), or fine- tuned (Kim et al., 2023b) on detection datasets. To address the distribution gap between image- level pretraining and region-level perception, miscellaneous techniques including prompt tuning (Du et al., 2022; Feng et al., 2022) have been proposed. In particular, CORA (Wu et al., 2023c) utilizes region prompting to refine CLIP image encoder for improved localization. DITO (Kim et al., 2024) introduces hhifted-window learning technique to the region-centric image-text pretraining.
METHOD
3.1 OVERVIEW
The objective of OVD is to train the detector on base categories Op and evaluate its performance on both base and novel categories CN, where CB n CN (Zareian et al., 2021; Bansal al., 2018). As illustrated in Figure 2, our framework follows DETR-style detectors (Carion et al., 2020; Zhu et al., 2020), where learnable object queries serve as a global inductive bias that incorporate image- specific contexts. To fully exploit the regional structure embedded in the visual-semantic space of VLMs, we establish a cyclic cross-modal knowledge transfer between language-guin -gidedd queries and teacher visual region features. The semantic prior embeddings are added to object queries as the input language queries to the detector decoder, while object crops are encoded by CLIP image encoder to form the teacher visual region features (Gu et al., 2021).
3.2 SEMANTIC PRIORS AS GUIDANCE
We first identify the problem of concept existence, i.e., whether not the concept exists in the given image. Such information acts as a semantic prior specific to the image. This problem intrin- sically dynamic and tailored to each specific image. Prefiltering irrelevant concepts can avoid
3

---

Published as a conference paper at ICLR 2025
Query Embedding
Prediction heads
Results
FFN
Class, Box
Backbone
Encoder
Decoder
FFN
Object
No
Training
Missing cake annotation
Class, Box
FFN
FFN
Object
Inference
FFN
Training Inference
Semantic prior
Detected cake
Classification by CLIP/ GVT
"a photo of {class)" C base
CLIP text encoder
Toure 00000
"u photo of {class) E Cuovet
"a photo of [class)" "
Fnovet
Class e Caase for train Class priore
Semantic prior embeddings
Call forinference
class
Coft
The overall architecture of our CCKT-Det. By querying the existence of object categories within an input image, we dynamically guide object queries to explore novel concepts using seman- tie priors, which enables awareness of novel categories.
Figure 2:
the detector to falsely predict classes that are non-existent in current image and ease the burden of recognizing novel objects the number of existent base objects is significantly reduced. How- ever, existing methods (Zang et al., 2022; Wu et al., 2023a; Kim et al., 2024) still intervene in the prediction of existent concepts with non-existent concepts after training.
We propose two alternatives to resolve the concept existence. The first approach follows the zero- shot classification procedure of CLIP. For each image in validation embeddings {t/} E RC DD of all classes is computed and subsequently activated by sigmoid function. The text embeddings are encoded from CLIP text encoder by filling category names into template prompts. Logits that are below threshold p are filtered out to ensure that only highly con- fident classes are identified present ones. However, CLIP lacks fine-grained visual perception abilities and behaves like bags-of-wordh (Yuksekgonul et al., 2023). To bypass this drawback, we instead employ a more robust MLLM (Wang al., 2023) that is proficient in multi-class identifi- cation to determine the presence of specific concept within an image. For example, we prompt the MLLM with "Question: Does [OBJ] exist in the image? Answer:". The model expected to respond with "Yes/No", making the problem of concept existence a binary classification problem. For more details of this operation, please refer to (Wang al., 2023).
similarity score with text
In contrast to previous methods, where object queries are static and fixed for each image after train- ing, we dynamically inject semantic prior information into object queries to form the language queries for each given image. Specifically, the semantic prior embeddings {4:} are text features of those remaining classes (above the threshold p) encoded by CLIP text encoder (Radford et al., 2021) via filling their category names into template prompts. These embeddings added to the learnable object queries to form the language queries. This process can be expressed as:
e Fe
where F& denotes the transformer decoder in detector parameterized by v 2 refers to visual image contexts from transformer encoder, and q stands for the original learnable object queries. Note that I are semantic priors tailored to each image. During training, / is selected based corresponding annotations the given image. In inference, we first leverage the aforementioned two approaches to determine which concepts present, then is selected according to those existent concepts. Figure illustrates the forward pass of our proposed method.
Comparisons with Conditional Matching and Language Guided Query Selection. OV- DETR (Zang et al., 2022) uses object queries conditioned on text embeddings for aware re- gression, with pseudo annotations and self-training guiding the decoder to propose novel objects. In contrast, we incorporate the semantic priors into object queries for cyclic knowledge transfer, effi- ciently mining VL.Ms' regional structure. Our approach also avoids OV-DETR's repetitive per-class
4

---

Published as a conference paper at ICLR 2025
Box + Cls costs
Nx Object querks
Def-DETR Decoder
R plcliookmum
Regional contrastive loss
Sampling
Visual Region
et BA wyxk.do
E1
71
Cache
Hungarian
e2
Matcher
:
CLIP Text
Em
I'm
CLIP
L. Lowest costs
Image Encoder
Encoder
igradient
(Base + Novel)
Semantic prior
Similarity distance costs
Mx candidate embs
Object crop
Figure 3: Illustration of our contrastive knowledge transfer scheme. We first align semantic-enriched regional embeddings with teacher's Aisualsemmatle space through contrastive loss. Regional em- beddings with the lowest Hungarian matching cost are then considered as positive pairs for distilla- tion, enabling explicit alignment of base objects and implicit learning of novel objects.
decoding by pre-filtering irrelevant concepts, enhancing both speed and performance. Grounding DINO (Liu et al., 2023) utilizes a text encoder to process textual inputs, selecting the top N image patch features object queries based their maximum similarity the text features. In con- trast, our method employs teacher VLMM//MILMS for semantic priors, which mitigates competition among similar concepts. Besides, we adapt the textual inputs to be specific each image rather than relying on fixed inputs. While Grounding DINO's s object queries may still include non non-existent concepts due cross-attention mechanisms, our approach focuses exclusively on concepts that present in the image.
3.3 REGIONAL CONTRASTIVE DISTILLATION
Following previous works (Carion et al., 2020; Zhu et al., 2020), our loss function is based on the optimal bipartite matching between predicted and ground-truth objects:
=rg min dEBN
Cmatch (yi (o(s))
The matching cost considers the accuracy of both class prediction and bounding box regression:
Ematch (ya: (i)) - (o (c+) (o Lsboz (b;, ba(t))-
In OVD, since the detector is trained only on annotations of base classes, the prediction of detector prone to biased toward base categories. In contrast, VLMs already encapsulate rich semantics of novel classes in their well-aligned visual-semantic space and are superior on base-to-novel gen- eralization. In addition to the negative log-likelihood used original matching, we also consider probability calculated by VLMS in the matching cost, defined as:
e.t"
(4)
T
||e||
where denotes sigmoid function, represents the visual region embeddings corresponding to object queries after the last decoder layer and projection head, and temperature for sharpness scaling. In inference, we only use the probability computed by VLMs for classification.
Since e in Eq. contextualizes the semantic priors the visual clues tailored the current image as it is progressively forwarded through the decoder, the matching in Eq. 3 and Eq. 4 performs reliable similarity comparison within single text modality. However, we argue that transferring visual knowledge from VLMs can fully recover and unleash the pretrained visual-semantic alignment. It noteworthy that semantic priors added to the object queries, which is the input to detector. At the output of the detector, distills visual knowledge from VLMs via proposed regional contrastive loss, hence this cycle strictly forces the object queries to bind knowledge from both visual and text modality of VLMs. Recent text-to-image generation models (Gal et al., 2022; Ruiz et al., 2023) learn to generate personalized and subject-driven concepts by binding visual features of concept randomly initialized token in the input embedding look-up table. Here, object
5

---

Published as a conference paper at ICLR 2025
queries are also learnable tokens, however, in contrastive transfer cycle, object queries learn to associate semantic priors with region-level visual features. Formally, the regional contrastive knowledge distillation loss is defined as the following:
log( (pp(1fe//))
Steacher-sstdeen
(5)
2j= j= exp(rf (//))
M
exp(e)rr/T) ش exp(e r/))
-stucent-rteacher
log(
(6)
In these equations, M represents the number of ground objects each image. The Hungarian matcher selects M region embeddings out of N candidates from detector output to form the match- ing pairs as shown in Figure 3, which are then aligned with their corresponding teacher features r. Teacher features are extracted offline, where object crops are processed by the image encoder of teacher and stored in a cache for computational efficiency.
The overall loss function L(y, (i) is defined as:
C(y. 0) Abbou Cobox (b, b) + Acontrust (r, + (p,
where the bounding box loss includes both L1 and generalized IoU loss (Rezatofighi et al., 2019). The regional contrastive distillation loss Lcontrast is the average of Eq. 5 and Eq. 6, and the classi- fication loss is focal loss (Lin et al., 2017).
Distilling from both positive and negative pairs. The objective used in existing knowledge distil- lation methods for OVD is typically L1 loss that aligns the region embedding of the student model with that of the teacher model. However, the representational relation is inherently structured, with dimensions exhibiting complex inter-dependencies (Tian et al., 2019). The LI loss is limited to dis- till proposals labeled as foreground ones from teacher individually. In contrast, we consider all other region candidates as negative samples, thereby providing a larger and more diverse set of negative instances for distillation. This approach can facilitate the capture of correlations and higher-order dependencies within the representation space.
4 EXPERIMENTS
Datasets and evaluation metrics. Following the OV-COCO benchmark (Zareian al., 2021), train the our model on 48 base categories and test on both the 48 base classes (CB) and 17 novel classes (Cx). We also conduct experiments and 405 frequent classes are treated as base categories, and 337 rare classes are considered novel categories. The main evaluation metric is Average precision (AP) for novel categories, i.e., AP For OV-LVIS, in (Wu al., 2023a;c), we report the averaged bounding box AP across IoUs from 0.5 to 0.95 for rare classes denoted as AP,-
LVIS (Gupta al., 2019), where 866 common
Implementation details. Our model built on Def-DETR (Carion et al., 2020). The backbone defaults to ResNet-50. We employ 3x training schedule following the standard practices. Due to the markedly greater number of classes in LVIS compared to COCO, we allocate 300 queries for COCO and 600 queries for LVIS, respectively. The model is trained with semantic guidance in Sec. 3.2 for the first 30 epochs with a base learning rate 10-4. Subsequently, training continues with the incorporation of regional contrastive distillation loss, accompanied by learning rate decay factor of 0. post the 3 31st epoch. A batch size of 2 6 IRTX-3090 GPUs) is employed, with Adam W optimizer. Gradient clipping is adopted with maximum norm 0.1. Following (Carion et al., 2020; Zareian et al., 2021), is set to 0.07, and the loss weights are set to AgIoU' = 2.0, AL1 = 5.0, Aeta 2.0, and Acontrast = 1.0. The number of template prompts set to 12 following (Cho al., 2024). For training efficiency, we follow the practice of ViLD (Gu et al., 2021) and OV-DETR (Zang al., 2022) by extracting region crop features offline, as well as the semantic priors See. 3.2.
4.1
MAIN RESULTS
Comparisons with the state-of-the-arts. Most OVD methods require weak pseudo annotations during training. For example, BARON+ (Wu et al., 2023a) and CoDet (Ma et al., 2024) require additional caption datasets in which novel concepts be discovered and mined. OV-DETR (Zang

---

Published as a conference paper at ICLR 2025
Methods
Supervisions
Backbone
AP (%)
AP50 (%)
Annotation:
Extra caption datasets, Weak/Pseudo Labels CB JCN
Detic (Zhou et al., 2022)
ImageNei21K & CC3M
RN50-C4 (24M)
27.8
42.0
OV-DETR (Zang et al., 2022)
Pseudo annotation
RN50 (24M)
29.4
52.7
RegionCLIP (Zhong et al., 2022)
CC3M & COCO Caption
RN50x4 (87M)
39.3
55.7
CoDet (Ma et al., 2024)
CC3M COCO Caption
RN50 (24M)
30.6
46.4
BARON+ (Wu al., 2023a)
COCO Caption
RN50-C4 (24M)
42.7
51.7
CORA+ (Wu et al., 2023c)
COCO Caption
4 (87M)
43.1
56.2
CFM-ViT (Kim al., 2023a)
LAION-2B
VIT-L/16(307M)
34.3
46,4
DITO (Kim et al., 2024)
LAION-2B
ViT-R/16 (86M)
36.6
48.8
DITO (Kim al., 2024)
DataComp-I1
ViT-L/16(307M)
40.2
54.6
Annotation: Instance-level labels in Cs
ViLD-ens (Gu et al., 2021)
CLIP
RN50-FPN (24M)
27.6
51.3
F-VLM (Kuo et al., 2022)
CLIP
RN50-FPN (24M)
28.0
39.6
BARON (Wu et al., 2023a)
CLIP
RN50-FPN (24M)
34.0
53.5
CORA (Wu al., 2023c)
CLIP
RN50 (24M)
35.1
35.4
CORA+ (Wu et al., 2023c)
CLIP
RN50x4 (87M)
41.7
43.8
BIND (Zhang al., 2024)
CLIP
ViT-B/16 (86M)
36.3
50.2
BIND (Zhang ct al., 2024)
CLЛP
VIT-L/16
(307M)
41,5
54.8
CLIP-Sell (Wu et al., 2023b)
CLIP
VIT-B/16 (86M)
37.6
CLIP- Self (Wu et al., 2023b)
CLIP
VIT-L/14
(307M)
44.3
OV-DQUO (Wang et al., 2024)
CLIP
RN50x4
(87M)
45.6
CCKT-Det (ours)
CLIP
RN50 (24M)
38.0
35.0
CCKT-Det (ours)
CLIP
SwinB (88M)
41.9
40.9
CCKT-Det++ (ours)
CLIP RN50 (24M)
45.3
46.9
CCKT-Det++ (ours)
CLIP
SwinB (88M)
46.0
46.2
Table 1: Result comparisons on OV-COCO (Zareian et al., 2021). Methods are categorized into two groups based whether additional supervisions beyond instance-level labels in Cg are utilized during training e.g., extra image-text datasets, weak or pseudo labels. CCKT-Det++ denotes that we more reliable semantic priors filtered by GVT (Wang et al., 2023) in inference.
Methods
Supervisions
Backbone
AP, (%)
AP (%)
ViLD (Gu ct al., 2021)
CLIP
RN50-FPN (24M)
16.
24o4
OV-DETR (Zang et al., 2022)
CLIP & Pseudo annotation
RN50 (24M)
17 4m.4
26, maaa
CCKT-Det++ (ours)
CLIIP
RN50 (24M)
18. 18.2
27. 27.1
RegionCLIP (Zhong al., 2022)
CLIP COCO Caption
R50x4 (87M)
22. 22.0mak
32.3mac
CORA+ (Wu et al., 2023c)
CLIP & COCO Caption
RN50x4 (87M)
28. ] bbox
CoDet (Ma et al., 2024)
COCO Caption CC3M
SwinB (88M)
29. mack
39.2maa
CCKT-Det++ (ours)
CLIP & Pscudo annotation
SwinB (88M)
32.
44.3
Table 2: Result comparisons on OV-LVIS (Gupta et al., 2019). CCKT-Det++ achieves competitive performance with ResNet-50 (i.e., RN50) as backbone and without any extra data. Performance would further boost when larger backbone and pseudo annotations available.
ct al., 2022) goes a step further by generating pseudo annotations self-training manner. Table presents our results OV-COCO (Zarcian et al., 2021). Among methods that utilize additional su- pervision, CORA+ using COCO caption dataset achieves the best with AP 43.1 % on novel classes, with comparable backbone SwinB, our CCKT-Det closely matches CORA+ (AP 41.9%) without any additional data during detector training stage, demonstrating the superior data efficiency of our method. When equipped with a stronger teacher model GVT (Wang et al., 2023) for more reliable semantic guidance as in Sec. 3.2, our CCKT-Det++ achieves a new oftt--of-the-rt performance with 46.0% AP-Y compared to all methods. We also present result comparisons with the state-of- ofate-oftthe--e arts on OV-LVIS in Table 2. Initially, we train CCKT-Deet on the LVIS dataset without using additional data, achieving competitive performance compared to ViLD (Gu et al., 2021) and OV-DETR (Zang al., 2022). However, it falls behind state-of-the-art methods that leverage caption supervision.

---

Published as a conference paper at ICLR 2025
COCO (Lin et al., 2014)
Objects365 (Shao et al., 2019)
Methods
AP (%)
APso (%)
APra (%)
AP (%)
APso (%)
APrs (%)
Supervised (Gu ct al., 2021)
46.5
67.6
50.9
25.6
38.6
28.0
ViLD (Gu al., 2021)
36.6
55.6
39.8
11.8
18.2
12.6
DetPro (Du al., 2022)
34.9
37.4
12.1
18.8
12.9
F-VLM (Kuo al., 2022)
32.5
53.1
34.6
11.9
19.2
12.6
BARON (Wu et al., 2023a)
36.2
55.7
39.1
13.6
21.0
14.5
CCKT-Det++ (ours)
38.5
53.2
42.1
15.2
20.9
15.8
Table 3: Result comparisons of the LVIS-trained model evaluated on the COCO (Zareian et al., 2021) and Objects365 (Shao et al., 2019) datasets are presented without any fine-tuning.
#
Def-DETR
Semantic Guiding
ContrasiCe Loss
Similarity Classification
AP (%)
25.4
2
32.6
31.7
4
33.7
38.0
Table 4: The ablation study results by integrating different components of method, including concerning semantic guidance, regional contrastive loss, and similarity classification (i.e., a facili- tating operation in the post-processing stage). While semantic guidance demonstrates effectiveness to a degree, its efficacy is constrained in the absence of regional contrastive training. Furthermore, fully harness the capabilities of the well-trained model, essential employ alongside the similarity-base based classification post-proce post-processing. The comprehensive integration of all components effectively completes the operational framework of our CCKT-Det.
This is attributed to our use of naive hand-crafted prompts for semantic priors as guidance, which may lack robustness in distinguishing fine-graines concepts in the LVIS label space. To address this, we enhance the dataset by incorporating highly confident (i.e., 0.9) predictions from CCKT-Det into the base categories and train more robust version, which results in 32.8% AP on rare classes.
Results on transfer detection setting. Table 3 gives our results under the cross-dataset transfer evaluation setting. We train OV-LVIS and evaluate on COCO and Objects365 v1 (Shao al., 2019) datasets without any additional fine-tuning, We can observe that our method demonstrates superior performance, surpassing F-VLM by +6.0 +6.0% AP and BARON by +2.3%/+1..6% AP, while significantly reducing the performance gap compared to supervised models on COCO (-17%) and 0bjects365 (-40%). CCKT-Det++ shows better performance especially at higher IoU threshold of AP7s, demonstrating superior accuracy and reliability compared to its counterparts.
4.2 ABLATION ANALYSIS
We conduct ablation studies OV-COCO (Zarcian al., 2021) to evaluate the effectiveness of each proposed component, hyperparameters, and scalability. Unless stated otherwise, experiments performed using ResNet50 backbone and 3x training schedule.
Semantic priors as guidance. Table 4 ablates on our proposed two components. We can observe that after removing the guidance of semantic priors, the language queries revert to the default object queries in Def-DETR, leads to a significant performance drop from 32.6 AP to 25.4 AP, high- lighting the importance of semantic guidance in enhancing the model ability to detect novel objects. Relying solely on semantic priors as guidance, we achieve 32.6 AP on novel classes even without any extra supervision, surpassing OV-DETR and CoDet that exploit additional data.
Regional contrastive distillation. As validated by the difference between the row 2 and row 5 in Table 4, the absence of distillation loss results performance decline of 5.4% AP on novel classes. As elaborated in Sec. 3.3, our regional contrastive loss establishes cyclical relationship with semantic priors, thereby fully leveraging the visual-semantic space of VLMs. Therefore, we obtain the conclusion that the regional contrastive loss should be used in conjunction with similarity classification to form cyclic knowledge transfer loop. Row and row 5 in Table are models with identical weights, the only difference lies in how they give classification predictions
8

---

Published as a conference paper at ICLR 2025
Teacher
Backbone
AP (%) Epochs
Semantic Priors Threshold p AP (%)
OV-DETR VIT-B-32 ResNet50
29.4
50
0.70
38.0
VIT-B-16 ResNet50
35.2
36
6
0.90
37.6
ViT-L-114 ResNe150
38.0
12
0.70
36.8
3
VIT-H-14 SwinB
41.9
12
12
0.90
36.2
Table 5: Performance grows steadily as the teacher becomes stronger, while maintaining the backbone parameters moderate level.
Table 6: Ablation study results on semantic priors. Performance is influenced by the accuracy of classification ability of teacher.
in inference, and row is similarity-score classification while line 3 derives the prediction that incorporates logits score from classification branch. Since the same model presents disparate results with different post-processes, we reckon the reason behind is: after training, the embedding space of CCKT-Det is pulled close to the hidden space of the teacher. Specifically, added with semantic prior features, the queries are refined by the decoder, then guided be aligned with visual features encoded by the teacher. The integration of textual and visual features within the detection loop sets optimization direction towards mimicking teacher on region level. Using vanilla post-process to give classification score fails to fully unleash the model's full potential. Instead, the classifi- cation predictions should align with the classification methodology of the teacher. For instance, CLIP (Radford et al., 2021) generates zero-shot class predictions based on similarity scores. These holistic operations collectively complete the cycle of our cyclic knowledge transfer for OVD.
Scaling to stronger teacher mod- els. To evaluate the effectiveness of our knowledge transfer strategy, keep the backbone parameters at a moderate level while scaling with stronger teacher models. As shown in Table and Figure 4, our method can consistently improve with more robust teacher models. This scaling effect is manifested in
SWIMB
46
RN50
BM90X4
CCKT-DET (ours)
CCKT-DFT+ (ours) 1
DITO
CORA
42
RegionCLIP
COCO Novel AP
CLIP-Self
40
RRNSD
RN50X4
VTL
RN55
38
VIT-B-16
MiT
OV-DQUO
two ways: 1) teachers trained on
larger datasets with more param-
36
RN5D
eters have a more aligned visual- semantic space from which our re- gional contrastive loss can trans- fer to the detector; 2) in the infer- ence phase, employ more ro- bust teachers to provide more ac- curate semantic priors that dynam- ically transfer semantic knowledge to the detector. Notably, other ap- proaches (Wu et al., 2023c; Kuo al., 2022) often directly scale the backbone which is the teacher model itself, resulting substan- tial computational burden (about 3 x more training time), as evidenced by models like ResNet50x4 (87M parameters) ResNet50x64 (420M parameters) used in F-VLM (Kuo et al., 2022), ViT-L/14 (307M parameters) and EVA02-L (304M parameters) in CoDet (Ma et al., 2024). Our method keeps student with moderate size but achieves comparative performance by distilling from stronger teach- ers and more reliable prior information. Our method also shows faster convergence compared to the standard setting used Def-DETR (Zhu et al., 2020) and OV-DETR (Zang et al., 2022).
34
VIT-5-16
32
RR50
VIT-B-16 VIT-B-1G
30
50
100
150
200
250
300
Parameters detector backbone (MB)
Figure 4: While most models tend to enhance performance with stronger backbones, typically resulting in inereased computational demands, method achieves competitive results utilizing the de- fault ResNet50 backbone.
Number semantic priors. As discussed Sec. 3.2, during inference, we leverage CLIP to resolve the concept existence problem, compensating for the absence of prior information available during training for each image. Specifically, we introduce a hyperparameter maximum number of concepts exceed a threshold p. If the number present concepts identified is less than L, the remaining concepts randomly sampled from I. until I is met. However, this process may introduce noisy
to represent the image may contain. Concepts are identified only if their scores

---

Published as a conference paper at ICLR 2025
CIAOTITO
Figure 5: More visualization results of CCKT-Dett++
Baseline
Our CCKT CCKT-DDet Det+++
Figure Visualization results of CCKT-Det++ compared with the baseline model.
concepts into the model as shown in Table 6. This motivates to resort to stronger MLLMs that are good at multi-class identification like GVT (Wang et al., 2023) to provide more reliable prior.
4.3 VISUALIZATION RESULTS
In Figure we visualize more prediction results of CCKT-Det++ on images with novel objects. is evident that our method achieves accurate object detection results, with all these images encom- passing both base categories and novel categories. Figure 6, we present object detection results of CCKT-Det++ alongside baseline model Def-DETR (Zhu et al., 2020) (i.e., corresponds to line in Table 4). the first line and the last line, CCKT-Det++ correctly detects novel objects "dog" and "car". In the second line and the third line, the bascline model incorrectly labels "skis" as "snowboard" and "bed" as "cake", while CCKT-Det++ correctly detected those novel objects.
CONCLUSION
We propose leverage semantic priors as guidance and regional contrastive knowledge distilla- tion loss that cyclicly and dynamically transfer knowledge from both text and image encoder of VLMs. Unlike previous state-of-the-are that use extra supervisions, we exploit the regional struc- ture embedded in the well-aligned visual-semantic space of VLMs without any additional data. The performance scales consistently as the teacher model is stronger while keeping the backbone param- eters moderate level. We hope findings could inspire the community to further explore the hidden space of VLMs and MLLMs for downstream perception tasks. the future, we will explore more efficient and lightweight approaches for open-vocabulary object detection.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
The authors would like to thank all anonymous reviewers for their positive comments and constructive suggestions. This work was partially supported by the Hong Kong SAR RGC General Research Fund under Grant 16208823 and ACCESS AI Chip Cen-
ter for Emerging Smart Systems, sponsored by InnoHK funding, Hong Kong SAR. Long Chen was supported by the Hong Kong SAR RGC Early Career Scheme (26208924), the National Natural Science Foundation of China Young Scholar Fund (62402408), and the HKUST Sports Science and Technology Research Grant (SSTRG24EG04),
REFERENCES
Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa. and Ajay Divakaran. Zero-shot object detection. In Proceedings of the European conference computer vision (ECCV), pp- 380,400, 2018. 1,3
Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision (ECCV), pp. 213-229, 2020. 3, 6
Xinlei Chen, Hao Fang, Tsung-Yr Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv, 2015.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum
contrastive learning. arXiv, 2020. 2
Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnb, Paul Hongsuck Seo, and Seungryong
Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4113-4123, 2024.
Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition (CVPR), pp. 1601-1610, 2021.
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14084-14093 2022.
Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptdet: Towards open-vocabulary detection using uncurated images. In Euro- pean Conference on Computer Vision (ECCV), pp. 701-717, 2022.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv, 2022. 5
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv, 2021. 3, 6, 7,8
Agrim Gupta, Piotr Dollar, and Ross Girshick, Lvis: A dataset for large vocabulary instance segmen- tation. In Proceedings of the IEFE/CVE Conference Computer Vision and Pattern Recognition (CVPR), pp. 5356-5364, 2019. 2, 7,
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729-97338 2020. 2
Olivier J Hénaff, Skanda Koppula, Jean- -aptiste Alayrac, Aaron Van den Oord, Oriol Vinyals, and Joao Carreira. Efficient visual pretraining with contrastive detection. In Proceedings of the IEFF/CVF International Conference Computer Vision (ICC)) pp. 10086-10096, 2021.

---

Published as a conference paper at ICLR 2025
Dahun Kim, Anelia Angelova, and Weicheng Kuo. Contrastive feature masking vecabulayy vi- sion transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15602-15612, 2023a.
Dahun Kim, Anelia Angelova, and Weicheng Kuo. Detection-oriented image-text pretraining for open-vocabulary detection. arXiv, 2023b. 3
Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region-centric image-language pretraining for open-vocabulary detection, 2024. 3, 7,7
Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. F-vim: Open-vocabulary object detection upon frozen vision and language models. arXiv, 2022. 3, 3,7,8.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-benche Bench- marking multimodal Ilms with generative comprehension. arXivv 2023a. 3
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre-training with frozen image encoders and large language models. In International conference on machine learning (ICML), pp. 19730-19742, 2023b.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of
the European conference on computer vision (ECCV), pp. 740-755, 2014. 2, 8
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017. 6
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in neural information processing systems (NeurIPS), volume 36, 2024.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang. Jie Yang. Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre- pre-training for open-set object detection. arXiv preprint Sttt::2033.05999 2023.
Chuofan Ma, Yi Jiang, Xin Wen, Zehuan Yuan, and Xiaojuan Qi. Codet: Co-occurrence guided region-word alignment for open-vocabulary object detection. In Advances in neural information processing systems (NeurIPS), 2024. 6, 7,9
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning (ICML), pp. 8748 8763, 2021. 1,3,4, 4,9
Shaoqing Ren. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv, 2015.
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 658-666, 2019. 1,6
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfrr Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the JEEF/CVF conference on computer vision and pattern recognition (CVPR), pp- 22500-22510, 2023. 5
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference computer vision (ICCV), pp. 8430-8439, 2019. 1, 2,
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 2565, 2018.
12

---

Published as a conference paper at ICLR 2025
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv, 2019. 6
Guangzhi Wang, Yixiao Ge, Xiaohan Ding, Mohan Kankanhalli, and Ying Shan. What makes for good visual tokenizers for large language models? arXiv, 2023. 1, 4, 7, 10
Junjie Wang, Bin Chen, Bin Kang, Yulin Li, YiChi Chen, Weizhi Xian, and Huifeng Chang. Ov- dquo: Open-vocabulary detr with denoising text query training and open-world unknown objects supervision. arXiv, 2024.
Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin. Aligning pretraining for detec- tion via object-level contrastive learning. Advances in Neural Information Processing Systems (NeurIPS), 34:22682-22694, 2021. 3
Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for ooca-locyyylary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15254 15264, 2023a. 4, 6, 8
Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. Clipself: Vision transformer distills itself for open-vocabulary dense prediction. arXiv, 2023b. 7
Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp- 7031-7040, 2023c. 7,9
Tete Xiao, Colorado Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10539-10548, 2021.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like ofgs-of-words, and what to do about it? In The Interna- tional Conference on Learning Representations (ICIR), 2023.
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In European Conference on Computer Vision (ECCV), pp. 106-122, 2022. 4, 7,9
Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEF/CVE Conference on Computer Vision and Patlern Recognition (CVPR), pp. 14393-14402, 2021. 1, 2, 7,8
Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, Tao Kong, and Ruihua Song. What matters in training gpt4-style language model with multimodal inputs? Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 7930-7957, 2024. 3
Heng Zhang, Qiuyu Zhao, Linyu Zheng, Hao Zeng, Zhiwei Ge, Tianhao Li, and Sulong Xu. Ex- ploring region-word alignment in built-in detector for open- vocabuaayy object detection. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16975-16984, 2024. 7
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging amm-uee--uddee with mt-bench and chatbot arena, 2023.
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region- based language- -mage pretraining. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 16793-16803, 2022. 3,7
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty- classes using image-level supervision. In European Conference on Computer Vision (ECCV), pp. 350-368, 2022. 7
13

---

Published as a conference paper at ICLR 2025
Chaoyang Zhu and Long Chen. A survey apen-vocabulary detection and segmentation: Past, present, and future. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv, 2023. 3
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv, 2020. 3, 5, 9, 10
14

---

Published as a conference paper at ICLR 2025
Tea mooder
Touder munder
enceder
T
7
Dingoral
Hungarian matcher
Standard contrastive Joss
(b) Assigning positive pair Hungarian matching
Figure 7: Illustration of how to form positive pairs.
A DETAILS ABOUT IMPLEMENTING CONTRASTIVE LOSS
In the implementation of contrastive loss, the initial step involves the construction of score matrix that quantifies the similarity between the representations of two instances within an embedding space. Following this, the info-NCE loss is computed, where the numerator consists the similarity scores derived from positive pairs. The denominator is subsequently calculated by aggregating the exponentiated similarity scores across all pairs, which includes both positive and negative instances. The loss function is defined as the negative log likelihood of the positive pair similarities relative to the total similarities across pairs. This methodology promotes the maximization of similarities for positive pairs while concurrently minimizing those for negative pairs, thereby effectively organizing the structure the embedding space.
In most contrastive learning frameworks, positive pairs typically identified by the diagonal ele- ments of the similarity matrix, as these elements reflect instances compared to themselves, represent- ing the highest similarity within batch. However, this paradigm does not align with our approach. In our context, regional-level object candidates do not inherently correspond to instance-level vi- sual features from teacher, thereby rendering the diagonal elements of the As previously discussed, we establish positive pairs through Hungarian matching. Consequently, from an implementation standpoint, we delineate positive pairs based the outcomes of Hungarian matching within the asymmetric similarity matrix.
matrix ineffectual.
15


Reviews:
[{"review_id": "f5dfGd6I8B", "reviewer": "Reviewer_6x5S", "summary": "This paper introduces CCKT-Det++, an open-vocabulary object detection framework designed to detect novel object categories without relying on extensive additional supervision. The approach leverages semantic priors as guidance and a regional contrastive knowledge distillation loss to improve the model's detection capability for novel classes. CCKT-Det++ utilizes visual-semantic embeddings from both VLMs and MLLMs to dynamically transfer knowledge from both image and text encoders, aligning the detector’s latent space with meaningful, structured knowledge.", "questions": "Most concerns are listed on above boxes. Hungarian matching based on CLIP embeddings is central to your contrastive knowledge transfer scheme, aligning regional embeddings with the teacher’s visual-semantic space. However, given that CLIP embeddings may not fully capture object-specific regional nuances, why would this Hungarian matching approach be expected to significantly improve performance?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work is easy to follow. Most of the used techniques are correct.", "weaknesses": "1. While CCKT-Det++ performs well by leveraging stronger teacher models, it is heavily reliant on the quality and alignment of these teacher models. If the teacher model has limitations or biases, these could propagate to CCKT-Det++, affecting performance and potentially introducing unintended biases.\n\n2. Although the student model maintains moderate parameters, using stronger teacher models during training and inference can introduce significant computational costs. Scaling the teacher models effectively requires access to substantial computational resources, which could limit practical applications for researchers without high-performance infrastructure.\n\n3. The approach relies on pseudo annotations, which are less accurate than human annotations. If the pseudo labels are of low quality, they could introduce noise into the training process, possibly harming model performance on more challenging or nuanced object classes.\n\n4. While semantic priors improve performance on novel objects, there is a risk that the model might over-rely on these priors, leading to reduced flexibility in identifying objects that fall outside its learned semantic scope. This dependency may affect its generalization to truly unseen classes in new environments.\n\n5. The method heavily relies on CLIP for both semantic priors and feature extraction, making it vulnerable to limitations inherent in the CLIP model. For instance, CLIP's biases in language and visual associations could impact detection accuracy and lead to incorrect classifications in culturally or contextually sensitive settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "njDetQWjH1", "reviewer": "Reviewer_B8e4", "summary": "In order to solve the problem of relying on a large amount of additional data for open vocabulary object detection, the authors propose CCKT-Det method. This method guides the learning of queries by pre-filtering and injecting semantic priors, and introduces regional contrast loss to improve the perception of queries to novel objects.", "questions": "1. Clearly state the motivation for this paper.\n2. It is necessary to give a clear definition of semantic prior and explain its difference from caption and prompts.\n3. Please explain why the performance of the base class is not shown in the experiment.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The frameworks are explicit and effectively illustrates the idea of the method.\n2. This method has an improvement in the detection performance of novel classes.", "weaknesses": "1. The problem of the authors to solve and the motivation are both ambiguous. The authors point out that some methods rely on a large amount of extra caption data, but many methods can achieve high performance without it, such as the quoted OV-DQUO.\n2. The authors introduce semantic prior, but do not clearly define the concept.\n3. Figure 3 shows that the semantic prior needs to go through the VLM's text encoder, what is the difference between the semantic prior and the caption? And what is the difference between semantic priors and the prompts that currently popular? Neither of these issues is addressed by the authors.\n4. The authors do not present the detection performance of the base class in the experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wv9t7ZCApl", "reviewer": "Reviewer_s2q1", "summary": "This paper presents CCKT-Det, a method for open-vocabulary object detection (OVD) that eliminates the need for extra supervision like image-text pairs or pseudo-labels. It leverages cyclic and dynamic knowledge transfer between language queries and visual region features from pretrained vision-language models (VLMs). The approach uses semantic priors for guiding query learning and a regional contrastive loss to enhance detection of novel objects. CCKT-Det achieves significant performance gains on the COCO benchmark, showing scalability with stronger VLMs while maintaining moderate computational requirements.", "questions": "1. Why does the Regional-Contrastive Loss in the ablation study cause drop of AP50 (from 32.6 to 31.7)?\n2. There are relatively few comparative methods on the OV-LVIS benchmark.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method is quite reasonable and can effectively transfer the visual and language capabilities of VLMs.\n2. The experiments on the OV-COCO benchmark are relatively comprehensive, showing a significant improvement in detecting novel categories.\n3. The ablation experiments are relatively thorough and comprehensive.", "weaknesses": "1. The introduction to the task setup is limited, and there is a lack of a focused, clear description and visualization of the workflow details of CCKT-Det during training and testing. Figure 2 is somewhat confusing.\n2. The AP performance of CCKT-Det on the OV-COCO benchmark is relatively poor, and its detection capability on base categories is not as strong as other methods. This may be the cost of improving detection capabilities for novel categories.\n3. The experimental results did not report the AP for the base category.\n4. There is no comparison or analysis of time complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HJR81iP4cJ", "reviewer": "Reviewer_kmme", "summary": "The paper introduces CCKT-Det, a novel framework for open-vocabulary object detection (OVD) that operates without any additional supervision beyond the base training set. The method leverages semantic priors and regional contrastive knowledge distillation to align the detector with the visual-semantic space of vision-language models (VLMs).  This paper presents semantic priors to boost the detector's capacity for novel object recognition and a unique loss function that heightens the detector's sensitivity to new objects by aligning region-level features. Based on these contributions, this work achieves start-of-the-art results on COCO benchmark.", "questions": "None", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper introduces a novel framework, CCKT-Det, for open-vocabulary object detection (OVD) that does not rely on any additional supervision beyond the base training set and achieves start-of-the-results on the COCO benchmark.\n﻿\n* Comprehensive experiments are conducted on multiple benchmarks (OV-COCO, LVIS, Objects365), demonstrating the method's effectiveness. The results show consistent performance improvements as the strength of the teacher model increases.  The paper includes thorough ablation studies to validate the effectiveness of each component of the proposed method.\n\n* The paper demonstrates that high-performance OVD can be achieved without the need for additional supervision, such as image-text pairs or pseudo annotations. This is a significant advancement, especially in scenarios where such data may not be readily available.", "weaknesses": "* It is better to add a more detailed result comparison in Tab.1 and Tab.2. For example, show the AP_Base result for COCO and AP_c and AP_f results for LVIS.\n* This method has shown promising results in detecting novel objects, but it seems not very good for detecting base objects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e8jmJ9qzIN", "reviewer": "Reviewer_TQtb", "summary": "The paper introduces Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection (CCKT-Det), a method designed to enhance the detection of novel objects in open-vocabulary settings without relying on additional supervision like image-text pairs or pseudo annotations. It leverages pretrained vision-language models (VLMs) to transfer knowledge effectively from base categories to novel ones. The key contributions include using semantic priors to guide object queries and a regional contrastive knowledge distillation mechanism to align the visual features between a student and teacher model. CCKT-Det achieves state-of-the-art performance on COCO and LVIS benchmarks, improving novel object detection without extra data, and scales efficiently with stronger VLMs while maintaining moderate computational overhead.", "questions": "- In the paper, you mention, “In contrast to previous works, where object queries are static and fixed for each image after training, we dynamically inject text embeddings {ti} as semantic priors into object queries to form the language queries for each image.” However, similar ideas have been explored in earlier works, such as GroundingDINO. It would be appropriate to supplement your references by citing these related works and discuss their similarities and differences.\n- In Table 4, the paper states that “While semantic guidance demonstrates effectiveness to a degree, its efficacy is constrained in the absence of regional contrastive training.” However, contrary to this claim, regional contrastive appears to have a detrimental effect on semantic guidance, as the ablation experiment did not demonstrate any significant effectiveness of regional contrastive training. This seems to contradict the statement in the paper, “As indicated in the third row of Table 4, the absence of this loss results in a performance decline of 5.4% AP50 on novel classes.” Based on the results, there appears to be no significant relationship between Regional-Contrastive Loss and Semantic Guiding, while it works in conjunction with Similarity Classification. This seems to be a clerical error.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is well-structured, with solid methodological rigor. The authors present comprehensive experiments and ablation studies that demonstrate the effectiveness of their proposed method. The proposed CCKT-Det framework has high significance in advancing the state of open-vocabulary object detection. By eliminating the need for additional supervision and showing competitive performance with existing methods that do require extra data, the paper addresses a key limitation in the field.", "weaknesses": "- Although you claim and emphasized CCKT-Det did not rely on the extra data, in contrary to the previous method, you use the extra MLLM to be a discriminator and generate the prior semantic guidance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)