Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection
Authors: Hanoona Abdul Rasheed
Muhammad Maaz
Muhammd Uzair Khattak
Salman Khan
Fahad Khan
Venue: NeurIPS 2022 Accept (2022)
Keywords: object detection
open vocabulary
vision-language pretraining

Abstract: Existing open-vocabulary object detectors typically enlarge their vocabulary sizes by leveraging different forms of weak supervision. This helps generalize to novel objects at inference. Two popular forms of weak-supervision used in open-vocabulary detection (OVD) include pretrained CLIP model and image-level supervision. We note that both these modes of supervision are not optimally aligned for the detection task: CLIP is trained with image-text pairs and lacks precise localization of objects while the image-level supervision has been used with heuristics that do not accurately specify local object regions. In this work, we propose to address this problem by performing object-centric alignment  of the language embeddings from the CLIP model. Furthermore, we visually ground the objects with only image-level supervision using a pseudo-labeling process that provides high-quality object proposals and helps expand the vocabulary during training. We establish a bridge between the above two object-alignment strategies via a novel weight transfer function that aggregates their complimentary strengths. In essence, the proposed model seeks to minimize the gap between object and image-centric representations in the OVD setting. On the COCO benchmark, our proposed approach achieves 36.6 AP50 on novel classes, an absolute 8.2 gain over the previous best performance. For LVIS, we surpass the state-of-the-art ViLD model by 5.0 mask AP for rare categories and 3.4 overall. Code: https://github.com/hanoonaR/object-centric-ovd.

Bridging the Gap between Object and Image-level Representations for Open-Vocabulan Detection
Hanoona Rasheed ', Muhammad Maaz",, Muhammad Uzair Khattak',
Salman Khan 12, Fahad Shahbaz Khan 13
MMohamed bin Zayed University of AI, UAE
2 AAustralian National University, Australia LLinköping University, Sweden
Abstract
Existing open-vocabulary object detectors typically enlarge their vocabulary sizes by leveraging different forms of weak supervision. This helps generalize novel objects at inference. Two popular forms of weak-supervision used in open- vocabulary detection (OVD) include pretrained CLIP model and image-level super- vision. We note that both these modes of supervision the detection task: CLIP is trained with image-text pairs and lacks precise localiza- tion of objects while the image-level supervision has been used with heuristics that do not accurately specify local object regions. In this work, we propose to address this problem by performing object-centrie eenerrcceenticccccc alignment of the language embeddings from the CLIP model. Furthermore, we visually ground the objects with only image- level supervision using pseudo-labeling process that provides high-quality object proposals and helps expand the vocabulary during training. We establish a bridge between the above object-alignment strategies via novel weight transfer func- tion that aggregates their complimentary strengths. In essence, the proposed model seeks to minimize the gap between object and image-centric representations in the OVD setting. On the COCO benchmark, our proposed approach achieves 36.6 APo0 novel classes, an absolute 8.2 gain over the previous best performance. For LVIS, we surpass the saate-of-the-eee ViLD model by 5.0 mask AP for rare categories and 3.4 overall. Code: https:// http://gtttuub.ooo wwwm/aanoooa//bbjeee. centrrc-eettrrroeeeee
not optimally aligned for
1 Introduction
Open-vocabulary detection (OVD) aims generalize beyond the limited number of base classes labeled during the training phase. The goal is to detect novel classes defined by an unbounded (open) vocabulary inference. Owing to the challenging nature of the OVD task, different forms of suek-superviiionn for novel categories are typically used, e.g., extra image-caption pairs to enlarge the vocabulary [.] image-level labels on classification datasets 2 and pretrained open-vocabulary classification models like CLIP 3.. The use of weak-supervise to enlarge the vocabulary is intuitive as the cost of annotating categoy detection datasets is monumental while the image-text/label pairs are readily available via large classification datasets or internet sources 3 5
One of the major challenges with enlarging vocabulary via image-level supervision (ILS) pretrained models learned using ILS is the inherent mis-match between region and image-level cues. For instance, pretrained CLIP embeddings used in the existing OVD models [6] 2 do not perform well in locating object regions 7 since the CLIP model is trained with full scale images. Similarly, weak supervision on images using caption descriptions or image-level labels does not convey the precise object-centric information. For label grounding in images, the recent literature explores expensive pretraining with auxiliary objectives
[ or use heuristics such as, the max-score or max-size boxes [2].
'Equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In this paper, we set out to bridge the gap between object and image-centric representations within the OVD pipeline. To this end, we propose to utilize high-quality class-agnostie and class-specifie object proposals via the pretrained multi-modal vision transformer (ViT) 8 The class-agnostic object proposals are then used to distill region-specific information in the CLIIP visual embeddings, making them suitable for local objects. Furthermore, the class-specific proposal set allows us to visually ground a larger vocabulary, thereby aiding in generalization to novel categories. Next, the final and important question is how to make visual-language (VL) mapping amenable to local object-centric information. For this purpose, we introduce a region-conditioned weight transfer process which closely ties together image and region VL mapping. In nut-shell, the proposed approach connects the image, region and language representations to generalize better to novel open ppen-vocabulary objects.
The major contributions of this work include:
We propose region-based knowledge distillation to adapt image-centric CLIP embeddings for local regions, thereby improving alignment between region and language embeddings. We show that the resulting well-aligned representations aid in improving the overall performance of our text driven OVD pipeline.
In order to visually ground weak image labels, our approach performs the high-quality object proposals from pretrained multi-modal ViTs. This helps enlarging the class vocabulary and therefore generalizes better to new object classes.
using
The above contributions mainly target the visual domain. In order to preserve the benefits of object-centric alignment in the language domain, we also propose to explicitly condition the (pseudo-labeled) image-level VL mapping on the region-level VL mapping via a novel weight transfer function. In this manner, we are the first to simultancously integrate object- centric visual and language alignment within a single architecture for OVD.
Our extensive experiments demonstrate the improved OVD capability of the proposed approach. On COCO and LVIS benchmarks, our method achieves absolute gains of 8.2 and 5.0 AP on novel and rare classes over the current SOTA methods. Further generalizability is demonstrated by our cross-dataset evaluations performed on COCO, OpenImages and Objects365, leading to consistent improvements compared to existing methods.
2 Related Work
Zero-shot Object Detection (ZSD): This setting involves detecting novel class objects at inference,
for which no visual examples are available during training. Zhu et al. [9] use semantic information with visual features to get proposals for both seen and unseen classes. Bensal et al. [10] show that
learning a good separation between background and foreground is critical in ZSD and propose to use multiple latent classes for modeling background during training. Rahman al.
propose
polarity loss to solve the ambiguity between background and unseen classes DELO generating good proposals for unseen classes by synthesizing visual features for unseen objects using a generative model. Gupta et al. [13] benefits from the contemporary cues in semantic and visual space ensuring better class separation for ZSD. Other works use additional learning signals, including unlabeled images from target domain [4] and raw textual descriptions from the internet [[5]. Although significant progress has been made on this topic [1 15 the inherent complexity of the task makes it challenging for the ZSD models to generalize well to unseen object classes
focuses on
Weakty-supervised Object Detection (WSOD): In this setting, only image-level labels are used to 18
approach object detection 16
19 20). or are used alongside the detection dataset to enlarge the detector vocabulary 21 23 Bilen el al. (24 proposed a weakly-supervised deep detection
17
network (WSDNN) that uses off-the-shelf region proposals [25] 26 and computes objectness and
recognition scores for each proposal using separate subnetworks. Cap2Det (27] operates in a similar
setting and uses raw text captions to generate pseudo-labels to guide image-level supervision. Li er
al. [28] uses iegmentaaiion-decction collaborative network (SDCN) for accurate detection under weakly- supervs- setting using only image labels. PCL (2 proposes to cluster the spatially adjacent proposals and then assign image labels to each cluster. CASD (30) argues that the detectors trained only with image-level labels are prone to detect boxes around salient objects and propose feature attention along with self-distillation to address the issue YOLO9000 31 and DLWL 32 augments
shows that using max-s proposal is an optimal choice for assigning image-level labels as does not rely on the predictions of the network being optimized and provides better signals for the novel classes
the detection training by assigning image-level labels to the max-score proposal. Detic 2
ii

---

Clar-amantic propenala
Begane Aoard
V-1 Image
ppeasi
Esccddr
Detedtien Lalaset Das
Backoooee
Pvoj Wp
Head
Weight Transfer
Function
Jursis
Cuff
Castiratton dataset Das
Backocne REN
Class apecilic propicala
Rol
Vro
Head
Wr
Tex
MLP
MLP
base
Dove
DV
An overview of our proposed object-centric framework for OVD. We pair two-stage object detector with fixed language embeddings from a pretrained visual-language (VI.) model, CLIIP 3 Our pro- posed pseudo-la labeling strategy Qpendd uses pretrained multi-modal VITs obtain high-quality class-agnostic and class-specific proposals. The overall pipeline follows stage-wise learning strategy. First, we introduce region-based knowledge distillation (RKD) to adapt image-centric CL.IP embeddings for local regions. Using the pretrained VI. image encoder as teacher model, we train the detector to induce point-wise and inter- embedding relationship alignment with our region embeddings using class-agnostic proposals from 2 Next, we utilize weakly-supervised learning framework combining instance-level labels from detection dataset and image-level labels from classification dataset which are visually grounded using Qpenns. This weak- supervision helps in enlarging the class vocabulary and generalizes the detector to novel classes. To preserve the benefits of object-centric alignment in the language domain Iearned via RKD, we explicitly condition the image-level VL mapping W on the learned region-level VL mapping WD via novel weight transfer function.
Figure 1:
We also operate in similar WSOD setting and use high-quality object proposals from pretrained multi-modal ViT 8 to enlarge detector vocabulary and generalize towards novel object categories.
Open-vocabulary Object Detection (OVD); In OVD, the objective is to detect target class objects not present in the training/base class vocabulary. A typical solution of the problem is to replace the classifier weights with text embeddings of the target vocabulary (e.g., GloVe B3 BERT [34),
CLIP (3 OVR-RCNN D uses BERT embeddings as classifier weights and proposes to use open- vocabulary captions to learn the vision-to-language mapping. It surpasses the ZSD approaches by a large margin. ViLD 6 uses pretrained CLIP (3] to distill knowledge into a two-stage object detector 33 and replaces the classifier weights with CLIP text embeddings obtained by ensembling multiple text prompts (e.g., a {category), a photo
a {category)). Gao al. [36] generate pseudo hounding-box labels using pretrained VL models for training open-vocabulary detector. All these methods use carefully designed manual prompts for generating text embeddings. DetPro [37] and PromptDet [38] replace these manual prompts with learnable tokens and achieve competitive results on novel/rare categories. However, in our work, we use fixed manual prompts and instead focus improving the object-centric representations for open-vocabulary object detection.
Object-centric Opea-Vocabulary Detection
Here, we first present a brief overview of the proposed open-vocabulary detection (OVD) framework. As discussed earlier, existing OVD methods use different forms of weak supervision that employ image-centric representations, making them less suited for the end detection task. Our proposed method aims bridge the gap between image and object-centric visual-language (VL) representations. We summarize the architectural overview of our method Fig. T The proposed design has three main elements. 1) Our region-based knowledge distillation (refer Sec. 3.2) adapts image-centric language representations to be object-centric. A VL mapping learns to align the local region representations of the detector to the language representations by distilling the detector's region representations with region representations from VL model (CLIP). 2) Given weak image-level supervision, we use labeli- from pretrained multi-mode VITs (refer Sec. (3.3) to improve generalization of the detector to novel classes. 3) For an efficient combination of the above two proposed components, we condition the VL mapping learned during the weak supervision on the VL mapping learned with region-based distillation via novel weight transfer function (refer Sec. (.4). Specifically, we follow a stage-wise learning strategy to first align the region and language embeddings using RKD, and use this distilled VL mapping for object-centric visual and language alignment in the subsequent stage.

---

3.1 Detection Pipeline: Preliminaries
In the open-vocabulary detection problem, we have access to an object detection dataset where the training set, Dact. comprises samples from the set of base object categories, Cg. The images of Dact are exhaustively annotated with bounding- -bx labels and corresponding class labels yr E CB. for the different objects in the image. Given an image 1 E TR. # x3, we design an open-vocabulary object detector to solve two subsequent problems: (1) effectively localize all objects in the image, (2) classify the detected region into one of the class label of Gest. which is provided by the user at test time. The categories during test time also include novel categories CN beyond the closed set of base categories seen during the training phase, i.e., Crest = Cp UC.
We convert a generic two-stage object detector B5 to an open- vocabulary detector by replacing the learnable classifier head with fixed language embeddings, T corresponding to the category names of Cust, that are obtained using large-scale pretrained VL model. Following [6], we use the text embeddings from CLIP text encoder [3] for classification, where only the embeddings of C3 categories, Tc, are used during training. Specifically, we generate the text embeddings offline, by processing the prompts corresponding to each category with a template of photo of {category)' through the CLIP text encoder. The Rol [35] head computes pooled feature representations Φ(r) of the proposals r generated by the region proposal network (RPN). These feature embeddings are projected to a common feature space shared by the text embedding T using linear layer f(.), which we represent as region embeddings, R = f(((r)) E IR For classification, we compute the cosine similarity between the region embeddings and text embeddings to find the matching pairs. During training, the regions that do not match with any of the ground ground-truths are assigned to the background category represented by fixed all zero embedding. We compute the cosine similarity by comparing each region to each base class, V = sim(r, b) = cos (R(r), Tb) V E Cn. The classification loss is a softmax cross-entropy (CE) where the logits are the cosine similarity scores,
1 N
8oftmax (
Cots
CoB
Vr- CB.
where 7 is the temperature, N is the total number of proposals per image, and r represents a single proposal with the ground-truth label yr.
3.2 Region-based Knowledge Distillation
In the OVD setting, we assume that f()) learns a VL mapping and aligns the output region embeddings of the detector with the corresponding CLIP text embeddings. However, the performance on novel categories not comparable what CLIP encoded embeddings would provide (refer Appendix (B] for details). We hypothesize that this performance gap is mainly due to two reasons, i) the data that has been used for training CLIP model consist of scene-centric images, making it less suitable for region classification, e.g., in our case where object-centric tightly bounded proposals are used, ii) the zero-shot generalization ability of the pair-wise trained CLIP image and text embeddings cannot be fully utilized due to the mismatch between regions representations from CLIP image encoder and our detector. Based on these insights, we propose region-based knowledge distillation (RKD).
The proposed RKD uses distillation in the detection pipeline by distilling region embeddings from high-quality class-agnostic proposals (F) obtained from a pretrained multi-modal ViT (MViT) (8) Note that we obtain both proposals using this pseudo-labeling process, which we refer to as 2pscudo- This is possible via using intuitive text queries to interact with the MVIT model that can locate generic objects and provides the corresponding set of candidate proposals. The queries can be generic or targeted, based on the task, e.g., 'all objects' to generate class-agnostic proposals, or 'every dog' for specific class.
(used in RKD) and class-specific (refer Sec. 3.3) object
For RKD, we compute class agnostic proposals on Dies using simple text query, 'all objects' and select top-K proposals (Fig. (3p). CLIP embeddings I(r) are then computed offline using the CLIP image encoder I(-). With the detector region embeddings and the corresponding CLIP region representations, we propose to use two types of distillation losses to improve the alignment.
(1) Point-wise embedding matching loss: The L1 loss matches the individual region embeddings R f(o(F)) with the CLIP region representations I(F).
1 Σ L = K
|| R - I(F) lh
(1)
iv

---

CLIP
Supervised (base)
RKD
Weight-transfer
Figure 2: Top-row: Similarity matrices computed on the CLIP (Sr) and detector (Sx) region embeddings for COCO novel classes. subset 100 randomly selected samples per category form batch represented by a column are grouped together. Our region-based distillation enforces the similarity patterns in the RKD model to be closer to the teacher model, CL.P, indicated by the bright colors along diagonals. Bottom-row: t-SNE plots of CLIP and detector region embeddings novel COCO categories. The CLIP aligned RKD and weight transfer detector embeddings shows improved separability among novel class features as compared the supervised detector region embeddings (figure best viewed in-zoom).
Using this criteria, our visual encoder, along with the VL projection layer f(-), approximates the CLIP image encoder and consequently aligns our region embeddings with the CLIP text embeddings.
(2) Inter-embedding relationship matching loss (IRM): It is a knowledge distillation based loss firm that instills inter-embedding relationships within our region representations be consistent to the CLIP region representations (39]. Instilling such inter-embedding relations would be beneficial as we know that the teacher model Z()), and the student model (our detector), are different in nature with respect their training methods (Fig. 2). The IRM loss is defined on pairwise similarity matrices of the two different sets embeddings. Specifically, with the top-K proposals computed from @pscudo- we compose K x K similarity matrices for I(F) and R denoted by S₁ and SR respectively. Notably, these matrices are normalized by L2 norm applied row-wise. The IRM loss is Frobenius norm || ||r, over the mean element-wise squared difference between Sz and Sp.
R. RT
Sz = I(P) I(F)T
SR
R
RT
112
I(f) I(F) 112'
firm
SR Sz ll
K
We weight the L1 and firm losses by factors B1 and 82, respectively. Together with the standard two- stage detector losses; RPN loss (Lrpa). regression loss (Freg) and classification loss (Lats) 35 40 the overall training objective with RKD be expressed as,
LRKD = Erpn + frey + Lats + B1 L1 + B2 Lirm-
3.3 Image-level Supervision with Pseudo Box Labels
In the open-vocabulary setting, a fundamental challenge to generalize the detector to novel classes. However, due to the daunting task of densely locating all objects in natural scenes, the existing detection datasets are of relatively smaller magnitude compared to the classification datasets, which easier to annotate. To this end, Zhou et al. 2 proposed to take advantage of large-scale image classification dataset during training to expand the detector's vocabulary. However, an important question is how effectively associate the region proposals of novel objects with the corresponding labels. We note that the existing approach uses heuristics such as selecting the whole image as a single box, or just the maximum sized box from the RPN, which can ignore potential objects (Fig. 3p).
We propose a weakly-supervised method to generalize the detector to novel categories by using pseudo-box labels from pretrained MViT [8). We follow 2 to train the detector with combination

---

(a) Class-specific Proposals
(b) Class-agnostic Proposals
Figure 3: (a) Class-specific Proposals: A visual comparison of heuristic methods (left) used for visual grounding in image-level supervision 2 with our proposed method (right). Using heuristic based approaches like selecting maximum sized box from the RPN can ignore local objects in the scene. In our method, we design class-specific text queries with known class labels for pseudo-labeling potential objects. (b) Class-agnostic Proposals: In region-based knowledge distillation (RKD). we induce better region-level alignment with fewer high-quality proposals from generalized class-agnostic proposal generator (8] We compare top-K RPN proposals (left) with top-K K multi-modal ViTs proposals used a class-agnostic manner (right).
of detection and classification dataset. A batch of data is prepared by combining data from the detection dataset Daet that exhaustively annotated with bounding-box box and class labels, with data from a classification dataset Das that only contains image-level labels. With Qpscudo. we obtain the pscudo-box labels on this classification dataset, which we use for image-level supervision (ILS). Specifically, consider a sample image I E Dets, which has a total of N ground-truth class labels, we generate object proposals offline with the of MVIT corresponding to these weak labels. Specifically, we construct N class-specific text queries {f., J with template 'every {category)', and obtain K proposals (Fx ] and corresponding confidence scores {
K for each query,
[r $1), (72, 52), (rk: sk]] (1, E Dals E N.
We select the top-1 proposal with the highest confidence score, as the pseudo-box label for a particular category. This gives us N high-quality pseudo-box labels for each image, corresponding to its N image-level category labels (Fig. 3d.. We compute the region embeddings R. for proposals F as,
Rr, f()(f))) i argmax ($i).
In the case of Datt: the training follows the standard two-stage RCNN training recipe. However, for Dels, only the classification loss is updated. We call this pseudo-max score, Lpms loss.
1 Σ BCE(V, y+), where V cos (R,, T).
Cym
(4)
We weight Cpms by factor and the overall training objective with our ILS can be expressed as,
CILS CILS-
Erpn + Lreg + Lets, Lyms:
if TE Daa
(5)
if 1 E Das.
3.4 Weight Transfer Function
To combine the alignment from region-based distillation (Sec. 3.2
with the benefits from weak supervision with pseudo-box labels (Sec. 3.3 a naive approach would be to train the detector with combination of losses: Li (1). firm 2 and Lyms 4 However, demonstrate that a simple combination of the two approaches does not lead to complimentary benefits, instead they compete with each other (Table 2). The additional supervision from pseudo-labels improves the generalization of the detector, while the region-based distillation works towards object-centric alignment in the language domain, thereby improving the overall performance of the detector. We aim to incorporate the benefits from the two approaches and preserve the object-centric alignment in the language domain. To this end, we use a weight transfer mechanism [4] from VL projection used in region- based distillation to the weak supervision by learning a weight transfer function. W- In other words, the VL projection function f(-) used during the weak image-level supervision is explicitly conditioned on the mapping function used for alignment in the distillation process. This way, both the transformations are tied together to reinforce mutual representation capability and avoid any conflict in the learned function mapping. Let the weights of the projection layer in RKD and weak
vi

---

image-level supervision be represented as Wp and Wp respectively. The weight transfer operation is given by,
W WT(WD) (Wo, PWo, Wo)):
WT: WD Wr-
Here, WD is kept frozen and we design Wy as a 2-layer MLP, Wo, followed by Wo, with LeakyReLU (p) activation with negative slope of 0.1. Further, we use skip connection across Wp by projecting the original representations using separate 2-layer MLP (Fig D. The total loss here
combination of LRKD (Eq. 33 and LILS (Eq. 5 loss, given by,
rpm + Lreg + fiels + 31 L1 + 82 firm + a Pmms-
Experiments
4.1
Datasets
We conduct our experiments on COCO 42 and LVIS v1.0 43 under OVD set- ting- For evaluation, we use the gener- alized ZSD setting where the classifier contains both base and novel categories. summariissss all the datasets used in work. Following 12, D. we use a subset of ImageNet-21K K having 997 overlapping LVIS categories and COCO captions dataset for ILS in LVIS and COCO experiments respectively (refer Appendix. for more details). For the pseudo-labeling process Opeendo, we use the MVīT pretrained on a Large-scale Modulated Detection (L.DDet) dataset [8] We ensure that MVTT pretraining dataset has no overlap with any of the evaluation datasets in our work. Additionally, in all our experiments we use pretrained MViT that we train using the author's provided code on filtered LMDet ($LMDet) dataset by entirely restricting any exposure to the novel/rare classes in evaluation.
Datasct
Datasct Type
Task
images
COCO
Detection Detection
OVD
118K
LVIS v1.0
OVD
100K
ImageNet-21I" COCO-Captions
Classification
IIS in LVIS
1.4M
Image-capen saage-captioning
IIS in COCO
118K
Flick330, GQA & Visual Genome
MVIT Pretraining
LMDet
L.M
LMDet
MVIT
LMDet
(excluding any overlap with novel categories)
Pretraining
0.8M
Table 1: Summary of the datasets used in our experiments.
COCO OVD: We use COCO-2017 dataset for training and validation. We follow the ZS splits proposed in [10], in which 48 categories are selected as base and 17 are selected as novel classes.
LVIS OVD: LVIS contains 1203 categories which are further split into frequent, common and rare categories. Inline with 6 2). we combine the frequent and common categories to form base classes and keep all rare classes as novel, resulting in 866 base and 337 rare classes.
Cross-transfer Datasets: To validate the adaptability of method, evaluate and compare results of our LVIS trained model on OpenImages 44] and Objects365 [45] and COCO 42] datasets.
4.2 Implementation details
We conduct COCO experiments using Faster R-CNN (35 with ResNet-50 backbone. We train the supervised-bass model on 48 base classes (CB) for 1x schedule (~~12 COCO epochs) and report box AP50- For RKD, finetune this model for another 1x schedule using box labels from Cp and class-agiostic proposals from the pretrained MVIT [8] This model is further finetuned for 1x schedule with ILS and the associated weight transfer function using class labels from COCO captions and corresponding class-specific proposals from MVIT. This sums to an overall 3x training schedule.
For LVIS experiments, we use Mask R-CNN 40 with federated loss [46 and sigmoid cross-entropy, and report mask AP. For RKD and weight transfer, we use the same training schodules as of COCO and report the average over three runs. For comparison with Detic (2). apply our proposed method on their strong CenterNetV2 [46] baseline under the same settings. It uses ImangeNet2 K pretrained backbone with 4x schedule using large scale jittering (LSJ) 47 augmentations. All of our models are trained using 8 A 100 GPUs with an approximate training time of and 6 hours for 1x schedule of COCO and LVIS respectively.
In our experiments, we use SGD optimizer with weight decay of 1 and momentum 0.9. We train for 1x schedule with batch size of 16 and initial learning rate of 0.02 which drops by factor of 10 at the 8th and th epoch. We set temperature 7 to 50. Our longer schedules experiments use 100- 1280 LSJ [47]. We use of 0.1 to weight Lpms- For computing CLIP embeddings use the
vii

---

CLIP model ViT-B/32 B] with imput size of 224x224, We use the query 'a photo of {category)'
for to compute the text embeddings for the classifier. For distillation, we use top 5 proposals from the pretrained MViT g evaluated with generic query, 'all objects', generating class-agnostic proposals. We refer to Appendix for additional details on the approach we use to generate class-agnostic and class-specific proposals from MVĪT. In COCO experiments, we set weights 31 and (92 to 0.15. In LVIS, we set 31 to 0.15 and 32 to 0.25, We choose these values using a randomized hyper-parametes search on the corresponding held-out datasets. The 2-layer MLP in our weight transfer function has hidden dim of 512, and a hidden dim of 1024 is used in the MLP skip connection across Wp in Fig. (refer Appendix C for more details).
4.3 Our Approach: Main results
Table shows the contribution of individual components in our proposed approach. Building on top of the supervised-base model, our region-based knowledge distillation (RKD) shows an absolute gain of 19.5 and 1.5 AP for COCO novel and base classes respectively, indicating the adaptability of image-centric CLIP embeddings for local regions. With pseudo-box labeled weak image-level supervision (PIS), novel class AP improves by 28.7, demonstrating generalization to novel classes and thus enlarging the detector's vocabulary. Naively combining the two approaches shows improvement, but struggles to maintain the gains from the individual components. In contrast, our weight transfer method suitably combines the complimentary benefits of both components (Fig. (2. achieving 36.6 AP on novel classes while maintaining performance on base classes.
Method
AP APnoval noval
AP base
AP
Table 2:
Effect of individual components in our method. Our weight transfer method provides complimentary gains from RKD and ILS, achieving superior re- sults as compared to naively adding both components.
1: Supervised (Base)
1.7
53.2
39.6
2: Base + Region based ditillation (RKD)
21.2
54.7
45.9
3: Basc + ILS with pscudo-box (PIS)
30.4
52.6
46.8
4: RKD PIS
31.5
52.8
47.2
5: RKD PIS Weight-transfer (Ours)
36.6
54.0
49.4
Open-vocabulary Detection COCO:
We compare our OVD results with previously established methods in Table 3 OVR-CNN learns a vision-to-languaae mapping with expensive pretraining. Detic uses ILS to improve detection on novel classes. We use novel weight transfer function to perform object-centric VL alignment and achieve 54.0 AP on the base classes, surpassing OVR-CNN and Detic by 8.0 AP and 0.2 AP respectively. On novel classes our method achieves 36.6 AP, the highest novel AP achieved over all methods. In comparison with ViLD, which trains for 8x schedule (~~ 96 epochs), our method with the same schedule provides 56.6 base AP, lagging by 2.9.
Method
Supervision
AP base
AP Aovovll
AP
WSDDN
19.6
19.7
19.6
image-level labels for Cp U UN
Cap2Dets 2
20.1
20.3
20.1
pretraining with captions Cp Cx
OVR CWN
46.0
22.8
39.9
box-level labels in Cs
internet sourced image-texi pairs box-level labels in Cp
ViLD+ 6
59.5
27.6
51.3
internet sourced image-text pairs
RegionCLIIP
pretraining with pscudo box-level labels
54.8
26.8
47.5
box-level labels in Cp
internet sourced image-text pairs image-level labels for Cp CN
Detie 2
47.1
27.8
45.0
Detic
53.8
28.4
47.2
box-level labels in Cp
internet sourced image-texi pairs image-level labels for Cy Uxx pseudo-box labels in Cx, box-leve labels Co
Ours Ours f
54.0
36.6
49.4
56.6
36.9
51.5
Table OVD results on COCO. Here Cg and CN represents the base and novel classes respectively. Thhe results quoted from M. VViLD and our methods are trained for longer 8x schedule (shown in gray). &We train detic for another 1x for fair comparison with our method. For ViL.D, we use their unified model that trains ViLD-text and Imaee together. For Detic, we report their best model.
viii

---

On novel classes, we achieve 36.9 AP surpassing ViLD by gain of 9.3. In contrast to ViLD design, our weight transfer function allows both RKD and ILS to provide complimentary gains without any negative competition among the two methods 6
Open-vocabulary Detection LVIS: Table 4 (left) compares our results with ViLD 6 on LVIS benchmark. With 3x training schedule (~ 36 epochs) we perform reasonably well compared to ViLD 32x schedule (~ 384 epochs), already surpassing the rare AP by 1.0 while having slightly lower performance on frequent classes. Extending our model to 8x schedule fills the gap, surpassing ViLD by 0.8 in frequent and 5.0 AP in rare classes respectively. In Table 4 (right), we compare our method with Detic by using their strong LVIS baseline that uses CenterNet netrror Following similar settings, we finetune their sox-eeiee model using our weight transfer method and show improvements.
network.
Method
Epochs
AP,
AP.
AP,
AP
Method
AP,
APa
APf
AP
ViLD
6
384
16.1
20.0
28.3
22.5
Box-supervised 2
16.3
31.0
35.4
30.0
Detic (Image Captions)
24.6
32.5
35.6
32.4
Ours
36
17.1
21.4
26.7
22.8
Ours
96
21.1
25.0
29.1
25.9
Ours
25.2
33.4
35.8
32.9
Table 4: OVD results on LVIS. (Left): Comparison with prior work ViLD, using their unified model (ViLD-text + ViLD-Image), show improvement across novel and base categories. (Right): We show the comparison with Detic, by building on their strong LVIS baseline using CenterNetV2 detector
Striet Open-vocabukan Setting: Inspired from Detie, we define our work under the weakly-supervisse open- vocabulary setting uses image-level labels for expand- ing the detector's vocabulary. However in this setting, the complete target vocabulary selected number of novel and base categories are used for ILS from ImageNet-21 in LVIS. To evaluate our model in an extensive open-vocabulary setting, we modify our ILS by considering a larger vocabulary. Specifically, we expand the vocabulary to five times its size in (2). by applying ILS from randomly sampled 5K categories from ImageNet-2Ik, in addition to the LVIS base classes. compares our strict OVD setting results with ViLD where our performance slightly degrades showing sensitivity to ILS. However, we expect gain with longer training as in Table In addition to above two settings, we train our LVIS model under stricter OVD conditions in non weakly-supervisee setting by only using LVIS base categories for ILS. We achieve an overall 21.71 AP which is close to the model trained using ILS from 997 categories (22.75 AP).
Method
Epochs
AP,
AP.
AP, AP
ViLD 6
384
16.1 20.0 28.3 22.5
is unknown, i.e., only a
Ours
36
16.0 20.2 26.3 21.8
Table 5: Performance on LVIS bench- mark using a strict OVD setting.
evaluation performance: We pro- vide cross-dataset evaluation of our model in Table and compare with prior OVD works. ViLD-text 6 and basec- 2 are box-supervised baseline mod- els for ViLD and Detic respectively. Our method builds on top of Detic-base and shows favourable results when directly transferred to cross-datasets without any -ataset-specific finetuning. We use our method trained on LVIS and report AP50 on COCO 42), OpenImages 44 and Objects365 (45).
Method
COCO OpenImages Objects365
ViLD-text
43.4
11.1
Detic-baser
55.3
37.4
19.2
ViLD
55.6
18.2
Detict
56.3
42.2
21.7
Ours
56.6
42.9
22.3
Table 6: Cross-dataset evaluation. +The re- sults evaluated using official implementation.
4.4 Analysis of RKD and ILS
Effect of Ragion-based Knowledge Distillation (RKD): We ablate the effect of C1 (Eq. I and
Cirm (Eq. 2 RKD approach on COCO (Table 77. The results show the importance of both loss functions, where using C1 loss over base model with top-5 proposals from MViT (8] improves the base and novel class by 1.9 and 15.0 AP (row-1 vs 3). Using Lirm in row-4 further improves the overall and novel class AP. To show the importance of using quality proposals in RKD, we compare the model trained with L, loss using top-5 RPN vs MVIT proposals (row-2 vs 3). All the models in rows 2-4 finetuned the base model.
Efffect of Weak Image-level Supervision (ILS): We compare different choices of ILSS in Table Our Cpms loss (Eq. is compared with previously adopted ILS approaches [1 32 2 (rows 2-3). In
ix

---

Method
AP novel
AP ham
AP
Method
novet
base
AP
1: Supervised (Base)
1.7
53.2
39.6
1: Supervised (Base)
1.7
53.2
39.6
2: Max-Score loss on RPN
15.9
48.2
39.7
2: RPN proposals L1 loss
4.0
54.9
41.6
Max-Size loss RPN
25.9
51.1
44.5
3: MVIT prop L1 loss
16,7
55.1
45.0
Max-Size of MVIT Pscudo-box on MViT
50.7
4: Li IRM loss
21.2
54.7
45.9
30.4
52.6
46.8
Table 7: Analysis on our region-based KD.
Table 8: Analysis on our weak IL supervision.
row-4, we generate class-agnostic object proposals using 'all objects' text query with multi-modal ViT's (MViTs) S and select max-size proposal for ILS. In row-5, our proposed ILS approach uses target specific "every {category]' text query with MVIT and selects top-1 proposal for each ILS category. Our method (row-S) shows better performance compared to other alternatives. Additionally, we present ablations on LVIS dataset in Appendix C
5 Qualitative Results
COCO
(b) LVIS
Figure 4: Qualitative results on (a) COCO and (b) LVIS images. For COCO, base and novel categories shown in purple and green colors respectively.
(a) Objects365
(b) OpenImages
Figure 5: Qualitative results of cross-dataset transfer of our LVIS OVD model on (a) Objects365 and (b) OpenImages. Without any finetuning, our method provides high-quality detections.
Conclusion
This paper develops a novel framework to leverage the representation and generalization capability of pre-trained multi-modal models towards improved open- vocauuocabulary detection (OVD). Specifically, we note that the existing OVD methods use weak supervision modes that are more image-centric, rather than object-centric for the end detection task. We proposed novel knowledge distillation approach together with object-level aseudo-labeling to promote region-wise alignment between visual and language representations. Our weight transfer module provide an integration mechanism to combine the benefits of knowledge distillation and object-level pseudo-labeling We demonstrate encouraging results four popular OVD benchmarks, demonstrating sound generalization ability.
Acknowledgements:: The computations were performed in the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.
X

---

References
[1] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[2] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krähenbühl, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. arXiv preprint arXiv:2201.0200 1,06005, 2022.
[3] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabricl Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International Conference Machine Learning, 2021.
[4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition. lece, 2009.
[5] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning. PMLR, 2021.
[6] Xiuye Gu. Tsung-Y Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. International Conference on Learning Representations, 2022.
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Thou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region- hasod language-image pretraining. arXiv preprint 2iv:: 2021.
[8] Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, and Ming-Hsuan Yang. Multi-modal transformers excel at class-agnostic object detection. arXiv preprint arXiv:2111.11430 2021.
[9] Pengkai Zhu, Hanxiao Wang, and Venkatesh Saligrama. Zero shot detection. IEEE Trunsactions on Circuits and Systems for Video Technology, 2019.
[10] Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. In The European Conference on Computer Vision, 2018
[11] Shafin Rahman, Salman Khan, and Nick Barnes. Improved visual-semantic alignment for zero-shot object detection. In Association for the Advancement of Artificial Intelligence, 2020.
[12] Pengkai Zhu, Hanxiao Wang, and Venkatesh Saligrama. Don't even for zero-shot detection. In Proceedings of the IEEE/CVF Conference Conferene Computer Vision and Puttern Recognition, 2020.
even look once: Synthesizing features
[13] Dikshant Gupta, Aditya Anantharaman, Nehal Mamgain, Vineeth N Balasubramanian, CV Jawahar, et al. A multi-space approach to zero-shot object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020.
[14] Shafin Rahman, Salman Khan, and Nick Barnes. Transductive learning for zero-shot object detection. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[15] Zhihui Li, Lina Yao, Xiaoqin Zhang, Xianzhi Wang, Salil Kanhere, and Huaxiang Zhang, Zero-shot object detection with textual descriptions. In Association for the Advancement of Artificial Intelligence, 2019.
[16] Yunhang Shen, Rongrong Ji, Yan Wang, Zhiwei Chen, Feng Zheng, Feiyue Huang, and Yunsheng Wu. Enabling deep residual networks for weakly supervised object detection. In The Earopean Conference on Computer Vision, 2020.
[17] Yunhang Shen, Rongrong Ji, Yan Wang, Yongjian Wu, and Liujuan Cao. Cyclic guidance for weakly supervised joint detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[18] Fang Wan, Chang Liu. Wei Ke, Xiangyang Ji, Jianbin Jiao, and Qixiang Ye. C-mil: Continuation multiple instance learning for weakly supervised object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[19] Ke Yang, Dongsheng Li. and Yong Dou. Towards precise end-to to end weakly supervised object detection network. In Proceedings the IEEE/CVF International Conference Computer Vision, 2019.
xi

---

[20] Yuanyi Zhong, Jianfeng Wang. Jian Peng, and Lei Zhang. Boosting weakly supervised object detection with progressive knowledge transfer. In The European Conference on Computer Vision, 2020.
[21] Ziang Yan, Jian Liang. Weishen Pan, Jin Li, and Changshui Zhang. Weakly-and supervised object detection with expecaaion--axxiiiaatooo algorithm. arXiv preprint arXiv:1702.08744) 2017.
[22] Bowen Dong, Zitong Huang, Yuelin Guo, Qilong Wang, Zhenxing Niu, and Wangmeng Zuo. Boosting weakly supervised object detection via learning bounding box adjusters. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[23] Shijie Fang, Yuhang Cao. Xinjiang Wang, Kai Chen, Dahua Lin, and Wayne Zhang. Wssod: A new pipeline for weakly-and semi-supervised object detection. arXiv preprint arXiv:2105. 11293, 2021.
[24] Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2016.
[25] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Amold WM Smeulders. Selective scarch for object recognition. International Journal of Computer Vision, 104(2)154-171, 2013.
[26] C. Lawrence Zitnick and Piotr Dollár. Edge Boxes: Locating Object Proposals from Edges. In The European Conference on Computer Vision. Springer, 2014.
[27] Keren Ye, Mingda Zhang, Adriana Kovashka, Wei Li, Danfeng Qin, and Jesse Berent. Cap2det: Learning to amplify weak caption supervision for object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
[28] Xiaoyan Li, Meina Kan, Shiguang Shan, and Xilin Chen. Weakly supervised object detection with segmentation collaboration. In Proceedings of the IEEEJCVF International Conference on Computer Vision, 2019.
[29] Peng Tang, Xinggang Wang, Song Bai, Wei Shen, Xiang Bai, Wenyu Liu, and Alan Yuille. Pel: Proposal cluster learning for weakly supervised object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
[30] Zeyi Huang. Yang Zou, BVK Kumar, and Dong Huang. Comprehensive attention self-distillation for weakly-supervised object detection. Advances in Neural Information Processing Systems, 2020.
[31] Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2017.
[32] Vignesh Ramanathan, Rui Wang, and Dhruv Mahajan. Diwl: Improving detection for lowshot classes with weakly labelled data. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[33] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Conference on Empirical Methods in Natural Language Processing. 2014.
[34] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arxiv: 1810.04805, 2018
[35] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R- R-CNN: Towards Detection with Region Proposal Networks. Advances in Neural Information Processing Systems, 28, 2015.
Object
[36] Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong. Towards open vocabulary object detection without human-proviee provided bounding boxcs. arXiv preprint 2021.
[37] Yu Du Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. arXiv preprint arXiv:2203. 14940, 2022.
[38] Chengjian Feng, Yujie Zhong, Zequn Jic, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptdet: Expand your detector vocabulary with uncurated images. arXiv preprint arXiv:2203. 1651, 2022.
[39] Frederick Tung and Greg Mori. Similarity-presevving knowledge distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2019.
[40] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2017.
xii

---

[41] Ronghang Hu, Piotr Dollár, Kaiming He. Trevor Darrell, and Ross Girshick. Learning to segment every thing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[42] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick, Microsoft coco: Common objects context. In The European Conference on Computer Vision, 2014.
[43] Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2019.
[44] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Ka- mali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, al. The open images dataset v4. International Journal of Computer Vision, 128(6))1956-1 1981, 2020.
[45] Shuai Shao, Zeming Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
[46] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Prohabilistic two-stage detection. arXiv preprint vrXiv:2103.07461 2021.
[47] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Yi Lin, Ekin D Cubuk, Quoe V Le, and Barret Zoph. Simple copy-paste a strong data augmentation method for instance segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2021.
[48] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211 211-222, 2015.
[49] KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. Proceedings of IEEE/CVF Conference on Computer Vision and Puttern Recognition. 2021.
[50] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr- modulated detection for end-to-en multi-modal understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The abstract and introduction clearly reflects the main contributions and scope of the paper.
(b) Did you describe the limitations of your work? [Yes] We have discussed the limitations of our work. Please refer to Appendix E
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Please refer Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read the ethics review guidelines and discussed the ethical implications of our work in Appendix G
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] Our findings and propositions are mainly based on experiments and empirical results. However, we have added relevant mathematical information in our theoretical formulations.
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We provide the code along with the instructions reproduce our main experiments in the supplemental material.
xiii

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We have provided all the training details including the data splits and hyperparameter choices in our paper. Please refer to sections 4.1 and 4.2
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] Due to the limited availability of compute resources, we have not reported these statistics.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Yes we have provided these details in the main paper. Please refer the section 4.2
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets... (a) If your work uses existing assets, did you cite the creators? [Yes] We have cited all relevent existing works and assets which are related/used in our work.
(b) Did you mention the license of the assets? [Yes] We provide license details of the
assets used in our work. Please refer section H
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We provide our code for reproducing main experiments of our work in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [N/A] We use publically available datasets for experiments. We have not explicitly discussed such consent in the main paper, but we have checked and made sure that all used datasets allowed to be used for research.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] We discuss this in the supplemental material.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
xiv


Reviews:
[{"review_id": "uePzSrnxwir", "reviewer": "Reviewer_wmjL", "summary": "This paper proposes an approach for training open-vocabulary object detection models, i.e. models that can detect objects based on a list of object categories that is given at test time, potentially including novel categories not seen during training.\n\nThe paper starts from what has become the standard open-vocabulary approach: A Faster-RCNN detection architecture that is combined with a CLIP image/text model to allow open-vocabulary classification. The paper points out that the CLIP model is trained using only image-level supervision, and proposes a two-stage training approach to adapt it for the object-level detection task:\n\n**First training stage: Region-based knowledge distillation:** In this stage, a pre-trained object detector is used to predict class-agnostic boxes on some dataset. A vision/language model (CLIP) is then used to extract image embeddings for the regions defined by the predicted boxes. These embeddings are then used as distillation targets for the region embeddings of the detection model. This approach closely follows the ViLD paper, although the present paper uses a different object detection model for the pseudo-labeling. One innovation is to add an “inter-embedding relationship matching loss”, which minimizes the difference between the similarity matrices of the CLIP (teacher) embeddings and the region embeddings of the student detector.\n\n**Second training stage: Image-level supervision with pseudo-box labels:** In this stage, a pre-trained open-vocabulary detector is used to add box annotations to images from a classification dataset, by using the image-level (classification) labels as queries for the detector. These pseudo-labels are then used to train the classification loss of the detection models. This approach follows the Detic paper, but the present paper adds a new component called \"weight transfer function\". The weight transfer function conditions the model in the second stage on the embedding projection learned in the first stage, which is supposed to prevent the image-level supervision from destroying the representations learned in the first training stage.\n \nThe method is evaluated on the COCO and LVIS open-vocabulary benchmarks and compared to relevant prior work. All proposed components are ablated in experiments.\n\n", "questions": "1. See \"Weakness 1\": Provide true open-vocabulary results.\n2. See \"Weakness 2\": Clarify MViT pretraining. If box annotations for \"novel\" classes were not held out for MViT training, this may inflate results. In this case, please provide results with novel-class annotations removed from all training stages.\n3. Is the IRM loss computed for regions from single images, or all regions within a batch?\n4. \"Complimentary\" --> \"complementary\"", "limitations": "Currently, the evaluation is not truly open-vocabulary, this needs to be addressed.\n\nSocietal implications are sufficiently addressed in the appendix.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. The paper is written very clearly. Even though the proposed methods are quite complex, they are motivated and explained well. It was a pleasure to read.\n\n2. While the method primarily combines existing approaches from ViLD and DETIC, it it contributes important innovations that improve the synergy between these approaches (the weight transfer function and IRM loss).\n\n### Weaknesses\n1. In their current form, the presented results are **not open-vocabulary**, in contrast to the claims in the paper: the paper restricts the training data to the target vocabulary (_“Following [2, 1], we use a subset of ImageNet-21K having 997 overlapping LVIS categories and COCO captions dataset for ILS in LVIS and COCO experiments respectively.”_). This vocabulary restriction means that the method is in fact not open-vocabulary, but closed-vocabulary, since the target vocabularies (LVIS and COCO) must be known at training time for this filtering. Please either use a non-restricted training vocabulary that is not specifically tailored to the test data (e.g. all ImageNet-21k labels) for all main results, or explicitly and prominently state that the results are not open-vocabulary. For a method to be “open-vocabulary”, no part of the training pipeline should have knowledge of the label space used during testing. I emphasize that the fact that prior work (Detic) does the same is not a reason to repeat this practice. If knowledge of the test label space is required to reach good results, then these results are not \"open-vocabulary\", and if such restriction becomes accepted in the literature, then the term \"open-vocabulary\" becomes meaningless.\n\n2. It is not clear how the MViT model used for pseudo-box prediction is trained. Does this model see any bounding box annotations during training? Is it ensured that none of these bounding box annotations include any of the \"novel\" classes used for evaluation?\n\nThe validity of the paper critically depends on the soundness of the evaluation. If the results hold for rigorous open-vocabulary evaluation, then the work represents and significant contribution to the field.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gy9MKU7SI_Z", "reviewer": "Reviewer_dzUw", "summary": "This work aims at training open-vocabulary object detectors by leveraging CLIP and pretrained multi-modal ViT. This work propose RKD to align the region representations from the RPN and that of CLIP image encoder. In order to better make use of external images (like ImageNet) which only provide image-level annotations, this work proposes to use MViT to generate pseudo boxes and labels and use them to self-train the detector. In addition, this work proposes to adopt a weight transfer function to alleviate the competition between the two losses. The results on COCO has a significant improvement over previous works.", "questions": "1. What is the performance on MViT on COCO or LVIS in the open-vocabulary setting?\n2. Do the author have some results without the use of MViT? I would like to see a fair comparison to Detic.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n1. The technical contents of the paper are quite a lot. I can tell that the authors put many efforts on the paper. It includes improvements on both knowledge distillation side (as in ViLD) and pseudo boxes labeling side (as in Detic). It is the first paper in attempt to combine the advantages of both knowledge distillation and self-training.\n2. The experiments are throughout, including results on different dataset and the ablation study.\n3. The use of weight transfer in this task is interesting and shown to help.\n\nCons\n1. My major concern is the use of MViT that makes the comparison against ViLD and Detic unfair. Because (1) model perspective: MViT can already detect objects, so using MViT in this task is like supervising a detector using another trained detector; (2) data perspective: MViT is pretrained on LMDet dataset, which is very large with human annotations. If the authors can address this concern of mine, I may consider raising the rating.\n2. The overall pipeline is quite complex, including the three training stages and how different losses/components work together. Moreover, the hyper-parameters need to be searched for different datasets, which makes me concern about the generalization of the proposed method.\n3. It feels like the authors combines many existing methods/models in order to achieve good performance. For example, the IRM loss is just the loss proposed in Similarity-Preserving Knowledge Distillation [39]. Without IRM, the proposed RKD degrades to the knowledge distillation proposed in ViLD. Besides, the weight transfer function is proposed in [41]. \n4. The total training on COCO takes 36 epochs, which is longer than Detic.\n5. The advantage of the proposed method on large scale datasets seems insignificant. Although the performance gain on COCO is significant, but the gap between the proposed method and Detic on LVIS and Object365 is marginal (ie. less than 1AP).\n6. There are too many abbreviations in the paper, such as RKD, ILS, PIS, IRM loss, which to some extent makes the paper less easy to read.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XjwqiiC4bSd", "reviewer": "Reviewer_SdFN", "summary": "This paper proposes a method for open-vocabulary detection where the goal is to train an object detector able to detect objects in an image given a text description of the desired object category. The method is evaluated by its ability to detect novel classes for which no bounding box supervision is provided during train time. In more details, during training, one has access to a dataset containing bounding box annotation (e.g. COCO detection) along with a dataset that has only image level annotation (e.g. COCO captioning). The proposed approach relies on 3 core ideas: (1) a region-based knowledge distillation (RKD) approach which aims at enforcing that the features extracted from the detection vision backbone (the student) match the ones of a multimodal CLIP visual encoder (the teacher), (2) A pseudo labelling technique to extract candidate bounding boxes on the dataset containing image level annotation and (3) a method to enforce better combination of the two previous techniques by tying parameters. They report performance on COCO and LVIS and show stronger performance compared to previous approaches. ", "questions": "**About MViT**\n\nAs discussed above, it is unclear if using MViT is coherent with the idea of trying to generalize the method to novel classes. The authors should reply to the following questions:\n\n- Among the datasets used to train MViT, what are the ones that are annotated with bounding box annotations? Among those, are there classes that overlap with the \"novel classes\" of COCO or LVIS?\n- Is it possible to say how much MViT alone is getting on the novel classes of COCO (it seems that it should be possible to evaluate MViT on it)?\n\n\n**Method clarifications**\n\nThere are various things that are not very clear from reading the method section. Questions are listed below:\n\n- L140: what is the difference between $\\mathcal{R}$ and $\\mathcal{R}(r)$?\n- L143: how do you take the cosine similarity against the background vector since it consists of a vector of all zeros (how does the normalization works in that case)?\n- L143: related, isnt it bad to assign boxes to background even when they could be from another class which is simply not annotated in the dataset? (e.g. a COCO images could contain objects that are not in the regular 80 COCO classes and therefore not annotated). \n- L 167: How do you obtain the CLIP embedding used for RKD? Do you extract them by cropping the input images and feeding the crop to the CLIP model or do you instead extract a ROI pooled feature from a feature grid computed over the whole image? \n- Figure 2: please add the total number of categories used here. From the matrix it sounds like there are 16 categories (16 * 100 images total) but I could only count 10 colors in the TSNE plot).\n- L205: what is N in practice for ImageNet21K and COCO captions? In particular how do you obtain image-level category label from coco captions? I might have missed this but I could not find those details.\n- Equation (4): what are the negative classes used in this equation? \n- About the weight transfer function: it is unclear to me why one needs to have 2 different set of weights $W_P$ and $W_D$. The section 3.4. would benefit from better details on what exactly the architecture is and why those weights are different? The notation $f$ is used for the projection function but I found it confusing as I could not exaclty relate what $f$ precisely was with respect to the different weights $W_P$ and $W_D$. See related questions in the experiment section below.\n\n**Question about experiment choices and details**\n\n- Optimization details (beginning of section 4.2). Why does the method consists of multiple phases of training? Isn't it possible to train everything at once with all the additional losses?\n- Related to the previous question, if everything was trained together, isnt it possible in theory to completely tie together $W_P$ and $W_D$ so that to have the exact same of weights for going from the visual feature space to the joint text-image CLIP embedding space?\n- Related to the previous 2 questions, please provide details about what exactly is trained and what is frozen in the phases of training. \n- Alternative to weight transfer (Table 2): it would be good to add the following two approaches:\n    * $W_P=W_D$ by freezing it from the second phase of training \n    * Cotraining with all the losses of L 229\n\n**Minor requests/typos**\n\n- Table 6 does not bold the right numbers. Table 7 is also missing a bold number (Supervised Base on AP_base)", "limitations": "The limitations are not discussed in the main paper. It would be good to do so: are there novel classes that suffer more than others? If yes, why? What are the important pieces of the method and how robust is the method to failure in those pieces? (e.g. the MViT component etc etc). Potential negative societal impact is also not discussed despite the method potentially being trained on large scale image and text datasets. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper obtains good results on LVIS and COCO that outperform the state-of-the-art (Detic),\n- The ideas are overall well motivated and the method is technically sound \n- The ablation study is overall well conducted (see more suggestions in the Questions section) \n\n**Weaknesses**\n\n- An important aspect of the method seems to be that it relies on the MViT approach to obtain region proposals (used both in RKD with class agnostic proposals as well as for the pseudo labeling approach, see for Example Table 6 and Table 7 which shows the importance of MViT especially for the AP on novel classes. However MViT is trained on 3 datasets, some of which have bounding box annotation (GQA, VisualGenome and Flickr). The authors should discuss whether or not these datasets contain bounding box annotation for classes that are considered \"novel\" in the evaluation. If yes then I believe this would be unfair compared to previous approaches. See Questions for more details about this. \n\n- Related to the previous point, it would be good to list in Table 3, what are the datasets that are used by each methods (e.g. what is the source image-text pairs for all methods etc etc). I encourage the authors to provide this information in the rebuttal as well. This will help to better compare against past work such as Detic.\n\n- The description of the approach is not always clear (see Questions section for a detailed list of those questions). It is important that the authors clarify this during the rebuttal.\n\n- Some additional experiments could be conducted (see suggestions in Questions) \n\n=== POST REBUTTAL\n\nAfter reading the rebuttal and other reviews, I am raising my score from 5 to 6. The authors have adequately addressed the important concerns I had about the novel class results. To the best of my knowledge, I believe that the results are now legit. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UUS25ObYcl-", "reviewer": "Reviewer_LKon", "summary": "This paper proposed a training pipeline for better dealing with open-vocabulary object detection. Specifically, it incorporates CLIP as a pre-trained vision-language embedding to significantly expand the potential object detection vocabulary and mitigated performance degradation due to the gap between image-centric and object-centric representation. In this paper three important components are proposed to enhance the system, including (1) region-based knowledge distillation to match the CLIP embedding for image patch (instead of full image) with text, thus the replaced text classification head can better fit the region detection; (2) image-level supervision with pseudo box labels which benefit from a recent multi-modal ViT (MViT) class-agnostic object detector; (3) a weight transfer function which combines (1) and (2) more effectively in a trainable approach. The overall system is validated on multiple benchmarks and achieves state-of-the-art performance, especially for novel object classes. ", "questions": "1. Although the authors perform detailed ablation studies to validate each component in their system, one critical aspect is not covered which is the performance of MViT on \"novel\" object proposal/detection. Specifically, since the region proposal is generated from MViT for both class-agnostic and class-specific approaches, one possibility is that the pre-training of MViT may have some leaked information to the student model. The authors did mention that \"we ensure that MViT pretraining dataset has no overlap with any of the evaluation datasets in our work.\" but I'm curious about details about the exclusion -- does it mean the eval images are never included in the pre-training or even the novel object classes are not exposed? Probably an even more careful ablation study would be replacing RPN in the baseline object detection with MViT so the effect of the enhanced proposal can be analyzed. Essentially I'm feeling that the quality of the system is heavily dependent on the $\\mathcal{Q}_{pseudo}$ but this paper did not provide sufficient ablations about it.\n\n2. Regarding the weight transfer function, I'm curious how this is trained -- is it being trained in an end-to-end fashion (I presume) together with the full system using loss defined in L229? Does it mean $W_P$ in the weight transfer function is now replaced by $W_\\tau$ and $W_D$? Probably it's better to explicitly show this relation in the equation below L229.", "limitations": "This author does not provide explicit explanations of limitations and societal impacts but I presume this system has similar issues with other systems (e.g. potential societal bias when trained with large-scale automatic collected datasets).", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The assumption for the performance degradation from CLIP to object detection is reasonable and the authors propose a solid solution to mitigate the gap. The image-centric and object-centric nature may lead to incompatibility of the visual encoder and textual embedding, thus introducing the $L1$ and $L2$ regularization objectives is intuitively helpful to bridge the gap.\n- The weight transfer function seems interesting and shows great potential for merging two components in a more harmonized way. The performance boost is quite significant and similar ideas may inspire other ensembling/multi-component fusion systems.\n- The details of the training and evaluation are provided and the paper is easy to follow.\n\nWeaknesses:\n\n- The model did demonstrate significant improvements over other approaches, although it doesn't seem clear to me how much of the distillation from MViT is involved in these improvements. See my Questions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept