DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection
Authors: Lewei Yao
Jianhua Han
Youpeng Wen
Xiaodan Liang
Dan Xu
Wei Zhang
Zhenguo Li
Chunjing Xu
Hang Xu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Open-world object detection，Visual-language model

Abstract: Open-world object detection, as a more general and challenging goal, aims to recognize and localize objects described by arbitrary category names. The recent work GLIP formulates this problem as a grounding problem by concatenating all category names of detection datasets into sentences, which leads to inefficient interaction between category names. This paper presents DetCLIP, a paralleled visual-concept pre-training method for open-world detection by resorting to knowledge enrichment from a designed concept dictionary. To achieve better learning efficiency, we propose a novel paralleled concept formulation that extracts concepts separately to better utilize heterogeneous datasets (i.e., detection, grounding, and image-text pairs) for training. We further design a concept dictionary (with descriptions) from various online sources and detection datasets to provide prior knowledge for each concept. By enriching the concepts with their descriptions,
we explicitly build the relationships among various concepts to facilitate the open-domain learning. The proposed concept dictionary is further used to provide sufficient negative concepts for the construction of the word-region alignment loss, and to complete labels for objects with missing descriptions in captions of image-text pair data. The proposed framework demonstrates strong zero-shot detection performances, e.g., on the LVIS dataset, our DetCLIP-T outperforms GLIP-T by 9.9% mAP and obtains a 13.5% improvement on rare categories compared to the fully-supervised model with the same backbone as ours.

DetCLIP: Dictionary-Enriceed Visual-Concept
Paralleled Pre-training for Open-world Detection
Lewei Yao Jianhua Han?*, Youpeng Wen Xiaodan Liang,, Dan Xu',
Wei Zhang", Zhenguo Li,, Chunjing Xu², Hang Xu 'Hnn Kong University of Science and Technology. "Huawei Noah's Ark Lab Shenzhen Campus of Sun Yat-Sen University
Abstract
Open-world object detection, ognize and localize objects described by arbitrary category names. The recent work GLIP formulates this problem grounding problem by concatenating all category names of detection datasets into sentences, which leads to inefficient interaction between category names. This paper presents DetCLIP, paralleled visual-concept pre-training method for open-world detection by resorting to knowledge enrich- ment from designed concept dictionary. To achieve better learning efficiency, we propose a novel paralleled concept formulation that extracts concepts separately better utilize heterogeneous datasets (i.e., detection, grounding, and image-text pairs) for training. We further design concept dictionary (with descriptions) from various online sources and detection datasets to provide prior knowledge for each concept. By enriching the concepts with their descriptions, we explicitly build the relationships among various concepts facilitate the open-domain learning. The proposed concept dictionary is further used to provide sufficient negative concepts for the construction the word-region alignment loss and to complete labels for objects with missing descriptions in captions of image-text pair data. The proposed framework demonstrates strong zero-shot detection performances, c.g., on the LVIS dataset, our DetCLIP-T outperforms GLIP-T by 9.9% mAP and obtains 13.5% improvement on rare categories compared to the fully-supervised model with the same backbone as ours.
general and challenging goal, aims
Introduction
only recognize and localize pre- defined number of categories. Their detection performance greatly relies on sufficient training data for each category, which requires expensive and coneumonn human annotations, especially for the rare classes that can only be distinguished from the expert. Even though a great effort has been made, existing publicly available detection datasets only have limited number of object categories, for instance, 80 for COCO 31. 365 for Object365 4% and 1203 for LVIS [6]. However, versatile open- world object detection is still out of reach mainly for two reasons: a) Due the long-tail problem (16), finding sufficient examples for rare categories is surprisingly challenging; b) Developing a larger detection dataset requires extremely costly and labor-intensive manual annotation.
Most stae-oof-the-at object detection methods 37 39
On the other hand, image-text pair data are cheap and abundant on the Internet. Recent vision- language (VL) pre-traie traitrain methods (e.g., CLIP (36), ALIGN (26)) utilize those data to extend their open-domain capacity and have shown good zero-shot ability on various downstream classification try to extend two-stage detector open-world
tasks. Intuitively, some works [5 51 48 12
"Equal contribution, Coorespponing author: xu.hang @huawei.com
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Caresse
Training
Delecing
Rama
DeICLIP
Figure I: Task definition of open-world detection. In this paper, DetCLIP aims to develop new pre-training pipeline with datasets from different domains (including detection dataset, grounding dataset and image-text pair dataset) as input, to deal with the open-wor woeldworld detection problem.
detector by distilling the learned image embeddings of the cropped proposal regions from a pre- trained VL model. However, this paradigm requires costly feature extraction from cropped images. There also exists discrepancy between instance-level features and conventional image-level features extracted from VI. models. [14] proposes self-training pipeline with pseudo labels (i.e., noun phrases in caption) generated by a pre-trained VL model [26] while the phrases in caption are always too limited to cover all the objects in an image. Moreover, their open-domane ability determined by the performance of the pre-trained VL models.
Another line of work, GLIP (28 further proposes grounding formulation of open-world detection by directly utilizing both detection and grounding data for pre-training. Specially for detection data, GLIP takes an image and text prompt sentence that concatenates all the category names as input (i.e., sequential formulation). A text encoder will output features for this sentence and then GLIP aligns them with the region features extracted from the image encoder. However, restricted by the max length of the input token size of the text encoder, it is difficult for GLIP to operate on large number of categories or extend to more detailed description of the categories. Furthermore, full attention matrix upon all the categories has to be learned in the text encoder, which is unnecessary and low-efficient for detection especially when the size of input categories increases.
To alleviate the above problems and further improve the open-domain ability, we present DetCLIP, a dictionary-enrichedd visual-concept paralleled pre-training method for open-world detection. Note that the "concepts" denotes the category names in detection data, and the phrases in grounding and image text pair data. Specifically, we first design novel paralleled concept formulation to improve learning efficiency. Instead of feeding the whole prompt text sentence into the text encoder like GLIP, DetCLIP extracts each concept separately and parallelly feeds them into the text encoder (i.e.. paralleled formulatin))) This paralleled formulation allows the model to avoid unnecessary interaction between uncorrelated categories/phrases, and produce longer description for each concept. By converting detection data, grounding data, and image-text pair data into paralleled formulation, DeICLIP can be pre-trained under different types of supervision to support both localization and open-domain capability.
During pre-training existing datasets often have a large domain gap and difference in their labeling space, e.g., the same object concept with different names or hierarchi- cal/inclusive structures of different concepts. However, it difficult to obtain these implicit relationships among con- cepts simply by their short names. In order to form a more unified concept space and provide prior knowledge (i.e., implicit relationships) for each input concept, we propose a novel concept dictionary to enrich our prompt text con- cepts during joint pre-training, as shown in Fig Firstly, we construct the dictionary with concepts extracted from online resources and existing large-scale detection datasets by considering both commonality and coherence. Based on this concept dictionary, DetCLIP can automatically enrich the current concept with the concepts and their descriptions existing in the dictionary to facilitate open- open-domain learning. To alleviate the partial label problem for the grounding and image-text pair data, DetCLIP further randomly samples concepts from the dictionary negative samples to efficiently pre-train with the alignment loss. Besides, all concepts in the dictionary are served as the additional category inputs for label completion the image-text pair data.
noley
unified hand- held mobile telephonn.
balldter Selephumm
Concept Dicionary
cridien, Janimall
fowl
Chickon
bred
Figure 2: DetCLIP aims at unifying the noisy text name space into a joint con- cept space by novel concept dictionary.
The proposed DetCLIP outperforms the state-of-the-an GLIP by large margins on large-scale open- world detection benchmark LVIS [16] with 1203 classes. Particularly, DetCLIP-T achieves around

---

9.9% mAP improvement over GLIP-T on LVIS without utilizing any images in LVIS during pre-
training. Compared to the baseline ATSS [53] model (with the same swin-T [33 backbone) trained on L.VIS, our DeCCLIP-T achieves 2.3% mAP improvement in the total performance and 13.5% mAP improvement on the rare classes.
2 Related Work
Vision-Language Pre-training (VLP). Current Vision-Language Pre-training is a natural extension and development of the successful pre-train-and-fin--uue scheme in the domains of natural language processing (NLP) [ and computer vision [[] community. Dual-stream methods such as CLIP [36] and ALIGN [26] have shown great zero-shot classification ability by performing cross-modal contrastive learning on large-scale image-text pairs from the Internet. Single-stream approaches
27 21 directly model the interaction between the visual and textual embeddings together by a model, which can perform tasks such as image caption and VQA. Recent
single transformer- approaches such as VLMo [6) and BLIP (25] further explore mixed architecture of single-stream and dual-stream models to enable a unified way of vision-languaa language understanding and generation. However, those approaches usually focus on whole-image representation learning and the pre-trained models are usually designed for retrieval/generation tasks. Current vision-languan pre-training approaches can not directly be applied to object detection task, i.e., core computer vision task.
General Object Detection. Object detection is core problem in computer vision. CNN-based object detection methods (one-stage detectors: YOLO 377 SSD 2
and ATSS K3 and two-stage
detectors:
Faster R-CNN [39] and R-FCN (7) usually use a classifier to map ROI (Region Of Interest) features into categories. Further improvement of the CNN-based detectors such as K-Net [54] and Panoptic-FCN [29] further introduce dynamic kernels and replace the static kernels in the convolution layers improve the flexibility of models. Recent transformer-basee methods such as DETR B and Deformable DETR 55 try to formulate the object detection as set prediction problem that can eliminate post-process NMS. Those methods are constrained to predefined categories, while our method tries to endow the detector with open-domain recognition ability which can detect any categories by learning a wide range of concepts.
Zero-shot Object Detection/ Open-vocabulary Object Detection. In the early setting of this area, zero-shot object detection aims generalize the detector from known categories (training) to unknown categories (inference). Under this setting, various works [5] try to find relationships between existing categories and unknown categories through pre- pre-trained semantic/text features 49 47) and so on. However, the evaluation under this setting is not general enough since people usually simply split the class name into known/unknown categories in the single dataset and the transfer learning is still under similar domain. On the other hand, inspired by the success of saiionlanguage-(V1
44 35 38
13 knowledge graphs A1 19
pre-training methods (e.g., CLIP (36) and their good zero-shot ability, several methods attempt to perform zero-shot detection
wider range
of domains by leveraging a pre-trained VL model. For example,
[15] tries to distill the learned image embeddings of the cropped proposal regions from CLIP (36)] to student detector. 14 proposes a self-training pipeline which utilizes Grad-CAM
42 and ALBEF [26.. However, those methods
very slow because the feature extraction is repeated and the image-level representation may be sub-optimal for the instance-wise tasks. Recently, GLIP (28) and X-DETR (2) try to align region and language features using dot-product operation and can be trained end-to-end on both grounding data and detection data. Our method aims to design an open-domain detector that learns new concepts efficiently and expands domain coverage from low-cost data from the Internet.
3 The Proposed Approach
This paper aims to develop a lasion-language pre-training pipeline to enhance new concept learning for open-world detection. To achieve this goal, our DetCLIP leverages new paralleled vision- concept per-training pipeline 3.1 for efficient training and a concept dictionary (Sec to provide external knowledge automatically enrich the current input concept and alleviate the partial- labeling problem of the grounding and image-text pair data. Our DetCLIP is pre-trained under hybrid supervision from detection data, grounding data and image-text pair data (.3
3

---

(ccapppry name woword in caction
grounding data My Mig Myy Mig Mg 1g
detection data
grounding data
detection grounding
We Ma
We Wo
embedding. embeddingg..
Text Encoder
Text Encoder
Text Encoder
detection data
Logts ED C
Image Encoder
Chensfizttion Looss
Alggonen Looes
Lunea
Aigganent
(a) Detection Data
(b) Grounding Data
(c) GLIP GLIP-style (Sequential)
(d) DetCLIP-style (Paralloled)
Figure 3: Comparison of different concept formulation strategies. (a) Traditional detection and (b) grounding training utilize different input formulations, (c) To utilize both detection and grounding datasets, GLIP converts the detection data into grounding format by formulating all classes into one sentence input. (d) Our DetCLIP introduces paralleled concept formulation to extract the phrase from the grounding data and regards each phrase as an individual input that is the same as the category name in the detection dataset. Red color denotes that the corresponding concept is in the image while the green indicates the concept not in the image.
3.1 Paralleled Concept Formulation
A robust open-world detector is required to be trained with sufficient data covering enough vision concepts. Current available detection datasets lack enough concepts and have limited capacity due to the costly annotation. Leveraging data from other sources, e.g., grounding data and image-text pair data, is a feasible solution to augment the semantic coverage. To enable training with heterogeneous supervisions, we required to find unified formulation for different formats of data.
comparison of the different concept formulation strategies that are used for pre- training with different types of data. As shown in Fig 3 (b)) traditional detection and grounding training utilize different input formulations. Detection treats each category as a fixed label and aligns region features to the pre-defined label space, while the grounding training leverages the whole caption sentence as input, models the attention between words, and adopts each token embedding in the output embeddings for aligmment. To utilize both detection and grounding datasets for pre- pre-training GLIP (28] converts the detection data to phrase grounding data, by replacing the object classification logits with the word-region alignment scores, which associated with sentence that concatenates all the category names (see Fig ())) i.e., the text input is ["person, bicycle, car, toothbrush").
We argue that this sequential form is not an effective formulation to model open-world object detection as a lasion-language task because (1) leads to unnecessary interaction between category names in the attention module; and (2) constraints the number of negative samples in contrastive learning due to the limited context length of text input. Ablation studies conducted for GLIP show that randomly shuffling the word order in the grounding training data can even bring slight improvement for the downstream detection task, indicating that the noun phrase is more critical for the detection task, compared to the context information.
To address the above problem, our DetCLIP introduces a paralleled concept formulation to train with different data sources. As shown in dig we extract the concept noun phrase for each bounding box and then feed them into the text encoder individually to obtain the corresponding text embedding. In our paralleled design, context information is removed and model dircctly learns language features from each separate concepts, which is more straightforward modeling for the detection task and improves the learning paralleled design can enable easy expansion of the augment of category de- scriptions (see Sec 3.2.2). Detailed paralleled formulation for different types of data are as follows:
Training Loss (Object365)
1.00
Sequential, 23.7% MAP(LVIS))
Paralleled, 27.8% MAP(LVIS))
2.00
2:20
1.1
1.50
efficiency (see Fig 4.. Furthermore,
2000 Training Iterations
3000
Figure 4: Efficiency comparison be- tween two formulations. Paralleled for- mulation learns faster and achieves better performance (27.8% v.s. 23.7%).
Detection data: Suppose there are k positive categories in an image, we first pad the category number to N by randomly sampling the negative categories, where N is a

---

predefined number of the concepts for constructing the alignment loss. More specifically, the text input P can be formulated
where Pn, represents for the n-th category name, e.g.,
P ["person"" "bicycle", "car",..., "toothbrush"]]
Then, we feed N category names as separate sentences into the text encoder and use the embedding of [end of sentence] token as the text embedding for each category. The N text embeddings are then concatenated and matched with ground truth bounding boxes to construct the alignment loss.
Grounding data:
We extract positive phrases for bounding boxes (provided by the grounding annotations), and drop other words in the caption. To align the input format with detection data, also pad the category number to N by sampling negative categories from proposed concept dictionary (see Sec 3.2.2). An example of input P for grounding data can be:
P l'a woman", "a herding dog", "three cattle", "negi" "regi",.... "negm" "].
where negi negm are sampled negative categories from the constructed concept dictionary. Then similar steps with detection data can be conducted to construct the alignment loss.
Image-text pair data: Image-text pair data only contains images and the corresponding captions, while without any annotated bounding boxes. To obtain object-level dense labels, we first use a pre- trained class-agnostic region proposal network (RPN) to extract object proposals. Then a powerful pre-trained Langua-a model like CLIP 36 or FIL.IP (50) is utilized to assign pseudo labels for the proposals (see Appendix for more details.). To tackle the concept-missing problem in the captions of image-text pair data, instead of using noun phrases in the captions as the candidate category names following [8 14], we propose assign open-domain categories to the object proposals via constructing large-scale concept dictionary (see Sec..2.2)) After obtaining object-level dense pseudo labels, we convert the image-text pair data to the same format as the detection and grounding data, and then a similar training procedure can be applied.
3.2 Concept Dictionary
In this work, we try to build an open-world object detector that can cover a wide range of concepts and can be applicable different types of datasets. However, the existing waaatiin//gouundnnn/meeeetteee pair datasets have very large domain gap and difference in their labeling space. For example, a boy can be annotated as "man", "child" or "people" the different datasets. Moreover, there often exists hierarchical//nclusiee relationships between different concepts. Knowing these implicit relationships can effectively facilitate the pre-training, however also clearly challenging to discover these relationships with only limited set of concept names.
Therefore, we propose to build large-scale concept dictionary form a unified concept space for different data sources and explicitly provide the useful relationships between various concepts through definitions. For example, the car is defined as "a motor vehicle with four wheels usually propelled by an internal combustion engine", and the motorcycle is defined as "a motor vehicle with two wheels and a strong frame". By aggregating these definitions, we can conclude that the car and the motorcycle are both motor vehicles with a difference in the number of wheels.
3.2.1 Constructing the Concept Dictionary
To construct unified concept dictionary O, collect concepts from multiple sources: (1) noun phrases extracted from the large-scale image-text pair dataset, i.e., YFCC100m; (2) category names of existing public detection datasets (c.g.. Objects365 43). OpenImages (24): (3) object names from the manually-collected concept database, i.c.. Things 18)). For object names from the detection datasets and Things, we directly add them into the dictionary after deduplication. For noun phrases extracted from the YFCC, we filter out the concepts that appear less than frequency of 100 or without definition in WordNet 33 to ensure the commonality. Our concept dictionary constructed by concatenating each word with its definition defi in WordNet: O = for de H where is the number of concepts in our dictionary. The constructed dictionary covers about 14k concepts along with their definitions, and can be updated by directly adding new concepts and corresponding definitions. Based on the constructed concept dictionary, we then propose techniques utilizing it to boost the pre- pre-training in the following section.
is then
5

---

Pacelated
Image toxt pair Datanet
person
persoc
dei human baing.
Disionary
Concept
Concage Eenchment
Pleuse lasiling
No interittonn betwween
PP
pertobtl composer
(Labal Completion)
Text Encoder 4,
Pertol
Grounding
clack
Anrolation
Concagt Ensichment
Enritmmet
Detociion
Detection
B chidd
a chil
Image
PULYT
receves tapttpp
Pb new lepitp
laaioo
Pb
Cous
Algemeett
txtter.
Paralisted Concept Formulation Groundingghrgge
snephert #Megathvv Samples
Fanc
Carz
(b)
Figure 5: Overall architecture and the details of utilizing concept dictionary O. (a) DeCCLIP contains an image encoder D; to obtain region features I' and a text encoder 4 to get embeddings F for each enriched concept P Then the region-concept alignment loss CALI is performed. Note that the box regression loss LREG is only adopted on detection datasets. (b) A concept dictionary O is
introduced to enrich the current concept with prior knowledge and provide negative category samples for construction of the alignment loss.
3.2.2 Knowledge Enrichment with Concept Dictionary
Concept Enrichment. Based on the designed concept dictionary, we first retrieve the definition for each input concept provide the prior knowledge (see Fi) During pre-training, for each concept Ps. in the training set, we can directly use its definition if p.. is included in the dictionary O. If cannot find a direct match in O we will try locate the most related concept in O by calculating a similarity matrix S E R The S' is calculated via the dot-product of the embeddings from pre-trained text encoder such as FILIP [50] with p., and all concept names {ot 1 as input. Then we can find the most related concept (or-. where /* = ar gmaz; (S' (()) in the dictionary, to retrieve an approximate definition de fi. The Pn is then enriched with the retrieved definition and reformatted as {pr} (Pn. fe }. An example of the enriched text input p {p% J is:
P I'person, a human being.", "bicycles, wheeled vehicle that has two wheels and is moved by foot pedals.". " "toothbrush, small brush has long handle used to clean teeth."]
Partial Annotation Enrichment. In the grounding or image-text pair data, only main objects that people care about are labeled in the caption, which is known as the partial labeling problem. Compared with standard detection datasets which have sufficient positive and negative classes for each image, pre-training with grounding and image-text pair datasets encounters two severe issues; lack of annotations of negative concepts for learning discriminative concept embeddings; 2) lack of annotations of partial positive concepts to efficiently train the model. For the first problem, DetCLIP randomly samples the concepts in the constructed dictionary O as the negative concepts to construct the alignment loss, instead of directly padding empty inputs (Fig Note that since the number of concepts in the dictionary O is large (i.e., about 14k), the probability that the sampled concepts are indeed in the image extremely small. For the second problem, to perform label completion on image-text pair data during pseudo labeling, we add all the concepts in dictionary {or {or)== as the additional category inputs, instead using the original noun phrase in the caption to calculate the similarity matrix. Therefore, the concepts shown in the image while not in the caption can also be labeled and then get pre-trained. An illustration is also shown in Fig 7 to qualitatively verify the effectiveness of label completion.
3.3 Model Architecture/Traiineee
Objective
As shown in Fig 5 the basic architecture of DeICLIP contains an image encoder & to generate the region features pl E R-M-D from the input image X, and a text encoder D, to obtain the text embeddings F e IRN- for the concepts in pt, where M. N denote for the number of extracted regions and input concepts, respectively. Then the alignment loss is constructed by calculating the alignment score S e IR N . for all region-text pairs.
F' (X), F 4b,(P), (F', Transpose(F""))
With the ground- ground-truth truth alignment matrix G p N-M the whole training objective L can be written as:

---

Table I: Zero-shot performance on LVIS [16 minival datasets. AP, AP. , indicate the AP values for rare, common, frequent categories, respectively. 'DH' and 'F" GLIP [28] baselines stand for the dynamic head [8] and cross- cross- fusion, respectively. Baselines with are implemented with our code-base. GoldG+ denotes the GoldG plus the COCO [1 caption dataset.
LVIS AP, / AP. AP,
MODEL
BACKBONE
PRE-TRAIN DATA
AP
MASK-RCNN 17
SWIN-T
LVIS
34.1
/9.1/ / 37.0
ATSS
SWIN-T
LVIS
33.6
39.7/32.4/31 37.2
A3SS *
SWIN-L
LVIS
43.9
30.6 43.7 46.3
MDETR 20
RN101
GOLDG+
24.2
20.9 24.2
GLIP--(A) 28
SWIN-T+DH+F
0365
18.5
14.2/ 23.4
GLIP-T(C) 28
SWIN-T+DHH
0365, 0365,GOLDG
24.9
17.7 19.5/
GLIP-T(28)
SWIN-T+ SWIN-T+DH+F
O365, GOLDGG. GOLDG.CAP4M
CAP4M
26.0
20.8 00.8/214/31.0
GLIP-L
40Ds, GOLDG,CAP24NN
37.3
34.2/3.3/441
GLIPv2-T
SWIN-T+DH+F
G36G,GOLDG,.AAAM
29.0
DETCLIP-T(A)
SWIN-T
0365 O365, GOLDG
28.8
26.0 28.0 / 30.0
DETCLIP-T(P T(B)
SWIN-T
34.4
26.9
33.9 36.3
DETCLIP-T
SWIN-T
O365, GOLDG, YFCCIM
35.9
33.2 / 35.7
36.4
DETCLIP-L
SWIN-L
O365, GOLDG, YFCCIM
38.6
36.0 38.3 / 39.3
L= CALI + LCEN + B CREG,
where CALIS LOEN, and LRGG denote the alignment loss, the centerness loss, and the regression loss, respectively. a and 3 represent the weight factor for LCEN and LREG, respectively. Following GLIP (28), we adopt the ATSS (53) detector as image encoder. We use the sigmoid focal loss (30) for ALI, the sigmoid loss for LCEN, and the GloU loss [40] for LREE.
4 Experimental Results
33 backbones with 32 GPUs. AdamW optimizer [22] is adopted and batch size set 128. The learning rate is set to 2.8x10 for the parameters of the visual backbone and detection head, and 2.8x10 for the language backbone. Without otherwise specified, all models are trained with 12 epochs and learning rate is decayed with factor of 0.1 at the 8-th and the -t epoch. The max token length for each input sentence set to 48. The number of the concepts N in text input is set to 150 and the number of region features M is determined by the feature map size and the number of pre-defined anchors. The loss weight factors a and are both set to 1.0. The training of DeICLIP-T with Swin-T backbone tasks 63 hours when 32 GPUs are used. MMDetection 66 code-base is used.
Implementation Details. We pre-train all the models based on
Training Data. Our model is trained with hybrid supervision from different kinds of data, i.e.. detection data, grounding data, and image-text pair data. More specifically, for detection data, we use sampled 0bjects365 V2 4 dataset (denoted as 0365 in the following sections) with 0.66M training images. Here we do not use the whole dataset since training with sampled data is more efficient and suffices to demonstrate the effectiveness of our method. For grounding data, we use gold grounding data (denoted GoldG) introduced by MDETR 20) Moreover, following GLIP (28). we remove the training samples contained in LVIS [66] dataset for fair zero-transfer evaluation, which results in 0.77M training data. For image-text pair data, we perform object-level dense pscudo labeling on YFCC100m [45] dataset with a pre-trained CLIP 36 model, and sample a subset of IM training images from the results containing objects with similarities scores above given threshold. Finally, our training set contains a total of 2.43M images. Compared to GLIP's 27M training data, DetCLIP only uses less than 10% data, but achieves better results. More details are in Appendix.
Benchmark Settings. We evaluate our method mainly on LVIS [16] which contains 1203 categories. Following GLIP 28 and MDETR 200. we evaluate on the 5k minival subset and report the zero-shot
fixed AP for a fair comparison. We do not focus the performance on COCO 31 since it only contains 80 common categories that are fully covered by the training dataset Objects365 (3), which
7

---

may not sufficient to reflect generalization ability of a model in the open-domain detection setting. To further study the generalization ability of our method, following GLIP [28]. we also evaluate the
averaged AP on other 13 downstream detection datasets published on Roboflow [T
4.1 Open-world Detection Results
We train our DeICLIP with two backbones, i.e., swin-T 3 and swin-I.. Distinct from GLIP 28 which introduces additional heavy modules like Dynamic Head [ and cross- fusion, we directly adopt the vanilla ATSS [53] as our vision encoder to keep our architecture as neat as possible. Following GLIP, for Swin-T backbone, we also train versions of models, i.c. DeiCLIP-T(A),, DeICLIP-T(B) and complete version DetCLIP-T, which differ in using different training data.
Table D reports the results on LVIS [6.. Results with LVIS as pre-training data stand for the fully- supervised models trained with annotated data. With our proposed paralleled formulation, introducing more training data from different sources can consistently improve the performance. Le., comparing DeiCLIP-T(A) trained with only detection data with DetCLIP-T trained with additional grounding and image-text pair data, we can observe a considerable performance gain (2888% AP v.s. 35.9% AP). Besides, befitting from the proposed effective framework, our DetCLIP models outperform their GLIP counterparts by a large margin, i.e., DetCLIP-T(A) (resp. DeiCLIP-T) surpasses GLIPP-T(A) (resp. GLIP-T) by 10.3% (resp. 9.9%). Besides, DeICLIP-T also significantly outperforms GLIPv2-T by 6.9%. Note that our models are more lightweight (without heavy DyHead and cross-modal fusion) and trained with fewer epochs (our 12 vs. GLIP's s 24) and much less data. In addition, our DeICLIP-T's zero-shot performance even beats the fully-supervised model with the same backbone by utilizing weak- annotated data like image-text pair data. Results on LVIS full validation dataset can refer to the Appendix.
Efficiency Comparison. To demonstrate the efficiency of our proposed DetCLIP, we directly compare the training and inference speed of DetCLiP-T with the GLIP-T 28 in Table With the same setting of training with 32 V100 GPUs, the total training time for GLIP-T is about 10.7K GPU hours (5X than us) due to its heavy backbone and more image-text pair training data. On the other hand, DetCLIP-T achieves the 2.3 FPS (0.43 s/image) on a single V100 when performing inference LVIS (16). while GLIP-T can only achieve 0.12 FPS (8.6 s/image). With much better training and infer- ence efficiency, DetCLIP-T can still outperform GLIP-T 9.9% on LVIS.
Table 2: Efficiency comparison LVIS.
MODEL
TRAINING
INFERENCE
GLIP-T DETCLIP-T
10.7K GPU HRS
0.12 FPS
2.0K GPU HRS
2.3 FPS
Qualitative Visualizations. We illustrate the bounding box predictions on LVIS [66 dataset from DecCLIP-T and GLIP-T (28) model in Figló We can observe that by adopting the paralleled concept formulation and the external knowledge from the concept dictionary, our DetCLIP can outperform GLIP both on the accuracy and completeness of the predicted labels, especially on the rare classes
4.2 Ablation Studies
Table 3 studies the effectiveness of two core components of DetCLIP, i.e., the paralleled formulation and the knowledge enrichment with concept dictionary. The first row stands for implementation of a GLIP-A (28 model, which is modeled with sequential formulation and uses only detection data for training. Due to the implementation discrepancy, our version can achieve 23.7% zero-shot AP on LVIS 16]. which is higher than the official's 18.5% and serves as a stronger bascline.
First, applying the paralleled formulation can bring significant improvements (row 2), boosting the performance to 27.8%. This indicates that paralleled formulation is much more effective than sequential formulation for modeling object detection as vision-language task. However, directly applying the same approach to a larger scale dataset with heterogeneous supervisions, e.g., detection plus grounding, can hurt the performance (row 4). We speculate this is because paralleled formulation weakens the interaction between text concepts, resulting in the model's 's inability to effectively con- struct the connections between semantic- concepts. Therefore, we introduce word definitions to

---

Table 3: Ablation Studies of different components on LVIS [6 minival and other 13 detection datasets ("13 DATA"). Where P.F. C.E. N.S, L.C stand for the paralleled formulation, concept enrich- ment, negative samples and label completion. The latter 3 techniques are realized by the proposed concept dictionary. Numbers in parentheses under LVIS column are AP,/AP//AP/ respectively.
PRE-TRAINING DATA
P.F
C.E.
N.S
L.C
LVIS
13 DATA
X
23.7 (16.6/20.5/27.7)
30.1
0365
X
X
X
27.8 (22.2/26. 8/29.77
30.7
V
X
28.8 (26.0/28.0/30.))
33.8
X
X
X
28.2 (21.6/25
(21.6/25..0/ 00/32.2)
33.7
/
X
32.2 (26.4/30.3/34.9)
35,9
0365 43
GOLDG 23
30.3 (22.6/27
022.6/27.4/34.. 4/34.22
36.3
34.4 (226.9/33..0 9/36.3)
38.8
35.5(32..8/449/336
39,9
0365 43
GOLDG 33
YFCC 45
35.9(33.2/35../36.44
43.3
Table 4: The impact of the scale of the concept dictionary. The numbers in parentheses indicate the size of the corresponding concept dictionary. The first row means without using the concept dictionary. Zero-shot performances LVIS [16] reported.
LVIS MINIVAL (AP, AP. AP
LVIS
MODEL
CONCEPT DICTIONARY
AP (AP, AP. AP
28.2 (21.6 25.0
20.9 (15.3/ 17.5 17.1)
0365 (-0.36K)
27.8 (22.3/23.7 /32.4)
21.6 (19.3/ 18.7
DETCLIP-T(B)
0365+THINGS (-1.9)
28.1 (20.8 24.88/24..// /4.2/32.44
20.5 11.00/17.0/200
DETECTION + IMAGE-TEXT (~14K)
34.4 (26.9 33.9 36.3)
27.2 (21.9
25.5 31.5)
OLIP-T
O
Figure 6: Qualitative prediction results from GLIP-T (28 (first row) and DetCLIP-T (second row) LVIS [16] dataset. DetCLIP-T can produce more complete and precise predictions than GLIP-T.
the class names to help bridge relationships between different concepts, which boosts the performance to 32.2% (row 5). Sampling negative categories from concept dictionary also helps better utilize grounding data, improving the performance to 34.4% (row 7). Further introducing image-text pair datasets like YFCC [45] can bring substantial improvement for rare categories (row 8), while utilizing concept dictionary for label completion during pseudo labeling finally improves the overall AP to 35.9% (row 9). A similar performance pattern is also observed on 13 downstream detection datasets.
Impact of Concept Dictionary's Size. To study the impact of the scale of the proposed concept dictionary, we build three concept dictionaries with different sizes by using: (1) class names from Objects365 433;; (2) class names from Objects365- + Things [8]; (3) class names in (2) plus noun phrases extracted from YFCC100m [45] We equip our DeiCLIP-T(B. with these three concept dictionaries and compare their performance in Table It can be seen that using small size dictionary, e.g., Objects365 Things, can even bring performance drop compared to without the dictionary, while scaling up the dictionary with nouns from YFCC can significantly improve the performance.

---

Pseudo labels with concept dictionary. Fig visualizes the results of label completion via concept dictionary. Multiple effectiveness can be observed. E.g., cases (a) and (b) demonstrate concept dictionary contributes to labeling bbox with the category names not shown in the captions. In (c), Sctup to raxsed for shott stop (3) mndio cnimaaiion smn scoring. This for MEA Thesis project.
LNER Guodky "A4" 6 No.0000
(b) at the bar Market Street Cade
[c] brown butterfly, resting on leaf.
(d)
Dwight D. lisenhower in ISR green livery Green Roy Railoadd Muszam.
Figure 7: Visualization of pseudo label results on YFCC [45] dataset. The texts below the images the corresponding captions. Top row: labeling results using the original captions; bottom row: labeling results with the designed concept dictionary. It helps produce higher-quality pseudo labels.
Box
cat
cattdeffat
catedefperson
person
personedel
persorvdefaat
0.30
X
x
0.06
0.56
0.15
Figure 8: Alignment scores when using text inputs with different definition settings. Cross mark means the scores are below the threshold. defz means we use the definition of category x. Results show that the definition can play an important role in classifying objects.
We speculate this is because a large dictionary can provide rich negative concepts for grounding data, encouraging the model to learn more discriminative features.
finer-grained pseudo label can be produced, i.e., 'antheraea polyphemus' v.s. the original 'brown butterfly', which helps the learning of rare categories. In (d), label noise in the caption is alleviated.
Importance of concept enrichment. To directly illustrate how the category definition helps the model achieve better detection performance, we conduct the experiments infer with different text inputs with DetCLIP-T utilized. We compare the three different cases, i.c., no definition, true definition and false definition (use definition of another category), and report the alignment score for different objects in Fig g Observations are: (1) True definition helps the model to better determine the category (i.e., increases the confidence score); (2) Wrong definition may confuse the model and bring the false positive samples; (3) Both the category name and definition matter to the model.
5 Conclusion
In this paper, we propose a novel open-world detection pre-training framework named DeICLIP, aiming at improving the open-domain ability and learning efficiency of the open-world detector. By unifying three kinds of supervision via paralleled concept formulation, our DetCLIP can learn from different domains and enhance the training efficiency. We further propose concept dictionary module to improve the discovery and coverage of the novel knowledge by importing external knowledge. The proposed usages of the concept dictionary achieves better open-world detection result in terms of both common and rare categories on LVIS. Experiments on multiple downstream detection datasets suggest that our DetCLIP is more powerful than current SOTA worn-world detectors such GLIP 28
ckdnoweessssnnts We gratefully acknowledge the support of MindSpore]] CANN (Compute Architecture for Neural Networks) and Ascend Al Processor used for this research.
htttps:/wwee mindspore.ccmm
10

---

References
[1] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. In NeurlPS, 2020. 2
[2] Zhaowei Cai, Gukyeong Kwon, Avinash Ravichandran, Erhan Bas, Zhuowen Tu, Rabul Bhotika, and Stefano Soatto. X-detr: A versatile architecture for instance-wise vision-langusge tasks, arXiv preprint arXiv:2204.00 05626, 2022.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229. Springer, 2020. ?
[4] Soravit Changpinyo, Wei-Lun Chao, and Fei Sha. Predicting visual exemplars of unseen classes for zero-shot learning. In ICCV, pages 3476-3485, 2017.
[5] Jiaoyan Chen, Yuxia Geng. Zhuo Chen, lan Horrocks, Jeff Z Pan, and Huajun Chen. Knowledge-aware zero-shot learning: Survey and perspective. arXiv preprint arXiv:2103.00000 000700 2021. 2
[6] Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint
arXiv: wwXiv 07155, 2019.
[7] Jifeng Dai, Li, Kaiming He, and Jian Sun. R-fen: Object detection via region-hased fully convolutional networks. In NeurlPS, 2016. 2
[8] Xiyang Dai, Yiapeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang, Dynamic head: Unifying object detection heads with attentions. In CVPR, pages 7373-7382, 2021.
4.1
Achal Dave, Piotr Dollár, Deva Ramanan, Alexander Kirillov, and Ross Girshick. Evaluating large- vocabulary object detectors: The devil is the details. arXiv preprint arXtv:2102.01066, 2021.
[10] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-traine traininii of deep bidirec- tional transformers for language understanding, In Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. In ICLR, 2020. 2
[12] Yu Du Fangyun Wei, Zihe Zhang, Miaojing Shi. Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. arXiv preprint arXiv:2203. 14940, 2022. [
[13] Mohamed Elhoseiny. Yizhe Zhu, Han Zhang, and Ahmed Elgammal. Link the head to the" h beak": Zero shot learning from noisy text description at part precision. CVPR, pages 5640-5649, 2017. 2
[14] Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong. Towards open vocabulary object detection without human-provided bounding boxes. arXiv preprint arXiv:2111.09455
2021.
3.1
[15] Xiuye Gu. Tsung-Ye Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.11 13921. 2021. 2
[16] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR, pages 5356-5364, 2019.
[17] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-can. In ICCV, pages 2961-2969,
2017. 4
[18] Martin N Hebart, Adam H Dickter, Alexis Kidder, Wan Y Kwok, Anna Corriveau, Caitlin Van Wicklin, and Chris Baker. Things: A database of 1,854 object concepts and more than 26,000 naturalistic object images. PloS one, 14(10)=2023399,, 2019. 3.2.1 4.2
[19] Chenhan Jiang, Hang Xu, Xiangdan Liang, and Liang Lin. Hybrid knowledge routed modules for large-scale object detection. In NewrIPS, 2018.
2

---

[20] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding, In ICCV, pages 1780-1790, 2021.
[21] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and- languuug- transformer without convolution
region supervision. In JCML, 2021. 2
[22] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint orXiv:1412.600 2014.
[23] Ranjay Krishna, Yuke Zhu. Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, Michael Bernstein, and Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJV, 2016. 3
[24] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Ka- mali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, el. The open images dataset v4. International Journal of Computer Vision, 128(7): 1981, 2020. 1.2.1
[25] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image wre-training for unified vision-language understanding and generation. arXiv preprint arXiv:2201. /2086, 2022. 2
[26] Junnan Li. Ramprasaath R. Selvaraju. Akhilesh Deepak Gotmare, Shafiq Joty, and Steven Xiong. Caim- ing andHoi. Align before fuse: Vision and language representation learning with momentum distillation. Technical Report arXiv:2107.07611 2021.
[27] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint 190.::908800357. 2019. -
[28] Liunian Harold Li, Pengchuan Zhang. Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, al. Grounded language-image pre-training. arXiv preprint arXiv:21 2021.
6
[29] Yanwei Li, Hengshuang Zhao, Xiaojuan Qi, Liwei Wang, Zeming Li, Jian Sun, and Jiaya Jia. Fully convolutional networks for panoptic segmentation. In CVPR, pages 223, 2021. 2
[30] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, ICCV, pages 2980-2988, 2017. 3.3
[31] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, 2014. ايل
[32] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander Berg. Ssd: Single shot multibox detector. ECCV, 2016. 2
[33] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.
[34] George Miller, Wordnet, an electronic lexical database. 1998.
3.2.1
[35] Ruizhi Qiao, Lingqiao Liu, Chunhua Shen, and Anton Van Den Hengel. Less is more: zero-shot learning from online textual documents with noise suppression. CVPR, pages 2249-2257, 2016. [
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In /CML, pages 8748-8766 8763. PMLR, 2021.
3.1
[37] Joseph Redmon. Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016.
[38] Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. Leamingg deep representations of fine-grained visual descriptions. In CVPR, 2016. 2
[39] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015. 02
[40] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In CVPR, pages 658-666, 2019. 3.3
12

---

[41] Ruslan Salakhutdinov. Antonio Torralba, and Josh Tenenbaum. Learning share visual appearance for multiclass object detection. In CVPR, 2011.
[42] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam Visual explanations from deep networks via gradient-based localization. In ICCV, pages 618 626, 2017.
[43] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng. Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: large-scale, high-quality dataset for object detection. ICCV, pages 8430 8439, 2019. 3.2.1
4.2
[44] Richard Socher, Milind Ganjoo, Christopher Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. NewrIPS, 26, 2013. 2
[45] Thomee, David A Shamma, Gerald Friedland. Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Y0mm100m: The new data in multimedia research. Communications of the ACM, 59(2):64-25 73, 2016. 4 4.2
[46] Wenhui Wang. Hangbo Bao, Li Dong, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-of mixturt-
arXiv preprint arXiv:2111.02358 2021. 2
[47] Xiaolong Wang, Yufei Ye. and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In CVPR, 2018. 2
[48] Johnathan Xie and Shuai Zheng. Zsd-yolo: Zero-shot yolo detection using vision-language knowledgedis- tillation. arXiv preprint arXiv:2 12066, 2021.
[49] Hang Xu, Can Jiang, Xiaodan Liang. Liang Lin, and Zhenguo Li. Reasoning-rema Unifying adaptive global reasoning into large-scale object detection. In CVPR, pages 6419-6428, 2019. P
[50]
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filipp Fine-grained interactive language-image pre-training, In ICLR, 2022.
3.1 3.2.2
[51] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. arXiv preprint arXiv:2203. 11876, 2022.
[52] Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang. Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. arXiv preprint urXiv:2206.08 05836. 2022,
[53] Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z Li. Bridging the gap between anchor- based and anchor- free detection via adaptive training sample selection. CVPR, pages 9759-9768, 2020.
3.3
4.1
[54] Wenwei Zhang, Jiangmiao Pang. Kai Chen, Chen Change Loy. K-net: Towards unified image segmentation. NeurlPS, 34, 2021. 2
[55] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the Appendix
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the
Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [Yes] We already include the data and instructions Section 4, and the code will be released upon acceptance.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 4 and Appendix.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [N/A] Our experiments are quite stable with multiple runs. All results are observed via fixed seed.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license the assets? [Yes] the used existing assets are open-source.
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] We use open source data and the data anonymized. (e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] We use open source data and the data is anonymized.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "yZZAUT13S4I", "reviewer": "Reviewer_rAnj", "summary": "The paper is developed based on GLIP:\n\n- To solve the problems of inefficient interaction between categories and the restriction of max input sentence length, it proposes a parallel method. Compared to the sequential input in GLIP, the paper extracts phrases as the input in a parallel manner.\n- To build implicit relationships between input phrases, it proposes a concept dictionary to provide the prior knowledge.\n\nThe paper is verified in the task of zero-shot detection on LVIS.\n", "questions": "- The notations in Formula (1) are odd due to the upper letter T and Transpose. Please consider changing it.\n- Experiments mainly use Swin-Transformer as the backbone. Can the backbone be ViT?", "limitations": "None.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The framework does make sense. I agree with the point that extracting the noun phrase is efficient for alignment between regions and concept words.\n- The proposed large-scale concept dictionary could be a future valuable toolkit for future open-vocabulary or vision-language research if it can be open-released. \n- Compared to GLIP, improvements are large.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lSwlXGuae3v", "reviewer": "Reviewer_aMQW", "summary": "This paper proposes an extension of GLIP for Open World Detection. The extension comprises two parts: one methodological by introducing a so-called parallel processing of the text data, and one more data related that allows the unification of many heterogeneous datasets for training. The latter also proposes a richer description of the categories -- deemed concepts in the paper -- through wordnet.\nThe paper reports significant improvements of GLIP. It is mainly evaluated on LVIS dataset", "questions": "On the negative side:\n- the methodological contribution of the parallel processing is rather thin and is actually the first thing that would come to mind when reading the original GLIP paper.\n- most of the improvements and contributions come from the dataset side. Although the framework for creating the dataset is worthy, it is more an engineering approach rather than an actual methodological contribution. After reading the paper you get mostly the impression that the authors created a much better dataset and then trained a (straightforward) extension of GLIP on it.\n- The evaluation on LVIS is sound and convincing. However with such big and heterogeneous data used, the zero shot capabilities of the method are not clear. Can the authors also evaluate following the VILD protocol (https://arxiv.org/pdf/2104.13921.pdf) where the method is trained on base classes and then explicitly evaluated on unseen classes?\n\n**Final Recommendation**\nAs said I will stick to my original score which is line with the score from other Rs.", "limitations": "I didn't see a paragraph for this in the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "On the positive side:\n- the authors study an important problem in literature that of open-world object detection where not many solutions are available\n- the parallel extension to GLIP, although straightforward, makes sense. The original GLIP sequentual formulation was problematic and did not scale well with a large number of categories\n- the authors did a good job in terms of proposing a framework for unifying many established and heterogeneous datasets including standard detection dataset, grounding data, and image/caption pairs. Actually, in my opinion, this is the biggest contribution of the paper. \n- the improvements over GLIP are significant. Even when trained on the same data, +5.1% is reported (table 3).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ke5CqQNASSq", "reviewer": "Reviewer_82yp", "summary": "This paper explores a parallel visual-concept pretraining method for open vocabulary object detection by resorting to knowledge enrichment from a designed concept dictionary.\nThey make the following contributions:\n1.a novel paralleled concept formulation to improve learning efficiency.\n2.a novel concept dictionary to enrich the prompt text concepts during joint pretraining.\n3.DetCLIP outperforms the state-of-the-art GLIP.", "questions": "See above ", "limitations": "See above ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1.  The method is simple and effective and I like the examples author offered here and there.\n2.  The model significantly improves the learning efficiency.\n3.  The experiment results seem solid. They evaluate the method on LVIS and transfer the model to 13 downstream detection datasets. \n\nWeaknesses\nI think the writing of the abstract and introduction needs to be significantly improved: they are too long especially for the abstract. And also many concepts were used directly without explanation/definitions/references at the first hand or not at all. For instance, what is GLIP, DetCLIP-T, GLIP-T, positive samples v.s. negative samples ? What is the grounding, what's its difference to image-text pair data ? what's the partial label problem ?   \n\nAuthors claimed their method is much more efficient and effective than GLIP in terms of data usage. However I notice that they have also utilized other models such as CLIP , FILIP. Are these models considered as part of the proposed method (are they multi-stage) ? How would this compare to GLIP ? \n\nL157 Can you provide further explanation that why interaction between categories is unnecessary?\n\nL169: Why context information is just removed. I know the context cannot fed into the text encoder individually. Maybe it can be added to every individual concept phrase. Any study for this  ?\n\nL243 : What is partial positive concept ?  Is it derived from the partial labelling problem, I don't see a clue.  \n\nLast, bold text is normally for title/subtitle as a standard practice. Be using it properly!  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HZVTm7KSOY", "reviewer": "Reviewer_5KVn", "summary": "The work proposes to build a concept dictionary which enables to perform visual-concept pretraining in a unified way across multiple heterogeneous datasets for open-world detection. With the proposed dictionary design, the proposed method can even perform partial annotation enrichment for label completion and negative concept sampling to further improve the performance.", "questions": "The experimental results are very thorough covering different open-vocabulary datasets, and also show promising performance improvement. I have few questions.\n\n1. Is the definition from the wordNet necessary to be used? If we only use the unified label, how much will it affect the performance?\nThe following work is not targeted at open-vocabulary detection, but it also proposes to perform contrastive learning using the unified space.\n\nYang, Jianwei, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. \"Unified contrastive learning in image-text-label space.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19163-19173. 2022.\n\n\n2. How is the pseudo labeling for label completion of using the all the concepts in the dictionary conducted? The dictionary size is usually very large (e.g., 14k). The paper describes to add the concepts in the dictionary as the addition category inputs. \n3. For concept enrichment and partial concept enrichment, how would the pretrained encoder FILIP affect the final performance? Do the authors try other text-encoders?\n4. How is the proposed method compared with recently released work, GLIPv2 (optional)?\nhttps://arxiv.org/abs/2206.05836", "limitations": "The authors do address the limitations and why there is no potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The built concept dictionary can be used to convert the annotations of heterogeneous datasets, including detection, visual grounding, etc, into a unified one.\n2. With the concept dictionary, the proposed method can perform concept enrichment with the assistance of other pretrained text encoder, FILIP in addition to negative concept sampling and partial annotation enrichment for those labels which are not well annotated in the existing datasets.\n3. The performance improvement in LVIS for rare and frequent categories is significant as compared with other previous state-of-the-art method.\n\nWeakness:\n\n1. The construction of the concept dictionaries rely on the wordNet, and thus  the concepts are mainly composed of those which are well-defined in WordNet. \n2. For the concept enrichment, it still requires a pre-trained text encoder, like FILIP, to match related concepts. The quality will depends on the reliability of the pretrained text-encoder used.\n3. When resolving the name ambiguity among different datasets, only the text part is used without exploiting the corresponding image information.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept