DesCo: Learning Object Recognition with Rich Language Descriptions
Authors: Liunian Harold Li
Zi-Yi Dou
Nanyun Peng
Kai-Wei Chang
Venue: NeurIPS 2023 poster (2023)
Keywords: Vision language
fine-grained recognition
object detection

Abstract: Recent development in vision-language approaches has instigated a paradigm shift in learning visual recognition models from language supervision. These approaches align objects with language queries (e.g. "a photo of a cat") and thus improve the models' adaptability to novel objects and domains. Recent studies have attempted to query these models with complex language expressions that include specifications of fine-grained details, such as colors, shapes, and relations. However, simply incorporating language descriptions into queries does not guarantee accurate interpretation by the models. In fact, our experiments show that GLIP, a state-of-the-art vision-language model for object detection, often disregards contextual information in the language descriptions and instead relies heavily on detecting objects solely by their names. To tackle the challenge, we propose a new description-conditioned (DesCo) paradigm of learning object recognition models with rich language descriptions consisting of two innovations: 1) we employ a large language model as a commonsense knowledge engine to generate rich language descriptions of objects; 2) we design context-sensitive queries to improve the model's ability in deciphering intricate nuances embedded within descriptions and enforce the model to focus on context rather than object names alone.  On two novel object detection benchmarks, LVIS and OminiLabel, under the zero-shot detection setting, our approach achieves 34.8 APr minival (+9.1) and 29.3 AP (+3.6), respectively, surpassing the prior state-of-the-art models, GLIP and FIBER, by a large margin.

DesCo: Learning Object Recognition with Rich Language Descriptions
Liunian Harold Li"
Zi-Yi Dou Nanyun Peng Kai-Wei Chang
University of California, Los Angeles
(liunian. harold. zdou, violetpeng, kwchang ucla. edu
Abstract
Recent development in vision-language approaches has instigated paradigm shift in learning visual recognition models from language supervision. These approaches align objects with language queries (e.g. "a photo of cat") and thus improve the models' adaptability to novel objects and domains. Recent studies have attempted to query these models with complex language expressions that include specifications of fine-grained details, such colors, shapes, and relations. However, simply incorporating language descriptions into queries does not guarantee accurate interpretation by the models. In fact, our experiments vision-language model for object detection,
show that GLIP, a often disregards contextual information in the language descriptions and instead relies heavily on detecting objects solely by their names. To tackle the challenge, we propose new eeseciption-connitiionnn recognition models with rich language descriptions consisting of two innovations: I) we employ a large language model as a commonsense knowledge engine to generate rich language descriptions of objects; 2) we design context-sensitive queries to improve the model's 's ability in deciphering intricate nuances embedded within descriptions and enforce the model to focus on context rather than object names alone. On two novel object detection benchmarks, LVIS and OminiLabel, under the zero-shot detection setting, our approach achieves 34.8 APr minival (+9.1) and 29.3 AP (+3.6), respectively, surpassing the prior state-of-the-ar models, GLIP and FIBER, by a large margin.
(DesCo) paradigm of learning object
Introduction
Training visual recognition models to classify detect objects with a fixed of pre-defined categories has been the convention for long time. However, models trained using this approach often encounter difficulties when adapting to unfamiliar concepts and domains. Recently, there has been a paradigm shift towards training visual recognition models with language supervision, using contrastive objective on a large amount of image-text data containing diverse range of visual concepts. These models can then be transferred to downstream tasks via language queries. For example, CLIP (36] can perform image classification using template query such as "a photo of (class name)"; GLIP 25 can perform object detection by querying the model with "Detect: person, cat, dog
Early applications of these models typically utilize simple language queries that consist of object names. However, language queries can convey much richer and more comprehensive information, such as object attributes, shapes, textures, and relations. These pieces of information can be especially useful for identifying novel visual concepts that do not appear in the training corpus specifying
Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Detect with specifications for shape & subpart
Detect with specifications for relation
Target Object
Confuseble Object
Target Object
Confusable Object
oclai:: 0.39
edbir: 0.58
tart: 0.55
clown: 0.72
clown: 0.74
clowm: 0.71
clown 0.30
Ours
Curs
GLIP
Qurs
Eclair kind of food, long, cylindrical pastry, filled with cream, topped with chocolate
Tart, kind of food, raund, could be filled with fruits, could be served with cream
A clown making balloon animal for pretty lady
A clown kicking ball for pretty lady
Detect with specifications for shape & subpart
Detect small objects
(w/o object name) Confusable Object
Target Objact
Target Objact
Confusable Object
tool: 0.59
033
Teaseen 0.5
0.33
2.24
Jock 2.00
Ours
GUP
Ours
GUP
Ours
GLP
Curs
GLIP
A kind of tool, wooden handle with round head, used for pounding or hammering
A kind of tool, long handle, sharp blade, could be used
Headset. kind of electronic device, two earpieces
Mistophone, kind of audio device, long. thin, could handled, could be connected...
chopping wood
connected to a band...
Comparison between our model (DESCO-GLIP) and the baseline (GLIP 25 Each image paired with a positive query for target object and negative query for confusable object. A successful model should locate the target object and ignore the confusable object in the image based on fine-gres gre-ggriied specifications for shapes, subparts, relations, etc. We highlight the descriptions that match and do not match to the queried object in blue and red, respectively. Results show that our model can successfully localize the target object and suppresses the negative query even for the difficult cases when the object name is not in the query or the object.
Figure I:
specific needs. For example, the concept of "mallet" can be described as "a kind of tool, wooden han- dle with round head"' (Figure I bottom-left). This decomposes object recognition into recognizing fine-grained details (such as attributes, sub-parts, shapes, etc.) and aligning them to the descriptions. Several studies [23] 4T 3T have explored the idea of guiding language-basee recognition models using such descriptive prompts. However, few existing models take complex queries into account during training. As a result, current models often struggle with recognizing intricate object names, attributes, and relations described in natural language sentences [33] 43
(see examples in Figure I
In this paper, we develop vision-language model capable of leveraging description-rich lan- guage queries to perform object detection. This work aligns with the recent surge of interest in instruction/proonttavan vision-language models (see discussion in Section 2). Our goal is to equip VLMs with the ability to comprehend complex language input describing visual concepts, similar the capability of large language models. We specifically study instruction/prompts in the form of descriptive queries. We focus on object detection, as it requires fine-graines recognition and more challenging than image-level tasks. However, our method can be generalized to other vision tasks such as classification and segmentation 611)
We identify two major challenges that prevent existing models from efficiently utilizing rich language descriptions: (1) Fine-grained descriptions are rare in image-caption data used for training current VLM This resembles the reporting bias phenomenon (34:: when writing captions for images, humans tend to directly mention the entities rather than give a detailed description. (2) Even when provided with data rich in descriptions, models often lack the incentive leverage these descriptions effectively. The main training objective is to align positive phrases with relevant regions while suppressing negative phrases. However, if positive/negative phrases can be distinguished without
"W count the label data used by models like GLIP as image-caption data because the labels are converted into captions through templates.
2

---

descriptions, the training mechanism fails to incentivize the model to use the provided description. For example, a positive phrase like "A toy bear holding a mallet, which has a wooden handle with a round head," and negative phrase like "A toy bear holding an ax, which has long handle and sharp blade," can be differentiated based solely on the words mallet and ax. This issue resembles the issue discovered by (53), where vision-language models ignore word order and treat a query as a "bag-of-words due to insufficient incentives from the contrastive objective. In addition, current models suffer severe hallucination when given natural language queries (in contrast to "template- like" queries) due to shortcuts introduced in training query formulation. This can be seen in the bottom-right picture of Figure I where GLIP hallucinates and predicts multiple wrong boxes for "microphone" while "microphone" does not appear in the image.
Based on the observations, we present Description Con---ooonnnddd (DEsCo) paradigm of learning object recognition models from language descriptions based on two synergistic ideas:
(1) Generating descriptions with large language models. Instead of learning from raw image- caption pairs, we use large language models as world knowledge base and generate detailed descriptions based on the original caption. We prompt GPT-3 [2] with "What features should object detection models focus on for (an entity in the caption J??" This serves as a scalable approach to transfer the image-caption data into image-description data.
(2) Context-sensitive query construction. As discussed, even if we provide descriptions during pre- training, models can still ignore the language context. Our solution is to create a "context-sensstive query", which is set of positive and negative phrases that can only be distinguished by reading the descriptions (Figure We explore two strategies: 1) constructing "Winograd-like" [13 43
queries by using large language models to generate confusable object descriptions and captions and 2) generalizing the original grounding task to allow full-negative queries, reducing hallucination.
We apply our approach fine-tune two tfate-of-the-ar language-conditioned object detection models GLIP (25 and FIBER 77.. We use the same raw training data as the baselines but convert the data into description-riid -rich queries. We evaluate our methods in two settings. (1) Zero-shot generalization to novel categories (LVIS (II), where we use GPT-3 to generate descriptions given class names. DESCO-GLIP (Tiny) improves upon GLIP (Tiny) by 10.0 APr, even outperforming the larger GLIP (Large); DESCO-FIIBES improves upon FIBER by 9.1 APr. (2) Zero-shot generalization to natural descriptions given by humans (OmniLabel (88). DESCO-GLIP and DESCO-FIBER improve upon the baselines by 4.5 AP and 3.6 AP, setting a new state-of- the-art performance level. Code is available
at ftttt:::/ thub. com http:/llnnnnnnnnnnee haw.miannaarlldli/eeeeeoo
2 Related work
Language-based visual recognition models. Visual recognition models typically trained to make predictions based on a fixed set of classes (20 26
39 33 58). The trained models are hard to generalize to open-domain settings. Recent studies have developed visual recognition models that take into account language queries, i.e. language- recognition. This line of research can be traced back to early work of generalizing image classification 42 and object detection [[] models with word embeddings. Recently, CLIP (36) reformulates image classification as image-text matching and pre-trains models on large-scale image-caption pairs to learn transferrable representations. They demonstrate strong zero-shot performance on various classification tasks. Recent work has applied the technique to fine-grained recognition tasks, such as object detection [17
10
25
57
56
3 32
7 (28) and segmentation [21 9 16
47 (54, 24].. These works either use pure image-text data as supervision [47], or reformulate labeled data into image-text data 21, or pseudo labels image-text
data with fine-grained labels 225 Orthogonal to architecture design or scaling-up, which is the focus of many prior studies, this paper points out that the vanilla way of using image-text data is insufficient and studies how to train these models to take more flexible and informative language queries.
Vision-language models with complex prompts. As vision recognition models become language- aware and language models become vision-aware 45] 60 22). there is growing interest in studying
whether these models can take complex language prompts, such as task instructions (e.g., GPV [12
18] SEEM (22, VisionI.LM L.M 46)). descriptions 23 or even dialogues (e.g., LLLVV (7). We study specifically descriptive prompts, which are especially useful for generalizing to novel categories and
customized detection needs; model that can understand descriptive prompts can also serve as the
3

---

backbone for supporting aforementioned other types of prompts. Similar to our work, K-LITE A1 proposes retrieve knowledge for a visual concept using external knowledge bases, then use the enriched concepts for image classification object detection; similar techniques have also been proposed by [31 48). DetCLIP [50] builds large-scale concept dictionary, based on which they provide definitions from WordNet. Different from these studies, our methods show that simply presenting the descriptions at training or inference time is not enough; we propose simple technique to force models to focus the provided descriptions (Section 3.2.2). Our work relates to line of work that seek to reveal and fix failure patterns of image-text matching models (e.g., CLIP) by
creating hard negative examples 43 53
37
3 Approach
In this section, we first briefly introduce language-based object detection models, then illustrate the details of our proposed approach.
3.1 Background
We give introduction language-based object detection models 17 25 7 which take language query and an image as inputs, and predict bounding boxes and their alignment to phrases in the language query. In the following, we use GLIP as an example.
Baseline: Grounded Language-Imann Image Pre-training (GLIP). At the center of these approaches is "reformulating any task-specific foxed-vocab classification problem as task-agnostic open-vocabule vision-language matching problem" (55]. The best example is CLIP which reformulates image classification image-fext matching. Similarly, GLIP unifies training data into grounding format: (T,Q, B, T). T is the image; Q is the text query: B E R is the bounding boxes; T E {0, 1 indicates the ground-truth alignment label between the N bounding boxes and K tokens in the query. The key is how to formulate the query with data from two kinds of sources:
N
Detection data. For object detection data such Objects365 [39], the query is the concatenation as a list of object classes, such as "Detect: person. bicycle. car. ... toothbrush". Note that negative object classes included in the query; this makes such query-based detection models similar to classical detection models.
Grounding data. Typically, is image caption, containing entities that are aligned to annotated object regions (35). For example, "A toy bear holding a mallet" the caption; "toy bear" and "mallet" are the "groundable" entities. For densely annotated grounding data (multiple captions for one image) [19], we can concatenate multiple captions into a longer query. Image-caption data (without annotated boxes) can be transferred into grounding data via pseudo labeling with grounding model (25).
Given I and Q, compute the alignment scores Sgrund between image regions and words in the query:
O,L=Enc(I, Q), Spround OL loss(Sgroun:: T) + Lioe
where I. E Kd is the contextual token features and O E R xd are the regions features. ENC is a
vision and language encoder that takes both image and text as inputs and fuses their representations. The training loss contains the region-word matching loss and localization loss Cioe as in conventional object detection models.
Inference with language query. At inference time, the model can be used to locate entities/class names appearing in the query. One could simply provide list of candidate object names (as in the detection data training format). (25) also show the promise of using descriptions for generalization to novel concepts; however, show that while GLIP be influenced by the description, it does not always take the details in the description into account.
3.2 Learning with language descriptions
To train object recognition models that fully utilize language descriptions, we propose to generate descriptions with large language models and construct context-sensitivy queries during training. The following subsections provide further details.

---

Fine-grained descriptions
A toy bear holding a mallet
Confusable caption
Detect: Mallet.
Mallet
Confusable object
GPT-3
GPT-3
Bear, Cat...
A kind of tool, wooden handle with round head, used for pounding... ... tool, long hancile, sharp blade,
A polar bear holding mallet. A toy bear holding a mallet...
A toy bear holding a mallet.
polar bear
tool
tool
mallet
toy
mallet
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Original training data for GLIP
Description-rich and context-sensitive data for DESCO-GLIP
Figure 2: Given the original training data of GLIP, we transform it to be description-rich and context- sensitive by: generating descriptions for entities and composing each of them with confusable object descriptions; 2) generating negative captions. We visualize the gold alignment labels (ground truth) between tokens and regions for the new data. Notably, words such tools are assigned both positive (blue block) and negative (red block) labels in alignment with the corresponding object depending on the context of the query. As such, the model requires understanding the description in order to make the correct prediction.
3.2.1 Description generation with large language models
Fine-grained descriptions could be scarce in image-caption data due to reporting bias. While this problem can be alleviated by scaling up the pre-traiming corpus, we show that large language models 2 can be used to effectively generate the descriptions and thus enrich our training corpus.
We leverage large language model transform query Q into description-rich query LLM(Q). In this work, we only focus on generating descriptions for entities mentioned in the original query. We construct vocabulary consisting of 10K entities appearing frequently in the pre-training corpus. For each entity, we prompt large language model: what features should object detection models focus on for {entity)? We find that large language models give high-quality re- sponses (see examples in Figure 1 and Figure 4) The specific prompts are included in the appendix.
3.2.2 Context-sensitive query construction
Can we simply add the description-rict data to the pre-training corpus? ? An intuitive idea is to append the description to the original entity to form a training prompt (e.g., "Detect: mallet. bear "Detect: mallet, kind of tool, wooden handle ... bear, kind of animal, ....). "). However, we find that models naively trained with these prompts still do exhibit "context-sensitivityy" i.e., they make predictions solely based on the entity names while ignoring other contexts (see Section T for quantitative analysis). As result, we observe no evident benefit in incorporating descriptions during inference (Table 3). In the following, we claborate on why the model learns to ignore the descriptions and propose solutions.
Model learn statistical shortcuts. We first illustrate that without careful design, the model could learn two statistical shortcuts that make them insensitive to descriptive queries.
(1) Entity shortcut. The model is trained to align the entities in the query to image regions (this includes predicting "no-alignment" for entities not appearing the image). Intuitively, if the aligmment can be predicted without relying on the context information in the query, then the model is incentivized to focus on the context information. Figure F Zillustrates this issue with an example. The left side shows the training data of GLIP, where the top query ("Detect: Mallet. Bear. Cat...") comes from detection data and the bottom query ("A toy bear holding a mallet.") comes from grounding data. The problem with such queries is that they can be grounded by only focusing on the entity names and ignoring the context. We denote the gold alignment label of regions as T, the entities in the query E, and the non-entity part (context) of the query as C. The mutual information
5

---

I[T; CIE, between C' and T given E and the image I would be low. That is, the non-entity parts of the queries do not affect the label of the region. Training models on such data will not encourage the model to focus on the descriptions as they provide no additional information. This is similar to the "memorization overfit" issue observed in [51] the model can simply choose to "memorize" the alignment between the entities and regions.
(2) Format shortcut (hallucinationn. Popularized by GLIP 25 a line of work adopts a unified view of phrase grounding and object detection: detection can be seen as language-context-Tng
grounding while grounding can be seen as language-contes cww-eettdeeedeeeeemmm detection. However, this unification still imperfect: phrase grounding (or referring expression [52) traditionally only concerns locating entities in a caption that always exist in the image; thus the model learns to always treat the natural- language queries (in contrast to the template-like queries) as positive and tries to ground every entity mentioned in the sentence. This will result in failure examples as illustrated picture of Figure [1] Such "hallucination" can be commonly seen on models trained on language grounding data [17]: these models are almost incapable of distinguishing positive and negative queries.
the bottom-right
Constructing context-sensitive queries. This motivates our solution of creating queries that are hard to solve without context (Figureßand Figure We explore two strategies in this study.
(1) We construct training queries similar to the Winograd format. For example, when training on detection data, instead of "Detect: mallet, a kind of tool, ...", we remove the entity name "maller" from the query and sample another description of "confusable" entity that is also a kind of tool. Pairing the descriptions of the two "'coffuable" entities creates a strong supervision signal (the middle example in Figure 2): the alignment label (0 or 1) of the word "tool" now depends on its context. The confusable entities are obtained by prompting the large language models as well. Similarly, for training on grounding data, we prompt language models to generate confusable (hard negative) captions that differ from the original captions only by few words (the example on the right in Figure 2}. Note that the label of the word "mallet" is now affected by the context: the first "mallet" is assigned 0 as the caption ("A polar bear holding mallet") is negative. Mixing in such hard negative captions encourages the model to focus on the context surrounding the entities, such as relations to other entities. To make the confusable caption generation process scalable for image-caption data, we first perform in-context inference and prompt GPT-3 to generate around 50K negative captions based on positive captions; then distill this knowledge to the open-sourced LLaMA-7B [44) model that is instruction-fiettuned using low-rank adaptation [4] and perform inference on large-scale image-caption data.
(2) To resolve the hallucination issue, we generalize the original grounding task: instead of always feeding the model query that contains at least one descriptiio/captiion matching the image, we allow the query contain only negative descriptions/captions (Figure B). Thus, the model cannot blindly ground all entities mentioned in the query; implicitly, it needs to perform image-text matching (36) as well as phrase grounding. This was partly done in GLIP (see Appendix C. the original GLIP paper), but the query still contains at least one positive entity.
Overview. In Figure 3 we summarize the overall data construction algorithm. Algorithm
B E RN are the bounding boxes of an image; F are M positive objects (entities) that appear in the image; V are the deseriptions of all candidate objects in a pre-defined vocabulary; T E (0, 1) ] X denotes the gold alignment between boxes and entities. We first prompt LLM to generate descriptions for the positive entities and propose confusable entities and descriptions; the prompt is included in the appendix (Line 3). The original entities are removed from the descriptions with Párop 0.5 (DropEntity, Line 4-5). Random negative descriptions from the vocabulary are added to the candicate description set (Line 7). We then randomly subsample the descriptions and concatenate them to form a final query q": this is because the total length of all the candidate descriptions is too large (Line 8). Boxes and the mapping relations between boxes and tokens accordingly adjusted (Line 9). Algorithm 2: C' is the original caption and E are M positive phrases we extracted from the caption. The last two lines of both algorithms are important: after SubSampleConce it is very likely that some positive sub-queries are dropped from Q: then LabelAssign would drop boxes that are mapped to the dropped sub-queries. The output B could end with fewer boxes or even no boxes. This is
hhaps:/github.ccmm/ooorppppppp

---

Algorithm Generating Queries for Detection Data
Algorithm Generating Queries for Grounding Data
Input: B (boxes), T (alignment matrix), E (positive entities), (vocabulary)
Input: B (boxes), T" (alignment matrix), E
Q -
(positive entities), V (vocabulary), C' (caption)
2: for i <- to M do
if random() < Pies then Q, T. B +- Algorithml (,,
3:
q. Q <- L...(promp,, F.))
2:
4:
if' raandom) Parop then
else
5:
q. 2.Q - DropEnnityyg, Q-) Q QuQU(quUQ
4:
Q < LLM C)
6:
5:
Q - u
7: Q f QU RandSample(()
q- -SubSampleConcat()
8: 9" + SubSampleConcat(()
7:
q; I, B LabelAssign(( C,
9: q' T, B LabelAssigo( T, E,
Figure 3: Algorithms for generating queries from detection data and grounding data.
different from the strategy in GLIP traditional object detection training recipe, where strive to keep all boxes provided.
Experiment
In this section, first investigate whether current models (GLIP) utilize language descriptions out-of-the-box: then we show that our method allows the model to utilize language descriptions and improves performance LVIS and OmniLabel significantly.
4.1 Can language-conditioned detection models utilize language descriptions?
As proof of concept, we first show the GLIP strug- gles to utilize language descriptions out of the box and analyze the failure patterns.
Model
ABox
AConf
GLIP 251
0.291
0.05
4.7
DESCO-GLIP
0.381
0.11
12.4
GLIP does not effectively utilize language descrip- tions. We make an attempt using descriptions to transfer GLIP to LVIS [11], which contains over 1,200 classes. The process is similar to that of (3I) For each category, prompt large language model (GPT-3) give details descriptions (as in Section We append the description to the original class name to form a new query. An example the queries can be seen shown in Figure (bottom row). Directly appending the description to the object name at inference time only degrades the performance: GLIP-T achieves 20.8 AP on rare categories while appending the descriptions makes the performance drop 12.2 AP. This is likely due to model hallucination on natural-language
Table 1: GLIP is insensitive to context changes compared to DESCO-GLIP.
queries.
GLIP is insensitive to context changes. Examining the model predictions, we find that the model not only does not utilize language descriptions; ignores descriptions and tends only focus on entity names, as we hypothesized. To quantitatively verify the phenomenon, we introduce context-sensitivig test, inspired by the WinoGround [43] benchmark. For each image, we provide the model with a positive guery q" describing query q describing a confusable object. The original object names are removed from the query. An example of the test is shown in Figure [ (bottom left), where the model is challenged to distinguish "mallet" and "ax". 4thh and q describe objects from the same general category (e.g., both are "a kind of tool") while differing in other aspects, similar to the Winograd test.
object that appears in the image and negative
Intuitively, a model can effectively utilize the descriptions, should exhibit two properties: 1) should give higher alignment scores to entities in gt compared to y 2) even if the model cannot "guess" the hidden entity, at least, the model predictions should change drastically when given two different descriptions. We thus introduce two metrics. 1) AP, which measures how accurate

---

LVIS VaniVal
17
OmniLabel [8
Model
Backbone
APr
APc
APi
AP
AP
APe
APd
APd-P
MDETR 17
RN101
20.9
24.9
24.3
24.2
4.7
9.1
MaskRCNN
RN101
26.3
34.0
33.9
33.3
RegionCLIP 57
ResNet-50
2.7
2.7
2.6
3.2
Detic 59
Swin-B
8.0
15.6
5.4
8.0
K-LITE 41
Swin-T
14.8
18.6
24.8
21.3
GroundingDINO-1
28
Swin-T
18.1
23.3
32.7
27.4
GroundingDINO-I
28
Swin-L
22.2
30.7
38.8
33.9
GLIP-I 25
Swin-L.
28.2
34.3
41.5
37.3
25.8
32.9
21.2
33.2
GLIP-T 25
Swin-T
20.8
21.4
31.0
26.0
19.3
23.6
16.4
25.8
DESCO-GLIP
Swin-T
30.8
30.5
39.0
34.6
23.8
27.4
21.0
30.4
FIBER-B [
Swin-B
25.7
29.0
39.5
33.8
25.7
30.3
22.3
34.8
DESCO-FIBER
Swin-B
34.8
35.5
43.9
39.5
29.3
31.6
27.3
37.7
Table 2: Zero-shot transfer to LVIS and OmniLabel. Numbers that grayed out supervised models. DESCO-GLIP and GLIP-T are directly comparable; DESCO-FIBER and FIBER-B are directly comparable; the rest are listed for reference and not directly comparable.
the model's predictions 2) ABox and AConf, which are the differences between the model's predictions for 9*t and q Aox measures the changes in box coordinates while AConf measures the changes in alignment scores of boxes. Details of the metries in the appendix.
We find that the baseline model not only cannot identify the correct description (low AP); but effectively ignores the language context (low ABox and AConf) (Table On average, the confidence of the predicted boxes changes only 0.05 between q and q One could see the examples in Figure I GLIP models make almost identical predictions for two different queries. Such insensitivity to language context makes it infeasible and unreliable to use descriptions to control model predictions.
4.2 Setup
In this section, we apply our approach to two lasion-language object detection models GLIP 25 and FIBER (7].
Models. The visual backbon GLIP and FIBER Swin Transformer [30 and the text backbones are BERT [6] for GLIP and RoBERTa (2 for FIBER. Both models use Dynamic Head [4] as the detection architecture. Built upon the two models, we introduce two model variants: DESCO-GLIP and DESCO-FIBER.
Datasets. Following GLIP 25 we train the models on 1) 0365 (Objects365 39 consisting of 0.66M images and 365 categories; 2) GoldG that is curated by MDETR 17 and contains 0.8M
images sourced from Flickr30k [35. Visual Genome 19 and GQA [15]: 3) CC3M 40 the web-scraped Conceptual Captions dataset with the same pseudo-boxes used by
GLIP. We down-sample CC3M to around 1.4M images save training costs, based on whether high-confidence boxes exist the image. As illustrated Section 3 we convert the text caption of each instance into a detailed language description to construct description-rich data.
To evaluate how well the models generalize to novel concepts, we perform zero-shot evaluation on the LVIS [II) and OmniLabel (38) datasets. LVIS is popular dataset that has over 1,200 object categories with challenging long tail of rare objects; OmniLabel recently proposed and focuses on object detection with diverse and complex object descriptions in a naturally open-vocabulay setting. For evaluation on LVIS, for each category, we append the GPT-3 generated description the category name; we group several descriptions into one query to save inference time. More details on evaluation are protocol without modifications. We also verify that the models still possess the ability to perform the conventional detection and grounding tasks as GLIP and FIBER, on COCO 26 and Flickr30K (35 The evaluation results are in the appendix.
appendix. For OmniLabel evaluation, we follow original evaluation
8

---

LVIS ViniVal
17
OmniLabel COCO
38
Context Sensitivity
Row Model
APr
APc
API
AP
APe
APd
APd-P
ABox AConf
AP
1 GLIPT
20.8
21.4
31.0
26.0
18.7
45.7
11.7
31.2
0.291
0.05
4.7
2 Description w/ Entity Name
20.5
23.9
35.5
29.2
23.6
47.4
14.7
36.0
0.293
0.06
5.7
+ Description w/o Entity Name
25.6
25.9
35.9
30.7
24.0
46.8
16.0
37.0
0.382
0.10
10.7
4 + Description w/o Name + Neg Cup
26.5
27.1
35.8
31.3
24.7
48.2
16.6
36.2
0.381
0.10
10.5
Table 3: Ablation study. Directly appending the description does not improve performance on rare categories (Row I Row 2, LVIS APr). Constructing context-sensitive queries crucial.
Implementation details. We initialize DESCO-GLIP from the GLIP-T checkpoint and DESCO- FIBER from the FIBER-B checkpoint. We fine-tune tune the models on both the original data and the new description-rich data. For DESCO-GLIP, we fine-tune with batch size of 16 and learning rate of 5 x 10-6 for 300K steps; for DESCO-FIBER, we fine-tune with a batch size of 8 and a learning rate of 10-5 for 200K steps. Experiments can be replicated with 8 GPUs each with 32GB memories.
4.3 Zero-shot transfer to LVIS and OmniLabel
LVIS. Our method shows notable improvements over the baselines on the LVIS MiniVal dataset (Table 2) The improvement is particularly prominent for rare object categories (APr), with an increase of 10.0 for GLIP and 9.1 for FIBER. Results on the Val 1.0 set are in the Appendix.
Omnil.abel Our method also shows improvements over baselines on the Omnil dataset (Ta- 2). OmniLabel assesses model performance using plain categories (APc), free-form descriptions (APd), and positive descriptions (APd-P). Because our models are trained with description data, they naturally excel in supporting such queries, leading substantial increases APd and APd-P compared to the baselines. Specifically, DESCO-GLIP achieves a notable improvement of +4.6, while DESCO-FIBER achieves an even more impressive improvement of +5.0. Furthermore, our model's effectiveness extends beyond free-form descriptions to plain categories as well, as illustrated in the table. This highlights the robustness of our method across different evaluation settings and its ability to achieve improvements in various types of queries. Our method wins the 1st place in the Omnilabel challenge 2023 all three tracks (see Appendix for details).
4.4 Ablation study
In this section, all ablation models are initialized from GLIP-T and trained for 100K steps.
Directly appending descriptions. We examine the impact of directly adding language descriptions to text queries, without incorporating context-sensitive query construction. The results are presented in Row of T The performance on rare categories (APr) sees no improvement but decreases. To further evaluate the sensitivity of the model to contextual changes, we conduct the same context sensitivity analysis as the one described in Section 4.1 The context sensitivity of the model almost remains unchanged (Row 1-2): ABox changes only 0.002 and AConf changes only 0.01. The results indicate that the model remains as insensitive to context changes as the baseline model. This suggests that the model struggles accurately interpret and effectively utilize the provided language descriptions when context-sensitive query construction is removed.
Dropping the entity name. As in Section 3.2.2
hypothesize that randomly removing the entity name can force the models concentrate the contextual in- formation. Remarkably, the results presented in Table (Row 2-3) demonstrate that this simple and intuitive approach proves to be highly effective. It significantly enhances the model's contextual sensitivity while con- currently improving object detection performance.
GPT
APr
APe
APf
AP
ada babbage
19.9
23.2
33.7
28.0
24.2
26.7
36.5
31.3
curie
24.7
28.4
38.2
32.8
davinci
30.8
30.5
39.0
34.6
Table 4: Detection performance improves when language model size grows.
Negative captions. We also investigate the effective- ness of using language models to generate hard negative

---

Scarecrow: 0.38
Rollerblade: 0.37
Rallerblade: 0.40 re
From GPT-Curie:
From GPT-Davinci:
From GPT-Curie:
From GPT-Davinci:
Scarecrow, kind of object,
Scarecrow, kind of decoration, made of straw, has hat and clothes, could have face.
Rollerblade, kind of sports equipment, wheels attached to boot, used skating
Rollerblade, kind of sports equipment, blades that rotate on the ground
tall, with straw in its mouth, could have hat, could be made of straw.
Figure 4: Detection performance of DESCO-GLIP improves when given better descriptions. GPT- Curie a smaller model than GPT-Davinci; gives less accurate descriptions for objects.
captions. As shown in Row 4 of Table including neg- ative captions can improve the model detection performance across datasets while preserving its robust contextual comprehension. These results indicate that this technique effectively enhances the model's s ability to grasp the subtleties embedded the given language descriptions.
Language description quality. We explore the effect of the language model size on detection performance. We evaluate the pre-trained DESCO-GLIP on LVIS with descriptions generated from the GPT familie As presented in Table higher-quality language models improve object detection performance. This finding highlights the importance of employing strong language models, as they possess the ability to embed valuable visual information through extensive pre-training. We showcase two examples in Figure
5 Conclusion and limitations
In this study, introduced new paradigm of learning object detection models from language supervision. We show that large language models can be used to generate rich descriptions and the necessity to construct context-sensitve queries. We hope that our method sheds light empowering vision models with the ability to accept flexible language queries.
While we greatly improve the models' ability to understand flexible language queries, our method has several limitations that can be addressed in future work. 1) We use a large language model to automatically generate the descriptions, which inevitably introduces noise as not all generated descriptions are accurate or beneficial for representation learning. Future work could consider automatically selecting useful descriptions sampled from the language model, similar The format the descriptions we explored is still limited (e.g., "[entity], J. a kind of (type), (list of simple features] ")) it might be useful to consider more diverse descriptions by prompting the language model with more diverse prompts. 3) Similar to large language models, querying our model also requires certain amount of prompt engineering. Future work could explore how make the model more robust to different kinds of queries.
[49]. 2)
Acknowledgement
We would like to thank members of UCLA NLP for their helpful comments. We also thank the reviewers for the valuable reviews. This work supported in part by DARPA MCS program under contract number N660011924033 DARPA ECOLE program No. HR001 12390060, and ONR grant 10014-23-1-27801 We also thank Amazon for AWS credits for computational resources. The views expressed are those the authors and do not reflect the official policy or position the Department of Defense the U.S. Goverment.
10

---

References
[1] Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. ECCV, 2018.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. In NeurlPS, 2020.
[3] Zhaowei Cai, Gukycong Kwon, Avinash Ravichandran, Erhan Bas, Zhuowen Tu, Rahul Bhotika, and Stefano Soatto. X-DETR: A versatile architecture for instance-wise vision-languas tasks. In ECCV, 2022.
[4] Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang. Dynamic Head: Unifying object detection heads with attentions. In CVPR, 2021.
[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hicrarchical image databasc. In CVPR, 2009.
[6] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[7] Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng. Jianfeng Gao, and Lijuan Wang. Coarse-to-fine vision-language pre-training with fusion in the backbone. NeurlPS, 2022.
[8] Sivan Doveh, Assaf Arbelle, Sivan Harary, Eli Sehwartz, Roci Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Lconid Karlinsky. Teaching structured vision & language concepts to vision & language models. In CVPR, 2023.
[9] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung Yi Lin. Open-vocabulary image segmentation. In ECCV, 2022.
[10] Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR, 2022.
[11] Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In CVPR, 2019.
[12] Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. Towards general purpose vision systems: An end-to-end lask-agnostic vision-language architecture. In CVPR, 2022.
[13] Graeme Hirst. Anaphora in natural language understanding: survey. 1981.
[14] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
[15] Drew A. Hudson and Christopher D. Manning. GQA: dataset for compositional question answering over roal-world images. In CVPR, 2019.
[16] Dat Huynh, Jason Kuen, Zhe Lin, Jiuxiang Gu, and Ehsan Elhamifar. Open-vocabulary instance segmenta- tion via robust cross-modal pseudo-labeling. CVPR, 2022.
[17] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. MDETR-Modulated detection for end-to-end multi-modal understanding. In CVPR, 2021.
[18] Amita Kamath, Christopher Clark, Tanmay Gupta, Eric Kolve, Derek Hoiem, and Aniruddha Kembhavi. Webly supervised concept expansion for general purpose vision models. In ECCV, 2022.
[19] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017.
[20] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[21] Boyi Li, Kilian Q Weinberger, Serge Belongic, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In /CLR, 2022.
[22] Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang. and Jianfeng Gao. Multimodal foundation models: From specialists ceneral-purposs assistants. arXiv preprint, 2023.
11

---

[23] Chunyuan Li. Haotian Liu, Liunian Li, Pengchuan Zhang. Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu. Zicheng Liu, Yong Jae Lee, et al. ELEVATER: A benchmark and toolkit for evaluating language- augmented visual models. NeurlPS, 2022.
[24] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang. Chunyuan Li, Zhang, and Jianfeng Gao. Semantic-SAM, Segment and recognize anything at any granularity. arXiv preprint, 2023,
[25] Liumian Harold Li, Pengchuan Zhang. Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training, In CVPR, 2022.
[26] Tsung- Yi Lin, Michael Maire. Serge J. Belongie. James Hays. Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
[27] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint, 2023.
[28] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding DINO: Marrying dino with grounded pre-training for open-set object detection, arXiv preprint, 2023.
[29] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemayer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint, 2019.
[30] Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[31] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLRR 2023.
[32] Matthias Minderer. Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Doso- vitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open- vocabulary object detection with vision transformers. In ECCV, 2022.
[33] Roozbeh Mottaghi. Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Wiin Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, 2014.
[34] Cory Paik, Stéphane Aroca-Ouellette, Alessandro Roncone, and Katharina Kann. The world of an octopus: How reporting bias influences a language model's perception of color. In EMNLP, 2021.
[35] Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k Entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In ICCV, 2015.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[37] Karsten Roth. Jae Myung Kim, A Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. ICCV, 2023.
[38] Samuel Schulter, Yumin Suh, Konstantinos M Dafnis, Zhixing Zhang, Shiyu Zhao, Dimitris Metaxas, et al. OmniLabel: A challenging benchmark for language-based object detection. arXiv preprint, 2023.
[39] Shuai Shao. Zeming Li, Tianyuan Zhang, Chao Peng. Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In /CCV, 2019.
[40] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACI, 2018.
[41] Sheng Shen, Chunyuan Li, Xiaowei Hu. Yujia Xie, Jianwei Yang, Pengchuan Zhang, Zhe Gan, Lijuan Wang, Lu Yuan, Ce Liu, et al. K-LITE: Learning transferable visual models with external knowledge. In NewrlPS, 2022.
[42] Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. NewrIPS, 2013.
12

---

[43] Tristan Thrush. Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In CVPR, 2022.
[44] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. arXiv preprint. 2023.
[45] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal sew-shot learning with frozen language models. NeurIPS, 2021.
[46] Wenhai Wang. Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng. Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. VisionLLM: Large language model is also an open-ended decoder for vision-centric tasks, arXiv preprint, 2023.
[47] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. GroupViT: Semantic segmentation emerges from text supervision. In CVPR, 2022.
[48] An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He. Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In ICCV, 2023.
[49] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison- Burcc, and Mark Yatskar. Language in Bottle: Language model guided concept bottlenccks for interpretable image classification. In CVPR, 2023.
[50] Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. DetCLIP: Dietionary-enriched visual-concept paralleled pre-training for open-world detection. NearlPS, 2022.
[51] Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn, Meta-learning without memorization. In ICLR, 2020.
[52] Licheng Yu. Patrick Poirson. Shan Yang, Alexander C. Berg, and Tamara L. Berg, Modeling context in referring expressions. In ECCV, 2016.
[53] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why Dision-language models behave like bag-of ofg-ds-words models, and what to do about it? In ICLR, 2023.
[54] Hao Zhang, Feng Li, Xueyan Zou. Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang, A simple framework for open-vocabulary segmentation and detection. In ICCV, 2023.
[55] Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang. Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. GLIPy2: Unifying localization and vision-language understanding. NeurlPS, 2022.
[56] Tiancheng Zhao, Peng Liu, Xiaopeng Lu, and Kyusong Lee. OmDet: Language-awarr aware object detection with large-scale lisionllagguage multi-dataset pre- pre-training. arXiv preprint, 2022.
[57] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. RegionCLIP: Region-based language-iman pretraining. In CVPR, 2022.
[58] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
[59] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty- thousand classes using lmage-level supervision. In ECCV, 2022.
[60] Deyao Zhu. Jun Chen, Xiaoqian Shen, Xiang Li. and Mohamed Elhoseiny. MiniGPT-4: Enhancing vasion-language understanding with advanced large language models. arXiv preprint, 2023.
[61] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai. Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel. image, and language. In CVPR, 2023.
[62] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. arXiv preprint, 2023.
13

---

A Approach
A.1 Description generation with large language models
We prompt davinci-003 with the following prompt:
Question:
What features should object detection models focus given input? Answer:
Input:
zucchini, Output: "type": "vegetable", "description": "cylindrical, green, smooth: could have brown and rough stems; could be sliced into round pieces; could have green leaves", "similar objects": ["cucumber", "eggplant", "green bean"]
Input: zebra, Output: "type": "animal", "description": "black white stripes; has a long mane", "similar
objects": |"horse", "girafte", "elephant")
Input: apple, Output: "type": "fruit", "description": "red, round, has stem", "similar objects": I"orange", "bamama", "pear"l
Input: wok, Output: "type": "cooking tool", "description": "round, deep, has handle", "similar objects":
p"pan". "pot". "frying pan")
ambulance, Oulput: "type": "vehicle", "description": "red; has glaring siren; could with stretcher", "similar objects": I l'police car", "taxi", "garbage truck"] lantern, Output: "type": "lighting tool", "description": "round; could be made of papers". "similar objects": : I'lamp,, "flashlight". "candle"] Input: (entity)
Input:
Input:
Table 5: Text prompt used to sample descriptions from large language models.
We construct a vocabulary of 10K entities by extracting noun phrases from the pre-training text corpus (GoldG and CC3M) using NLTK. As we also use Objects365 for training, we add the categories from Objects365 to the entity vocabulary. We query the language model to generate descriptions for the entities.
A.2 Context-sensitive query construction
As shown in Table []] when prompting the language model, we also ask the model to name few "similar objects". Thus, when constructing query of descriptions, we include both positive descriptions and several negative descriptions for such "similar objects". GLIP has a max query length of 256 tokens. On average, can pack 8 descriptions into query. We randomly drop descriptions if the length exceeds the length limit.
For image caption data, we use to prompt in Table generate negative captions, Note that while the prompt asks the LLM to generate object descriptions and confusable objects, we find that the prompt in Table [able(5ggin givee better results for this purpose; thus we do not use the description and confusable objects given by prompt Table 6
A.3 Query construction
Algorithm Generating Queries for
Algorithm Generating Queries for Grounding Data
Delection Data
Input: T, F.E. V
Input: D.C
Q -- RandSample(V > E)
Q + RRandSample(D) \ (C))
2: q- + ShuffleConcate( E U Q-)
2: q" + ShuffleConcate( UC)
3: q" + LabelAssign(y" T. E)
3: q" T + LabelAssign(q" .P.
Figure Algorithms for generating queries from detection data and grounding data for GLIP, Algorithm 1: Compare to DesCo, note that no positive entities are dropped from the query; thus B is involved in the query construction, Algorithm' D is the corpus image captions. Compare to DesCo, note that the positive caption C' is always included in the query; this creates the ballucination issue.
14

---

Given a caption, perform the following:
1. Extract the visually concrete objects and give deseription for what object detection models should focus
on for given input. Avoid common objects such as "person". "dog". "actor". "cat", "man", "woman", "boys", "girls".
2. Name some negative visual objects can be confused and give deseriptions. Focus on the visual differences between the positive and negative objects.
3. Write some hard negative captions for given positive caption, by replacing the nouns, adjectives, and verhs with visually similar but wrong words. Do not replace the nouns that refer to people.
Positive caption: an ambulance is rushing down the street with its lights flashing
1. Visually concrete objects and descriptions: "ambulance": "a kind of car, red light, red and white patterns on the hody"
2. Objects that can he confused with the above objects: "police car": "a kind of car, blue and red glarring light, blue and white patterns". "taxi": "a kind of car, yellow, has a notch"
3. Negative captions: ["a police car is rushing down the street with its lights flashing".
Positive caption: the aircraft carrier is enveloped in fog as it sits in its berth
1. Visually concrete objects and descriptions: "aircraft carrier": "a kind of ship. large, has a long deck, has runways, has airplanes on the deck
2. Objects that can be confused with the above objects: "cruise ship": "a kind of ship, ususally white, luxury, has swimming pools", "cargo ship": "a kind of ship, has boxes on the deck"
3. Negative captions: |'the shipping boat is enveloped in fog as it sits in its berth", "the cargo ship is enveloped
in fog as it sits in its berth"]
Positive caption: the beans in the red silicone cup
1. Visually concrete objects and descriptions: "beans": "a kind of vegetable, small, round, usually green"
2. Ohjects that can he confused with the ahove ohjects: "apples": "a kind of fruit, usually red", "heeds": "a kind of decorations, small, round, colorful"
3. Negative captions: ('the beans in the green silver cup.", "the apples in the red silicone cup.", "the beans in the red porcelain cIp."]
Positive caption: bulldozer on a building site 1. Visually concrete objects and descriptions: "bulldozer": "a kind of heavy machinery, has large blade in front, used for pushing debris or soil" 2. Objects that can be confused with the above objects: "excavator": "a kind of heavy machinery, has long arm and scoop, used for digging and moving materials", "crane": "a kind of heavy machinery, has long arm and hook, used for lifting"
3. Negative captions: l'crane on a demolition site", "backhoe on a construction site"|
Positive caption: A girl standing in a shallow creek, wearing stilts
1. Visually concrete objects and descriptions: "stilts": "a kind of shoes, has two long poles"
2. Objects that can he confused with the above objects: "high heels": "a kind of shoes, has a high heel", "sandels": "a kind of shoes, has strap, usually worm in summer"
3. Negative captions: ["A girl standing in a shallow creek, wearing sandels", "A girl rumning in deep river,
wearing stilts"|
Positive caption: (caption)
Table 6: Text prompt used to sample negative captions from large language models.
B Experiments
B.1
Context-sensitivity test
To create the test set, we go over the LVIS validation set and for each image that has a rare object (defined by the LVIS taxonomy), we query the model with qt (the description of the rare object, without the object name) and a q (the description of confusable object, without the object name). The confusable object comes from prompting the LLM as shown in Table 5 Then we calculate the difference between the predictions made by the model for q+ and q
For q* and qT, the model will give two sets of predictions (two lists of boxes). We first match the two lists of boxes by their loU overlap. Then ABox is the percentage boxes that have high IoU overlap (>0.5) between the two sets of predictions; AScore is the confidence score changes for those matched boxes. With higher ABox and AScore, the model predictions change more drastically. The evaluation data and code will be released upon acceptance.
15

---

B.2 Evaluation on COCO and Flickr30K
COCO
Flickr30K Val
Model
mAP
R@D 1
R@5
R@10 10
GLIP-T
44.6
83.6
95.8
97.3
DESCO-GLIP
45.8
85.3
95.8
97.3
FIBER-B
49.3
87.1
96.1
97.4
DESCO-FIBER
48.9
86.9
96.4
98.0
Table 7: DESCO-GLIP and DESCO-FIBER maintain similar performance to GLIP and FIBER on common object detection and phrase grounding. DESCO-GLIP and DESCO-FIBER are fine-tuned with smaller batch size than GLIP and FIBER; thus some minor performance drops are expected.
In Table 7 we report the performance on common object detection (COCO, zero-shot) and Flickr30K. The performance of DESCO-GIIP and DESCO-FIBER is similar to GLIP and FIBER. GLIP-T" the checkpoint that achieves the best performance on LVIS released in https://githui: com/mi wwwccroootttiiii it has slightly worse performance than GLIP-T on cOCO. We used GLIP-T* to initialize DESCO-GLIP, thus we compare with GLIP-T in this case.
B.3 Evaluation on LVIS
LVIS Val
Model
APr
APe
APF
AP
GLIP
10.1
12.5
25.5
17.2
DESCO-GLIP
19.6
22.0
33.6
26.2
FIBER
18.0
21.6
35.0
26.3
DESCO-FIBER
23.0
26.3
38.5
30.5
Table 8: DESCO-GLIP and DESCO-FIBER achieve strong performance LVIS val 1.0.
For evaluation on LVIS, for each category, we generate descriptions with prompts in Table 6 When doing inference into a long prompt results a extremely long prompt. We adopt the same approach as GLIP and FIBER; we construct multiple prompts where each prompt only contains few category descriptions; the model is queried multiple times and we aggregate the outputs across the multiple forward passes.
image, as the vocabulary is large, concatenating descriptions of all categories
We evaluate models on the full validation set of LVIS using the fixed AP protocol. It can be seen that our approach achieves large performance gains compared to the baselines.
B.4 OmniLabel Challenge 2023 winning entry
We
submit DESCO-FIBER to the OmniLabel Challenge 2023 and won the 1st place. Different from the model reported in the main text, the submitted version is first initialized from RESCo-FIBER, then undergoes second-stage pre-training on Objects365, GoldG, CC3M, and RefCOCO 52 (including RefCOCO, RefCOCO+, RefCOCOg), with batch size of 8 for another 200K steps.
16


Reviews:
[{"review_id": "K5Y06vyqfA", "reviewer": "Reviewer_Wg2t", "summary": "This work revisits a key aspect of open-vocabulary object detection: what makes for the class names? It shows that GLIP, the state-of-the-art vision-language model for open vocabulary object detection, behaves like bag of words and could not effectively understand text descriptions of the objects to detect. To tackle this challenge, it turns to LLMs to extend the class names to detailed descriptions to solve the label scarcity issue. It also creates hard negative descriptions to ensure the model focuses on the context. Experiments on two challenging benchmarks (LVIS & OminiLabel) shows strong results and validates the effectiveness of this work.", "questions": "- More examples, especially failure cases of the LLM-generated descriptions could better help understanding the model.\n- Currently, the text seems to be more like a general description in terms of representative visual attributes (low-level) and semantic/meaning of the object (higher-level). One interesting direction is to create more diverse descriptions in different aspects (lower/higher-level), and diagnosis the contribution of each. Another question is how well could such fixed text descriptions cope with diverse (or OOD) visual appearances.", "limitations": "limitations are discussed in the last section", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "*Originality*: Though the high-level idea of extending the understanding of class names with LLMs shares similarity with [a], the application of this work is fairly original in this field.\n\n*Quality*: This work is well-motivated via re-evaluation of GLIP. The idea is practically simple and clear, and its effectiveness is well-supported by strong performance on two challenging benchmarks. Overall it is insightful and well-shaped for acceptance.\n\n*Clarity*: The delivery is fairly smooth and clear, and I find no major issues in understanding this paper.\n\n*Significance*: Recent works in open-vocabulary object detection still focus on mining region-word (noun) correspondence. I believe the message/intuition from this paper could be helpful for the community.\n\n*Reproducibility*: Code & data release could be very helpful. Release upon acceptance is promised.\n\n[a] Menon & Vondrick, Visual Classification via Description from Large Language Models, ICLR 2023.", "weaknesses": "I did not find major concerns in this work. See next section for questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the view!\n\n1. Failure cases of the LLM-generated descriptions\n\nThank you for the suggestion! In Figure 4, we included failure cases of the smaller LM; but indeed even the largest LM can generate inaccurate descriptions. We will include examples in the next version.\n\n2. More diverse descriptions\n\nThank you for this valuable suggestion! As a first paper on this topic, we opted for a rather simple description style and left the exploration of different description styles to future work.\n\n3. How well could fixed text descriptions cope with diverse (or OOD) visual appearances?\n\nIndeed objects will have various appearances that are hard to summarize / OOD for fixed descriptions. Our assumption is that during training, the descriptions given by LLM should align well with the objects in most cases and the model learns the correct region-word correspondence; during test time, users could hand-craft OOD queries for OOD appearances when in need. E.g., “Detect a chair with three legs”."}, {"review_id": "rBGhmMreZ2", "reviewer": "Reviewer_BSUS", "summary": "The paper tackles the problem of open-world (language-based) object detection and proposes two main things. First, it uses an LLM to obtain detailed descriptions of the object classes of interests and, secondly, proposes a training methodology that forces the network to pay attention to such descriptions through the use of (language) hard negatives.", "questions": "\nI have no qualms with the paper as is. A couple minor things the authors might (or might not) find useful:\n\n- I like the results split for \"rare\" vs \"frequent\" objects, I think it is quite informative, but it is just mentioned in passing. Adding this split in a table and be more complete might be useful? More generally, some insight into what type of categories benefit the most (as in, finer-grained? ambiguous?...) and which ones don't would improve the paper. \n\n- the hard negatives technique and the explanations are a hot topic so there's several recent papers. I won't add many arxiv papers, but here's two the authors might find interesting, exploring somewhat similar strategies (though for different problems):\nCVPR'23 - Teaching Structured Vision & Language Concepts to Vision & Language Models\nfrom arxiv: Waffling around for Performance: Visual Classification with Random Words and Broad Concepts", "limitations": "yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Good paper overall, ticks all boxes. It uses some convincing advanced baselines in a very interesting and current setting, identifies shortcomings, proposes a reasonable solution, and shows the effectiveness well. ", "weaknesses": "Nothing important to say. As a minor comment, it is a bit hard to be sure about the setting the paper is tackling. The title talks about \"object recognition\", but the problem is object detection. Then it mentions grounding too. And it is not clear where the bounding boxes come from. Are they proposed by the network? are they given? (one could go to the glip paper, but better be self-contained). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review!\n\n1. Clarification on the setting\n\nThank you for the suggestion!  We used “object recognition” as an umbrella term because we believe the proposed technique can be applied to other vision tasks as well. \n\nIndeed the specific experiment setting is object detection. We followed the GLIP approach so grounding and detection are effectively the same task – locating entities mentioned in a text prompt; thus we abused the terminology. The bounding boxes are predicted by the network; it is an end-to-end detection system without relying on pre-computed boxes.\n\nWe will revise the paper to make the settings clearer.\n\n2. \"Rare\" vs \"frequent\" objects\n\nThank you for the suggestion. The rare/common/frequent split is following the definition in the LVIS dataset. The APr/APc/APf refer to AP for the rare/common/frequent categories. \n\nIt might be hard to completely separate the improvement for rare and common categories as they can be interlinked. We observe that when our model learns to detect rare objects better, it is also less likely to mistake some rare objects as other confusable common objects; thus APc and APf will also improve with better rare object recognition ability due to less false-positives for the common/frequent categories.\n\n3. Thank you for the references! We will update the references to include these papers.\n"}, {"review_id": "XFfIQLxDdH", "reviewer": "Reviewer_hkVU", "summary": "This paper leverages rich descriptions generated by LLMs to encourage GLIP/FIBER to pay more attention to object attributes aside from object names when performing detection tasks. A more robust model is resulted from training on object-description alignment as the model is able to leverage attributes for recognition as well as avoiding confusable objects.\n\nExperiments show that key to the success of this method is the inclusion of hard negative descriptions without object names, in order to effectively motivate the model to use the information in addition to object names. By doing so, the model also overcomes one of the previous limitations where a detection model is always predicting objects even though the description mentions non-existing objects in the given image.", "questions": "In Table 3, what is APd-P as apposed to APd?", "limitations": "More experiments and analyses could be done to better justify the proposed training method.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper leverages hard negatives to force a VL model to learn the object-attributes alignment in addition to merely mastering object-noun alignment. This brings significant improvement in zero-shot recognition of long-tail objects.\n2. This paper enriches captions in existing image-text datasets with large language models. Those enriched text descriptions can be used to train better language-aware vision models.\n3. Overall, this paper demonstrates a viable path of distilling the knowledge present in LLMs into better object grounding abilities in VL models.", "weaknesses": "1. There lacks a test for the false-positive rate. The proposed training setup with carefully hard negatives is alleged to address the issue mentioned in line 181-184. However, there lacks evaluation on how much improvement there is with respect to hallucinating entities that are mentioned by the description but don't exist in the image.  \n2. There lacks a test for how well the model is able to \"reject\" a grounding attempt based on misaligned attributes. For example, when the image contains a red umbrella, Line 195-196 (\"Note that the label of the word mallet is affacted by the context\") suggests that the expected behavior is to ground \"red umbrella\" while not predicting any grounding if the text says \"yellow umbrella\", even though the entitiy name is correct. It'd be helpful to evaluate how well the model captures the nuances exemplified by these two cases. \n3. Lack of error analysis. For example, in what cases do the relative confidence scores for Q+ and Q- flip after Desco training?\n4. As shown in Figure 3, although the descriptions might contain multiple entity names, it seems that the target entity is always the first token, while entities appearing later in the description are supposed to be \"modifiers\". Would this bring an undesirable bias? How well does the model perform if the target entity appears in a different location in the description?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and constructive comments.\n\n**1. False-positive rate / reject grounding attempt / error analysis**\n\n&nbsp;\n\nA. The main metric AP penalizes false positives\n\nWhile we did not explicitly include a false positive metric, the AP metric we used already penalizes false positives. During evaluation, we will feed the model **both positive and negative** queries; the model will predict boxes and their alignment scores to all queries; we draw the precision-recall curve considering boxes for both positive and negative queries and then calculate Average Precision (AP). \n\nThe model will be penalized on AP if it does not successfully reject negative queries. If it outputs a lot of high-confidence boxes for negative queries, such false-positive boxes will out-rank other true-positive boxes and this would bring down its precision at a given recall level.\n\n&nbsp;\n\nB. Difficulty of designing a comprehensive “flip rate” metric\n\nWe also considered designing a “flip rate” metric (how often “the relative confidence scores for Q+ and Q- flip”) during development of the project but we find that this will lead us back to the classical AP metric. \n\nTo elaborate, we find it hard to define a clear-cut flip-rate metric; for one image, the model will output many bounding boxes for both Q+ and Q- and there are often multiple ground-truth bounding boxes; it is hard to decide when the predictions are flipped when we have to consider the IoU matching between predicted boxes and ground truth boxes. \n\nIn addition, oftentimes, the system improves by reducing its confidence for Q- as well as suppressing wrong boxes for Q+, which is hard to capture via a “flip rate” metric. For example, in Figure 1 top-left, the GLIP model gives Q+ (“eclair”) 0.58 confidence and Q- (“tart”) 0.57, while our system gives Q+ 0.37 while completely suppressing Q-. However, this does not constitute a “flip” as for GLIP, conf_Q+ > conf_Q-. For another instance, in Figure 1 bottom-left, GLIP gives multiple wrong boxes for Q+ while DesCo does not; such improvement is also hard to evaluate using a “flip-rate” metric. But the AP metric will capture and reward such improvement.\n\nBased on the above reasons, we decided to use the standard AP metrics. We add two new metrics in Section 4.1, \\delta box and \\delta conf, which measures how much the prediction changes between Q+ and Q-; we believe that combining these with the AP metric can show that DesCo indeed enables context-conditioned detection.\n\nWe are happy to discuss the metric design if the reviewer has specific recommendations. We can also add more qualitative examples to showcase such “flip” in the next version.\n\n&nbsp;\n\n&nbsp;\n\n**2. “The target entity is always the first token”**\n\nThank you for pointing this out. Indeed this could bring a bias towards the first entity token. We believe this could be mitigated by including more diverse formats of descriptions (e.g., where the target entity appears in a different location), which we leave for future work.\n\n&nbsp;\n\nQuestion\n\nThe APd-P metric is introduced in the OmniLabel paper that only feeds the model positive queries (no negative queries). Thus the model will not be penalized from false-positives on negative queries.\nIn fact, in Table 4, comparing line 4 and line 3, the APd-P drops slightly but APd increases; this means that with hard negatives, the model more effectively rejects negative queries.\n"}, {"review_id": "THucSZVJo4", "reviewer": "Reviewer_xbf9", "summary": "This paper is about open-vocabulary object detection, in which a model is given an image and a sequence of object descriptions and has to predict bounding boxes matching the descriptions. While open-vocabulary detectors can in principle take arbitrarily detailed object descriptions as input, they tend to focus on the main noun in the description and ignore further details in the description. This is problematic at inference time, where the model might be prompted with descriptions of objects that it has not seen during training, and where leveraging additional detail in the description becomes necessary to solve the task.\n\nThe present paper addresses this issue by augmenting the training data with detailed object descriptions and then forces the model to use these descriptions.\n\nFirst, standard object detection datasets are augmented with detailed object descriptions by querying a large language model (GPT-3) to provide additional details about object categories occurring in the detection datasets. The language models are also used to generate hard negative descriptions for similar object categories.\n\nSecond, open-vocabulary object detectors are trained using these descriptions. By combining positives with easily confusable negative descriptions, the model is forced to attend to the details in the description. A crucial detail is that the description should not explicitly name the target object type, such that the model is forced to use the description to solve the task.\n\nUsing two off-the-shelf open-vocabulary object detectors (GLIP and FIBER), the experiments show that the proposed data augmentation improves performance on the LVIS and OmniLabel benchmarks.\n\nAblations show that the biggest boost in performance comes from adding detailed descriptions that do not explicitly name the target entity to the training data. Hard negatives provide a small additional boost.\n", "questions": "Questions\n\n1. LVIS evaluation (Appendix B.3): What exactly do you mean by “fixed AP protocol”? Do you mean the evaluation proposed in https://arxiv.org/abs/2102.01066? That paper makes various recommendations and doesn’t specify a precise protocol. How exactly do you modify the standard LVIS AP metric to “fix” it? Please provide code or a detailed descriptions sufficient for reproduction.\n\n2. Suggestion for improving the paper: Build on the unique possibilities afforded by LLMs and study those more. Scale up, study prompt choices, study description diversity etc.\n\nTypos\n\n* Typo in abstract: OminiLabel.\n* L92 missing word: “…pretrains models large-scale image-caption pairs to learn transferrable representations.”\n* Appendix L37 typo: “protocal”", "limitations": "The paper does not discuss limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Natural language supervision is an important trend. Therefore, general ideas for how to clean or augment natural language to provide better supervision are valuable. The idea to remove the entity name from descriptions goes in that direction.\n2. The analysis of how detection performance is affected by the LLM used for data generation (Table 4) is significant because it shows that language quality is an important determinant for downstream task performance. This is useful guidance for other works using LLMs for data generation.\n", "weaknesses": "1. The contributions are insufficient for a NeurIPS paper. The primary contribution is to apply the Winograd format to query construction for object detection. While it is useful to know that this helps in the case of object detection, the idea has been employed extensively in related contexts. Further, using an LLM to generate the object descriptions is not well motivated and not necessary at the scale explored in the paper. Datasets of object descriptions of the size used here (~10000 entities) already exist; it’s not necessary to use an LLM for that. There are many interesting questions that could be explored to make the contributions more substantial. For example, does performance improve when scaling up to larger numbers of object descriptions, which is possible with LLMs? Does the LLM prompt matter? Could similar results be achieved with existing datasets by simply masking the entity name?\n\n2. The paper lacks detail in the methods and experiments. The proposed approaches could not be reproduced from the given descriptions. For example, much more detail is needed about the generation of negative captions. The paper briefly mentions that negative captions are generated using a multi-stage process that involves first prompting a language model, then training a second language model to generate more data. But no details are given about how the first LLM is prompted or how the second one is trained. Please describe this process in sufficient detail to reproduce it.\n\n3. The paper makes some misleading claims. For example, L279 claims that the proposed method “surpasses supervised models by a substantial margin”. The models labeled as “supervised” in Table 1 are very far from the state of the art of supervised LVIS detectors, which reach around 60 AP and 50 APr. This is therefore a meaningless and misleading comparison.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review. \n\n1. “Insufficient Contributions”\n\nWe list our core contributions in the general response, and we would like to emphasize the following points.\n\nAs Reviewer Wg2t pointed out, DesCo is the first approach to support complex-language based detection, while “recent works in open-vocabulary object detection still focus on mining region-word (noun) correspondence”. Our paper solves **an important and timely problem** of how to improve detection systems with language instructions. This problem is featured in the recent OmniLabel Challenge 2023 at CVPR and our system achieves the best performance. As we showed in the ablation study, the improvement is not only from engineering efforts but the novel design of our training algorithm. \n\nWhile there are some arxiv/contemporary papers that touch on ideas of using hard negative examples (as noted by Reviewer BSUS), they mostly focus on image classification. We study a more difficult problem (detection/dense prediction).  The model is essentially asked to perform classification over thousands of potential regions for one image (in contrast to outputting one label per image); it is thus more prone to output false-positive boxes for negative queries. And our approach successfully reduces such hallucinations.\n\nWe are happy to give a more in-depth discussion and comparison if the reviewer could give pointers to relevant literature.\n\n&nbsp;\n\n2. \"Using LLMs is not necessary\"\n\nLLM is an easy and scalable tool to obtain training descriptions of desired style. One relevant large-scale “description” dataset is WordNet, which does not include detailed visual specifications. For example, for the concept “eclair”(Figure 1 top-left), the WordNet definition is “oblong cream puff” while LLM gives “a kind of food, long, cylindrical pastry, filled with cream, topped with chocolate”. In addition, the 10K entities are automatically extracted from the grounding+CC3M datasets; many of the entities are multi-word phrases (e.g., “stop sign”) which do not have definitions in WordNet. Any additional pointers to related datasets are welcome and we will cite and discuss them in the paper.\n\nLLM also allows easy transfer to downstream tasks where we can query any novel test concepts on the fly. In fact, for the main evaluation benchmark LVIS, we do not explicitly make sure the evaluation categories are included in the pre-training description corpus; rather, by word matching, only 699 out of 1200 LVIS categories appear in our 10K entity list; this again shows the flexibility our approach offers. Thus, replacing the LLM in our paper with existing hand-labeled resources would involve significant efforts in data curation and taxonomy mapping.\n\n&nbsp;\n\n3. Detail in the methods and experiments\n\nThank you for the suggestions. We have added the prompt used to generate the negative captions (Table 1) as well as the negative augmentation algorithm (Figure 1) in the attached PDF file. We will also open-source the code for reproducing the experiments.\n\n&nbsp;\n\n4. Misleading claims\n\nThank you for the comment. We will remove this claim and re-exam other claims.\n\n&nbsp;\n\nResponse to questions:\n\n1. Thank you for checking the cited paper. The fixed AP metric is included into LVIS Challenge 2021 as an official metric (https://www.lvisdataset.org/challenge_2021) and we used the implementation provided by Dave et al. (master). The evaluation metric is also adopted by MDETR, GLIP, FIBER, DetCLIP. We will make the evaluation protocol more clear in the next version.\n\n2. Thank you for suggesting these interesting directions. As a first paper working on complex language-based detection, we focus on reducing the bag-of-words/hallucination problem and opted to use a rather general prompt/description style, following Menon et al. Thus, we left the exploration of different prompts to future work; the fact that our approach works without extensive prompt engineering also speaks to its effectiveness.\n\nMenon & Vondrick, Visual Classification via Description from Large Language Models, ICLR 2023.\n"}, {"review_id": "0V77Nt4tEo", "reviewer": "Reviewer_9ZAR", "summary": "The paper describes an approach to open world object detection that generates detailed object descriptions with GPT to improve performance. The main advance of the paper is showing that the removal of the entity name, and just using it's description, forces the network to learn the \"context\". This is turn leads to better performance on zero-shot tasks. Results are shown on the LVIS and OmniLabel datasets in comparison to GLIP and FIBER.", "questions": "Where the COCO images removed from training, similar to CLIP?\n\nWhen generating hard negatives (Line 194), how can you ensure they're actually negative?\n\nThe example of the mallet in the paper states a mallet has a wooden handle, but the image contains a mallet with a red handle (could be made from any material). Is the model really learning a grounding for the context, or just querying with a bag of words?\n\nOn line 176, it's stated the \"mutual information still stays zero\". Is this true? The mutual information may be low, but is it really zero? ", "limitations": "The paper does not provide potential negative societal impact of their work. \n\n", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper clearly demonstrated that the additional contextual information needed to be presented to the network properly to see a performance boost.\n\nThe authors' results are clearly better than FIBER and CLIP. ", "weaknesses": "The writing of the paper make it difficult to understand the approach. I had to read it multiple times to understand how the models were trained and tested. It's still not 100% clear how all the diverse datasets used for training where converted to description-rich data. The paper states \"As illustrated in Section 3, we convert the text caption of each instance into a detailed language description to construct description-rich data.\". This should be made more concrete.\n\nThe paper doesn't describe the training of CLIP and FIBER. A little more background would help better differentiate the contributions of this paper.\n\nThe main novelty of the paper is removing the entity name from the description to force the network to use the contextual description. It is not clear this is of sufficient novelty/interest for NeurIPS. \n\nThe authors don't explore other prompts for generating object descriptions. This would have greatly added to the paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. More details on the training and testing procedure.\n\nThank you for raising the issue and we will include more details in the next version. \n\nWe have included the detailed data generation process in the attached PDF (Figure 1) and we briefly introduce the process in the following. DesCo is trained with two kinds of data: <image, entity> data (the typical object detection data) and <image, caption> data. For the latter, we randomly choose one of two ways to augment it:\nExtract the entities and append descriptions to it. We will also sample “confusable” descriptions for other entities and optionally drop the entity from the query.\nWe add hard negative captions generated by a language model; the original caption may be dropped from the query.\nFor the former, we just conduct Augmentation A.\n\n&nbsp;\n\n2. \"Describe the training of CLIP and FIBER\"\n\nThank you for the suggestion. We will add more descriptions on the training recipe of the baseline models. For FIBER, the detection training stage is the same as GLIP thus we skip it.\n\n&nbsp;\n\n3. Novelty \n\nPlease check the general response.\n\n&nbsp;\n\n\n4. \"Explore other prompts\"\n\nThank you for the suggestion and we believe it would be an interesting direction to explore. For this paper, as a first step towards complex language-based detection, we focus on proposing techniques to reduce the bag-of-words/hallucination problem and opted to use a rather general prompt/description style. Thus, we left the exploration of different prompts to future work; the fact that our approach works without extensive prompt engineering also speaks to its effectiveness.\n\n&nbsp;\n\nQuestions:\n\n- COCO images are removed from training. The pre-training source data are exactly the same as GLIP-T.\n\n- Indeed the language model could give false-negative captions; but manual inspection of the data suggests that this is rare. Our empirical results suggest that the method works well despite such noise.\n\n- That’s a great observation! Indeed, even if we train our system on such context-sensitive data, there is \nno guarantee that the model “understands” the context completely. How much the model learns will depend on the “difficulty” of contrastive learning pairs we construct; thus such a phenomenon can be eased if we could prompt the LM to generate descriptions that require deeper contextual understanding.\n\n- Thank you for pointing this out. The original intention is that if the region labels can be **completely** determined solely by the entity name in all cases, then the mutual information is zero. This is a strong assumption and the mutual information for natural grounding data is not strictly zero. We will revise the wording of the statement.\n"}]

Author rebuttal:
We thank all the reviewers for the constructive reviews and suggestions! We are encouraged that the reviewers find our paper “**insightful and well-shaped for acceptance**” (Reviewer Wg2t); “demonstrates a viable path of distilling the knowledge present in LLMs into better object grounding abilities in VL models” (Reviewer hkVU);  we compare with “convincing advanced baselines in a **very interesting and current setting**” (Reviewer BSUS).

Below we would like to summarize the key contributions of our paper: 

(1). Enabling VL models to understand complex language queries is an important yet challenging task; we point out an overlooked issue of prior attempts: simply presenting descriptions at training time does not encourage the model to utilize them.

(2). We propose a novel approach of generating detailed descriptions and construct Winograd-style training examples using large language models. Our idea of utilizing natural language supervision from LLM for multimodal learning will “be helpful for the community” (Reviewer Wg2t). 

(3). DesCo is the first approach to support complex-language-based object detection and shows significant performance improvement on challenging benchmarks such as LVIS, while previous work is still restricted to “mining region-word (noun) correspondence” (Reviewer Wg2t) or focusing on simpler tasks such as image classification. Our paper solves an important and timely problem of how to improve detection systems with language instructions. This problem is featured in the recent OmniLabel Challenge 2023 at CVPR. Our system achieves the best performance on the leaderboard.


Meta-review: This paper proposes DesCo, a novel method that supports complex-language based object detection. It's well-written and motivated, and provides extensive experiments on several datasets, showing that DesCo outperforms existing methods on object recognition accuracy, generalization to unseen domains. The paper also provides qualitative examples of the generated descriptions, demonstrating the interpretability of the model. Most reviewers showed positive feedback to this paper. Although they raised some question as well, most of the questions are not about the novelty or the technical problem of the paper, but about how to make this work stronger and more solid. The authors have also addressed most of the reviewer comments in their rebuttal and promised to incorporate the feedback in the final version. Therefore, I recommend to ACCPET to this paper as a strong contribution to the field of object recognition and natural language processing.
Decision: Accept (poster)