SlowGuess/NeurIPS_2023_OCR huggingface dataset:
different from the strategy in GLIP or traditional object detection training recipe, where we strive to keep all boxes provided.
In this section, we first investigate whether current models (GLIP) can utilize language descriptions out-of-the-box; then we show that our method allows the model to utilize language descriptions and improves performance on LVIS and OmniLabel significantly.
4.1 Can language-conditioned detection models utilize language descriptions?
As a proof of concept, we first show the GLIP struggles to utilize language descriptions out of the box and analyze the failure patterns.
Model	ΔBox	ΔConf	AP
GLIP [25]	0.291	0.05	4.7
DEsCo-GLIP	0.381	0.11	12.4
Table 1: GLIP is insensitive to context changes compared to DESCo-GLIP.
GLIP does not effectively utilize language descriptions. We make an attempt at using descriptions to transfer GLIP to LVIS [11], which contains over 1,200 classes. The process is similar to that of [31]. For each
category, we prompt a large language model (GPT-3) to give details descriptions (as in Section 3) We append the description to the original class name to form a new query. An example of the queries can be seen shown in Figure 1 (bottom row). Directly appending the description to the object name at inference time only degrades the performance: GLIP-T achieves 20.8 AP on rare categories while appending the descriptions makes the performance drop to 12.2 AP. This is likely due to model hallucination on natural-language-like queries.
GLIP is insensitive to context changes. Examining the model predictions, we find that the model not only does not utilize language descriptions; it ignores the descriptions and tends to only focus on entity names, as we hypothesized. To quantitatively verify the phenomenon, we introduce a context-sensitivity test, inspired by the WinoGround [43] benchmark. For each image, we provide the model with a positive query  $q^{+}$ describing an object that appears in the image and a negative query  $q^{-}$ describing a confusable object. The original object names are removed from the query. An example of the test is shown in Figure 1 (bottom left), where the model is challenged to distinguish "mallet" and "ax".  $q^{+}$ and  $q^{-}$ describe objects from the same general category (e.g., both are "a kind of tool") while differing in other aspects, similar to the Winograd test.
Intuitively, if a model can effectively utilize the descriptions, it should exhibit two properties: 1) it should give higher alignment scores to entities in  $q^{+}$  compared to  $q^{-}$ ; 2) even if the model cannot "guess" the hidden entity, at least, the model predictions should change drastically when given two different descriptions. We thus introduce two metrics. 1) AP, which measures how accurate
Model	Backbone	LVIS MiniVal [17]	OmniLabel [38]
APr	APc	APf	AP	AP	APc	APd	APd-P
MDETR [17]	RN101	20.9	24.9	24.3	24.2	-	-	4.7	9.1
MaskRCNN [17]	RN101	26.3	34.0	33.9	33.3	-	-	-	-
RegionCLIP [57]	ResNet-50	-	-	-	-	2.7	2.7	2.6	3.2
Detic [59]	Swin-B	-	-	-	-	8.0	15.6	5.4	8.0
K-LITE [41]	Swin-T	14.8	18.6	24.8	21.3	-	-	-	-
GroundingDINO-T [28]	Swin-T	18.1	23.3	32.7	27.4	-	-	-	-
GroundingDINO-L [28]	Swin-L	22.2	30.7	38.8	33.9	-	-	-	-
GLIP-L [25]	Swin-L	28.2	34.3	41.5	37.3	25.8	32.9	21.2	33.2
GLIP-T [25]	Swin-T	20.8	21.4	31.0	26.0	19.3	23.6	16.4	25.8
DESCO-GLIP	Swin-T	30.8	30.5	39.0	34.6	23.8	27.4	21.0	30.4
FIBER-B [7]	Swin-B	25.7	29.0	39.5	33.8	25.7	30.3	22.3	34.8
DESCO-FIBER	Swin-B	34.8	35.5	43.9	39.5	29.3	31.6	27.3	37.7
Table 2: Zero-shot transfer to LVIS and OmniLabel. Numbers that are grayed out are supervised models. DESCO-GLIP and GLIP-T are directly comparable; DESCO-FIBER and FIBER-B are directly comparable; the rest are listed for reference and not directly comparable.
the model's predictions are. 2)  $\Delta$  Box and  $\Delta$  Conf, which are the differences between the model's predictions for  $q^{+}$  and  $q^{-}$ .  $\Delta$  Box measures the changes in box coordinates while  $\Delta$  Conf measures the changes in alignment scores of boxes. Details of the metrics are in the appendix.
We find that the baseline model not only cannot identify the correct description (low AP); but it effectively ignores the language context (low  $\Delta$  Box and  $\Delta$  Conf) (Table 1). On average, the confidence of the predicted boxes changes only 0.05 between  $q^{+}$  and  $q^{-}$ . One could see the examples in Figure 1. GLIP models make almost identical predictions for two different queries. Such insensitivity to language context makes it infeasible and unreliable to use descriptions to control model predictions.
4.2 Setup
In this section, we apply our approach to two vision-language object detection models GLIP [25] and FIBER [7].
Models. The visual backbon of GLIP and FIBER is Swin Transformer [30] and the text backbones are BERT [6] for GLIP and RoBERTa [29] for FIBER. Both models use Dynamic Head [4] as the detection architecture. Built upon the two models, we introduce two model variants: DESCo-GLIP and DESCO-FIBER.
Datasets. Following GLIP [25], we train the models on 1) O365 (Objects365 [39]), consisting of 0.66M images and 365 categories; 2) GoldG that is curated by MDETR [17] and contains 0.8M human-annotated images sourced from Flickr30k [35], Visual Genome [19], and GQA [15]; 3) CC3M [40]: the web-scraped Conceptual Captions dataset with the same pseudo-boxes used by GLIP. We down-sample CC3M to around 1.4M images to save training costs, based on whether high-confidence boxes exist in the image. As illustrated in Section 3, we convert the text caption of each instance into a detailed language description to construct description-rich data.
To evaluate how well the models generalize to novel concepts, we perform a zero-shot evaluation on the LVIS [11] and OmniLabel [38] datasets. LVIS is a popular dataset that has over 1,200 object categories with a challenging long tail of rare objects; OmniLabel is recently proposed and focuses on object detection with diverse and complex object descriptions in a naturally open-vocabulary setting. For evaluation on LVIS, for each category, we append the GPT-3 generated description to the category name; we group several descriptions into one query to save inference time. More details on the evaluation are in the appendix. For OmniLabel evaluation, we follow the original evaluation protocol without modifications. We also verify that the models still possess the ability to perform the conventional detection and grounding tasks as GLIP and FIBER, on COCO [26] and Flickr30K [35]. The evaluation results are in the appendix.
Row Model	LVIS MiniVal [17]	OmniLabel COCO [38]	Context Sensitivity
APr	APc	APf	AP	AP	APc	APd	APd-P	ΔBox	ΔConf	AP
1	GLIP-T	20.8	21.4	31.0	26.0	18.7	45.7	11.7	31.2	0.291	0.05	4.7
2	+ Description w/ Entity Name	20.5	23.9	35.5	29.2	23.6	47.4	14.7	36.0	0.293	0.06	5.7
3	+ Description w/o Entity Name	25.6	25.9	35.9	30.7	24.0	46.8	16.0	37.0	0.382	0.10	10.7
4	+ Description w/o Name + Neg Cap	26.5	27.1	35.8	31.3	24.7	48.2	16.6	36.2	0.381	0.10	10.5
Implementation details. We initialize DESCO-GLIP from the GLIP-T checkpoint and DESCO-FIBER from the FIBER-B checkpoint. We fine-tune the models on both the original data and the new description-rich data. For DESCO-GLIP, we fine-tune with a batch size of 16 and a learning rate of  $5 \times 10^{-5}$  for 300K steps; for DESCO-FIBER, we fine-tune with a batch size of 8 and a learning rate of  $1 \times 10^{-5}$  for 200K steps. Experiments can be replicated with 8 GPUs each with 32GB memories.
4.3 Zero-shot transfer to LVIS and OmniLabel
LVIS. Our method shows notable improvements over the baselines on the LVIS MiniVal dataset (Table 2). The improvement is particularly prominent for rare object categories (APr), with an increase of 10.0 for GLIP and 9.1 for FIBER. Results on the Val 1.0 set are in the Appendix.
OmniLabel. Our method also shows improvements over baselines on the OmniLabel dataset (Table 2). OmniLabel assesses model performance using plain categories (APc), free-form descriptions (APd), and positive descriptions (APd-P). Because our models are trained with description data, they naturally excel in supporting such queries, leading to substantial increases in APd and APd-P compared to the baselines. Specifically, DesCo-GLIP achieves a notable improvement of +4.6, while DesCo-FIBER achieves an even more impressive improvement of +5.0. Furthermore, our model's effectiveness extends beyond free-form descriptions to plain categories as well, as illustrated in the table. This highlights the robustness of our method across different evaluation settings and its ability to achieve improvements in various types of queries. Our method wins the 1st place in the OmniLabel challenge 2023 on all three tracks (see Appendix for details).
In this section, all ablation models are initialized from GLIP-T and trained for 100K steps.
Directly appending descriptions. We examine the impact of directly adding language descriptions to text queries, without incorporating context-sensitive query construction. The results are presented in Row 2 of Table 3. The performance on rare categories (APr) sees no improvement but decreases. To further evaluate the sensitivity of the model to contextual changes, we conduct the same context sensitivity analysis as the one described in Section 4.1. The context sensitivity of the model almost remains unchanged (Row 1-2):  $\Delta$ Box changes only 0.002 and  $\Delta$ Conf changes only 0.01. The results indicate that the model remains as insensitive to context changes as the baseline model. This suggests that the model struggles to accurately interpret and effectively utilize the provided language descriptions when context-sensitive query construction is removed.
Dropping the entity name. As in Section 3.2.2, we hypothesize that randomly removing the entity name can force the models to concentrate on the contextual information. Remarkably, the results presented in Table 3 (Row 2-3) demonstrate that this simple and intuitive approach proves to be highly effective. It significantly enhances the model's contextual sensitivity while concurrently improving object detection performance.
Negative captions. We also investigate the effectiveness of using language models to generate hard negative
Table 3: Ablation study. Directly appending the description does not improve performance on rare categories (Row 1 v.s. Row 2, LVIS APr). Constructing context-sensitive queries is crucial.
GPT	APr	APc	APf	AP
ada	19.9	23.2	33.7	28.0
babbage	24.2	26.7	36.5	31.3
curie	24.7	28.4	38.2	32.8
davinci	30.8	30.5	39.0	34.6
Table 4: Detection performance improves when language model size grows.
From GPT-Curie:
Figure 4: Detection performance of DESCo-GLIP improves when given better descriptions. GPT-Curie is a smaller model than GPT-Davinci; it gives less accurate descriptions for objects.
From GPT-Davinci:
captions. As shown in Row 4 of Table 3, including neg-
native captions can improve the model detection performance across datasets while preserving its robust contextual comprehension. These results indicate that this technique effectively enhances the model's ability to grasp the subtleties embedded in the given language descriptions.
Language description quality. We explore the effect of the language model size on detection performance. We evaluate the pre-trained DESCO-GLIP on LVIS with descriptions generated from the GPT families<sup>4</sup>. As presented in Table 4, higher-quality language models improve object detection performance. This finding highlights the importance of employing strong language models, as they possess the ability to embed valuable visual information through extensive pre-training. We showcase two examples in Figure 4.
In this study, we introduced a new paradigm of learning object detection models from language supervision. We show that large language models can be used to generate rich descriptions and the necessity to construct context-sensitive queries. We hope that our method sheds light on empowering vision models with the ability to accept flexible language queries.
While we greatly improve the models' ability to understand flexible language queries, our method has several limitations that can be addressed in future work. 1) We use a large language model to automatically generate the descriptions, which inevitably introduces noise as not all generated descriptions are accurate or beneficial for representation learning. Future work could consider automatically selecting useful descriptions sampled from the language model, similar to [49]. 2) The format of the descriptions we explored is still limited (e.g., "  $\{\text{entity}\}$ , a kind of  $\{\text{type}\}$ ,  $\{\text{list of simple features}\}$ ); it might be useful to consider more diverse descriptions by prompting the language model with more diverse prompts. 3) Similar to large language models, querying our model also requires a certain amount of prompt engineering. Future work could explore how to make the model more robust to different kinds of queries.
We would like to thank members of UCLA NLP for their helpful comments. We also thank the reviewers for the valuable reviews. This work was supported in part by DARPA MCS program under contract number N660011924032, DARPA ECOLE program No. HR00112390060, and ONR grant N00014-23-1-2780. We also thank Amazon for AWS credits for computational resources. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.
<sup>4</sup>https://platform.openai.com/docs/models
A Approach
A.1 Description generation with large language models
We prompt davinci-003 with the following prompt:
Question: What features should object detection models focus on for a given input? Answer:  
Input: zucchini, Output: "type": "vegetable", "description": "cylindrical, green, smooth; could have brown and rough stems; could be sliced into round pieces; could have green leaves", "similar objects": ["cucumber", "eggplant", "green bean"]  
Input: zebra, Output: "type": "animal", "description": "black and white stripes; has a long mane", "similar objects": ["horse", "giraffe", "elephant"]  
Input: apple, Output: "type": "fruit", "description": "red, round, has a stem", "similar objects": ["orange", "banana", "pear"]  
Input: wok, Output: "type": "cooking tool", "description": "round, deep, has a handle", "similar objects": ["pan", "pot", "frying pan"]  
Input: ambulance, Output: "type": "vehicle", "description": "red; has a glaring siren; could with a stretcher", "similar objects": ["police car", "taxi", "garbage truck"]  
Input: lantern, Output: "type": "lighting tool", "description": "round; could be made of papers", "similar objects": ["lamp", "flashlight", "candle"]  
Input: {entity}
Table 5: Text prompt used to sample descriptions from large language models.
We construct a vocabulary of  $10K$  entities by extracting noun phrases from the pre-training text corpus (GoldG and CC3M) using NLTK. As we also use Objects365 for training, we add the categories from Objects365 to the entity vocabulary. We query the language model to generate descriptions for the entities.
A.2 Context-sensitive query construction
As shown in Table 5, when prompting the language model, we also ask the model to name a few "similar objects". Thus, when constructing a query of descriptions, we include both positive descriptions and several negative descriptions for such "similar objects". GLIP has a max query length of 256 tokens. On average, we can pack 8 descriptions into one query. We randomly drop descriptions if the length exceeds the length limit.
For image caption data, we use to prompt in Table 6 to generate negative captions. Note that while the prompt asks the LLM to generate object descriptions and confusable objects, we find that the prompt in Table 5 gives better results for this purpose; thus we do not use the description and confusable objects given by prompt in Table 6.
A.3 Query construction
Algorithm 1 Generating Queries for Detection Data
Input:  $T,E,V$  1:  $Q^{-}\gets \mathrm{RandSample}(V\setminus E)$  2:  $q^{*}\gets \mathrm{ShuffleConcat}(E\cup Q^{-})$  3:  $q^{*},T\gets \mathrm{LabelAssign}(q^{*},T,E)$
Algorithm 2 Generating Queries for Grounding Data
Input:  $T, D, C$   
1:  $Q^{-} \gets \text{RandSample}(D \setminus \{C\})$   
2:  $q^{*} \gets \text{ShuffleConcat}(\{C\} \cup Q^{-})$   
3:  $q^{*}, T \gets \text{LabelAssign}(q^{*}, T, C)$
Figure 5: Algorithms for generating queries from detection data and grounding data for GLIP. Algorithm 1: Compare to DesCo, note that no positive entities are dropped from the query; thus  $B$  is not involved in the query construction. Algorithm 2:  $D$  is the corpus of image captions. Compare to DesCo, note that the positive caption  $C$  is always included in the query; this creates the hallucination issue.
Given a caption, perform the following:
1. Extract the visually concrete objects and give a description for what object detection models should focus on for an given input. Avoid common objects such as "person", "dog", "actor", "cat", "man", "woman", "boys", "girls".
2. Name some negative visual objects can be confused and give descriptions. Focus on the visual differences between the positive and negative objects.
3. Write some hard negative captions for a given positive caption, by replacing the nouns, adjectives, and verbs with visually similar but wrong words. Do not replace the nouns that refer to people.
Positive caption: an ambulance is rushing down the street with its lights flashing
1. Visually concrete objects and descriptions: "ambulance": "a kind of car, red light, red and white patterns on the body"
2. Objects that can be confused with the above objects: "police car": "a kind of car, blue and red glarring light, blue and white patterns", "taxi": "a kind of car, yellow, has a notch"
3. Negative captions: ["a police car is rushing down the street with its lights flashing",]
Positive caption: the aircraft carrier is enveloped in fog as it sits in its berth
1. Visually concrete objects and descriptions: "aircraft carrier": "a kind of ship, large, has a long deck, has runways, has airplanes on the deck"
2. Objects that can be confused with the above objects: "cruise ship": "a kind of ship, usually white, luxury, has swimming pools", "cargo ship": "a kind of ship, has boxes on the deck"
3. Negative captions: ["the shipping boat is enveloped in fog as it sits in its berth", "the cargo ship is enveloped in fog as it sits in its berth"]
Positive caption: the beans in the red silicone cup
1. Visually concrete objects and descriptions: "beans": "a kind of vegetable, small, round, usually green"
2. Objects that can be confused with the above objects: "apples": "a kind of fruit, usually red", "beeds": "a kind of decorations, small, round, colorful"
3. Negative captions: ["the beans in the green silver cup.", "the apples in the red silicone cup.", "the beans in the red porcelain cup."
Positive caption: bulldozer on a building site 1. Visually concrete objects and descriptions: "bulldozer": "a kind of heavy machinery, has a large blade in front, used for pushing debris or soil"
2. Objects that can be confused with the above objects: "excavator": "a kind of heavy machinery, has a long arm and scoop, used for digging and moving materials", "crane": "a kind of heavy machinery, has a long arm and hook, used for lifting"
3. Negative captions: ["crane on a demolition site", "backhoe on a construction site"]
Positive caption: A girl standing in a shallow creek, wearing stilts.
1. Visually concrete objects and descriptions: "stilts": "a kind of shoes, has two long poles"
2. Objects that can be confused with the above objects: "high heels": "a kind of shoes, has a high heel", "sandels": "a kind of shoes, has a strap, usually worn in summer"
3. Negative captions: ["A girl standing in a shallow creek, wearing sandels", "A girl running in a deep river, wearing stilts"]
Positive caption: {caption}
Table 6: Text prompt used to sample negative captions from large language models.
B.1 Context-sensitivity test
To create the test set, we go over the LVIS validation set and for each image that has a rare object (defined by the LVIS taxonomy), we query the model with a  $q^{+}$  (the description of the rare object, without the object name) and a  $q^{-}$  (the description of a confusable object, without the object name). The confusable object comes from prompting the LLM as shown in Table 5. Then we calculate the difference between the predictions made by the model for  $q^{+}$  and  $q^{-}$ .
For  $q^{+}$  and  $q^{-}$ , the model will give two sets of predictions (two lists of boxes). We first match the two lists of boxes by their IoU overlap. Then  $\Delta$  Box is the percentage of boxes that have high IoU overlap ( $>0.5$ ) between the two sets of predictions;  $\Delta$  Score is the confidence score changes for those matched boxes. With higher  $\Delta$  Box and  $\Delta$  Score, the model predictions change more drastically. The evaluation data and code will be released upon acceptance.
B.2 Evaluation on COCO and Flickr30K
Model	COCOMAP	Flickr30K Val
R@1	R@5	R@10
GLIP-T*	44.6	85.6	95.8	97.3
DESCO-GLIP	45.8	85.3	95.8	97.3
FIBER-B	49.3	87.1	96.1	97.4
DESCO-FIBER	48.9	86.9	96.4	98.0
In Table 7, we report the performance on common object detection (COCO, zero-shot) and Flickr30K. The performance of DESCo-GLIP and DESCo-FIBER is similar to GLIP and FIBER. GLIP- $T^{*}$  is the checkpoint that achieves the best performance on LVIS released in https://github.com/microsoft/GLIP; it has slightly worse performance than GLIP-T on COCO. We used GLIP- $T^{*}$  to initialize DESCo-GLIP, thus we compare with GLIP- $T^{*}$  in this case.
B.3 Evaluation on LVIS
Table 7: DESCo-GLIP and DESCo-FIBER maintain similar performance to GLIP and FIBER on common object detection and phrase grounding. DESCo-GLIP and DESCo-FIBER are fine-tuned with a smaller batch size than GLIP and FIBER; thus some minor performance drops are expected.
Model	LVIS Val
APr	APc	APf	AP
GLIP	10.1	12.5	25.5	17.2
DESCO-GLIP	19.6	22.0	33.6	26.2
FIBER	18.0	21.6	35.0	26.3
DESCO-FIBER	23.0	26.3	38.5	30.5
Table 8: DESCo-GLIP and DESCo-FIBER achieve strong performance on LVIS val 1.0.
For evaluation on LVIS, for each category, we generate descriptions with prompts in Table 6. When doing inference on an image, as the vocabulary is large, concatenating descriptions of all categories into a long prompt results in a extremely long prompt. We adopt the same approach as GLIP and FIBER; we construct multiple prompts where each prompt only contains a few category descriptions; the model is queried multiple times and we aggregate the outputs across the multiple forward passes.
We evaluate models on the full validation set of LVIS using the fixed AP protocol. It can be seen that our approach achieves large performance gains compared to the baselines.
B.4 OmniLabel Challenge 2023 winning entry
We submit DESCo-FIBER to the OmniLabel Challenge  $2023^{5}$  and won the 1st place. Different from the model reported in the main text, the submitted version is first initialized from DESCo-FIBER, then undergoes a second-stage pre-training on Objects365, GoldG, CC3M, and RefCOCO [52] (including RefCOCO, RefCOCO+, RefCOCOg), with a batch size of 8 for another 200K steps.
<sup>5</sup>https://www.omnilabel.org/challenge/challenge-2023