SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lianian Harold Li* Zi-Yi Dou* Nanyun Peng Kai-Wei Chang
{liunian.harold.li, zdou, violetpeng, kwchang}@cs.ucla.edu
Recent development in vision-language approaches has instigated a paradigm shift in learning visual recognition models from language supervision. These approaches align objects with language queries (e.g. "a photo of a cat") and thus improve the models' adaptability to novel objects and domains. Recent studies have attempted to query these models with complex language expressions that include specifications of fine-grained details, such as colors, shapes, and relations. However, simply incorporating language descriptions into queries does not guarantee accurate interpretation by the models. In fact, our experiments show that GLIP, a state-of-the-art vision-language model for object detection, often disregards contextual information in the language descriptions and instead relies heavily on detecting objects solely by their names. To tackle the challenge, we propose a new description-conditioned (DesCo) paradigm of learning object recognition models with rich language descriptions consisting of two innovations: 1) we employ a large language model as a commonsense knowledge engine to generate rich language descriptions of objects; 2) we design context-sensitive queries to improve the model's ability in deciphering intricate nuances embedded within descriptions and enforce the model to focus on context rather than object names alone. On two novel object detection benchmarks, LVIS and OminiLabel, under the zero-shot detection setting, our approach achieves 34.8 APr minival (+9.1) and 29.3 AP (+3.6), respectively, surpassing the prior state-of-the-art models, GLIP and FIBER, by a large margin.
Training visual recognition models to classify or detect objects with a fixed set of pre-defined categories has been the convention for a long time. However, models trained using this approach often encounter difficulties when adapting to unfamiliar concepts and domains. Recently, there has been a paradigm shift towards training visual recognition models with language supervision, using a contrastive objective on a large amount of image-text data containing a diverse range of visual concepts. These models can then be transferred to downstream tasks via language queries. For example, CLIP [36] can perform image classification using a template query such as "a photo of {class name}"; GLIP [25] can perform object detection by querying the model with "Detect: person, cat, dog ...".
Early applications of these models typically utilize simple language queries that consist of object names. However, language queries can convey much richer and more comprehensive information, such as object attributes, shapes, textures, and relations. These pieces of information can be especially useful for identifying novel visual concepts that do not appear in the training corpus or specifying
Figure 1: Comparison between our model (DESCO-GLIP) and the baseline (GLIP [25]). Each image is paired with a positive query for target object and a negative query for confusable object. A successful model should locate the target object and ignore the confusable object in the image based on fine-grained specifications for shapes, subparts, relations, etc. We highlight the descriptions that match and do not match to the queried object in blue and red, respectively. Results show that our model can successfully localize the target object and suppresses the negative query even for the difficult cases when the object name is not in the query or the object.
specific needs. For example, the concept of "mallet" can be described as "a kind of tool, wooden handle with a round head" (Figure 1, bottom-left). This decomposes object recognition into recognizing fine-grained details (such as attributes, sub-parts, shapes, etc.) and aligning them to the descriptions. Several studies [25, 41, 31] have explored the idea of guiding language-based recognition models using such descriptive prompts. However, few existing models take complex queries into account during training. As a result, current models often struggle with recognizing intricate object names, attributes, and relations described in natural language sentences [53, 43] (see examples in Figure 1).
In this paper, we develop a vision-language model capable of leveraging description-rich language queries to perform object detection. This work aligns with the recent surge of interest in instruction/prompt-aware vision-language models (see a discussion in Section 2). Our goal is to equip VLMs with the ability to comprehend complex language input describing visual concepts, similar to the capability of large language models. We specifically study instruction/prompts in the form of descriptive queries. We focus on object detection, as it requires fine-grained recognition and is more challenging than image-level tasks. However, our method can be generalized to other vision tasks such as classification and segmentation [61].
We identify two major challenges that prevent existing models from efficiently utilizing rich language descriptions: (1) Fine-grained descriptions are rare in image-caption data used for training current VLMs². This resembles the reporting bias phenomenon [34]: when writing captions for images, humans tend to directly mention the entities rather than give a detailed description. (2) Even when provided with data rich in descriptions, models often lack the incentive to leverage these descriptions effectively. The main training objective is to align positive phrases with relevant regions while suppressing negative phrases. However, if positive/negative phrases can be distinguished without
2We count the region-label data used by models like GLIP as image-caption data because the labels are converted into captions through templates.
descriptions, the training mechanism fails to incentivize the model to use the provided description. For example, a positive phrase like "A toy bear holding a mallet, which has a wooden handle with a round head," and a negative phrase like "A toy bear holding an ax, which has a long handle and a sharp blade," can be differentiated based solely on the words mallet and ax. This issue resembles the issue discovered by [53], where vision-language models ignore word order and treat a query as a "bag-of-words" due to insufficient incentives from the contrastive objective. In addition, current models suffer severe hallucination when given natural language queries (in contrast to "template-like" queries) due to shortcuts introduced in training query formulation. This can be seen in the bottom-right picture of Figure 1, where GLIP hallucinates and predicts multiple wrong boxes for "microphone" while "microphone" does not appear in the image.
Based on the observations, we present a Description-Conditioned (DESCO) paradigm of learning object recognition models from language descriptions based on two synergistic ideas:
(1) Generating descriptions with large language models. Instead of learning from raw image_caption pairs, we use large language models as a world knowledge base and generate detailed descriptions based on the original caption. We prompt GPT-3 [2] with "What features should object detection models focus on for {an entity in the caption}?" This serves as a scalable approach to transfer the image-caption data into image-description data.
(2) Context-sensitive query construction. As discussed, even if we provide descriptions during pretraining, models can still ignore the language context. Our solution is to create a "context-sensitive query", which is a set of positive and negative phrases that can only be distinguished by reading the descriptions (Figure 2). We explore two strategies: 1) constructing "Winograd-like" [13, 43] queries by using large language models to generate confusable object descriptions and captions and 2) generalizing the original grounding task to allow full-negative queries, reducing hallucination.
We apply our approach to fine-tune two state-of-the-art language-conditioned object detection models GLIP [25] and FIBER [7]. We use the same raw training data as the baselines but convert the data into description-rich queries. We evaluate our methods in two settings. (1) Zero-shot generalization to novel categories (LVIS [11]), where we use GPT-3 to generate descriptions given class names. DesCo-GLIP (Tiny) improves upon GLIP (Tiny) by 10.0 APr, even outperforming the larger GLIP (Large); DesCo-FIBER improves upon FIBER by 9.1 APr. (2) Zero-shot generalization to natural descriptions given by humans (OmniLabel [38]). DesCo-GLIP and DesCo-FIBER improve upon the baselines by 4.5 AP and 3.6 AP, setting a new state-of-the-art performance level. Code is available at https://github.com/liunian-harold-li/DesCo.
Language-based visual recognition models. Visual recognition models are typically trained to make predictions based on a fixed set of classes [20, 5, 26, 39, 33, 58]. The trained models are hard to generalize to open-domain settings. Recent studies have developed visual recognition models that take into account language queries, i.e. language-based recognition. This line of research can be traced back to early work of generalizing image classification [42] and object detection [1] models with word embeddings. Recently, CLIP [36] reformulates image classification as image-text matching and pre-trains models on large-scale image-caption pairs to learn transferrable representations. They demonstrate strong zero-shot performance on various classification tasks. Recent work has applied the technique to fine-grained recognition tasks, such as object detection [17, 10, 25, 57, 56, 3, 32, 7, 28], and segmentation [21, 9, 16, 47, 54, 24]. These works either use pure image-text data as supervision [47], or reformulate labeled data into image-text data [21], or pseudo labels image-text data with fine-grained labels [25]. Orthogonal to architecture design or scaling-up, which is the focus of many prior studies, this paper points out that the vanilla way of using image-text data is insufficient and studies how to train these models to take more flexible and informative language queries.
Vision-language models with complex prompts. As vision recognition models become language-aware and language models become vision-aware [45, 60, 22], there is a growing interest in studying whether these models can take complex language prompts, such as task instructions (e.g., GPV [12, 18], SEEM [62], VisionLLM [46]), descriptions [23], or even dialogues (e.g., LLaVa [27]). We study specifically descriptive prompts, which are especially useful for generalizing to novel categories and customized detection needs; a model that can understand descriptive prompts can also serve as the
backbone for supporting aforementioned other types of prompts. Similar to our work, K-LITE [41] proposes to retrieve knowledge for a visual concept using external knowledge bases, then use the enriched concepts for image classification or object detection; similar techniques have also been proposed by [31, 49, 48]. DetCLIP [50] builds a large-scale concept dictionary, based on which they provide definitions from WordNet. Different from these studies, our methods show that simply presenting the descriptions at training or inference time is not enough; we propose a simple technique to force models to focus on the provided descriptions (Section 3.2.2). Our work relates to a line of work that seeks to reveal and fix failure patterns of image-text matching models (e.g., CLIP) by creating hard negative examples [43, 53, 8, 37].
In this section, we first briefly introduce language-based object detection models, then illustrate the details of our proposed approach.
We give an introduction to language-based object detection models [17, 25, 7], which take a language query and an image as inputs, and predict bounding boxes and their alignment to phrases in the language query. In the following, we use GLIP as an example.
Baseline: Grounded Language-Image Pre-training (GLIP). At the center of these approaches is "reformulating any task-specific fixed-vocab classification problem as a task-agnostic open-vocabulary vision-language matching problem" [55]. The best example is CLIP which reformulates image classification as image-text matching. Similarly, GLIP unifies training data into a grounding format:  $(I,Q,B,T)$ .  $I$  is the image;  $Q$  is the text query;  $B\in R^{N\times 4}$  is the bounding boxes;  $T\in \{0,1\}^{N\times K}$  indicates the ground-truth alignment label between the  $N$  bounding boxes and  $K$  tokens in the query. The key is how to formulate the query with data from two kinds of sources:
- Detection data. For object detection data such as Objects365 [39], the query is the concatenation as a list of object classes, such as "Detect: person. bicycle. car. ..., toothbrush". Note that negative object classes are included in the query; this makes such query-based detection models similar to classical detection models.
- Grounding data. Typically,  $Q$  is an image caption, containing entities that can be aligned to annotated object regions [35]. For example, "A toy bear holding a mallet" is the caption; "toy bear" and "mallet" are the "groundable" entities. For densely annotated grounding data (multiple captions for one image) [19], we can concatenate multiple captions into a longer query. Image_caption data (without annotated boxes) can be transferred into grounding data via pseudo labeling with a grounding model [25].
Given  $I$  and  $Q$ , we compute the alignment scores  $S_{\mathrm{ground}}$  between image regions and words in the query:
$$
O, L = \operatorname {E n c} (I, Q), S _ {\text {g r o u n d}} = O L ^ {\top}, \mathcal {L} = \operatorname {l o s s} \left(S _ {\text {g r o u n d}}, T\right) + \mathcal {L} _ {\text {l o c}}
$$
where  $L \in \mathbb{R}^{K \times d}$  is the contextual token features and  $O \in \mathbb{R}^{N' \times d}$  are the regions features. Enc is a vision and language encoder that takes both image and text as inputs and fuses their representations. The training loss contains the region-word matching loss and a localization loss  $\mathcal{L}_{\mathrm{loc}}$  as in conventional object detection models.
Inference with language query. At inference time, the model can be used to locate entities/class names appearing in the query. One could simply provide a list of candidate object names (as in the detection data training format). [25] also show the promise of using descriptions for generalization to novel concepts; however, we show that while GLIP can be influenced by the description, it does not always take the details in the description into account.
3.2 Learning with language descriptions
To train object recognition models that fully utilize language descriptions, we propose to generate descriptions with large language models and construct context-sensitive queries during training. The following subsections provide further details.
Figure 2: Given the original training data of GLIP, we transform it to be description-rich and context-sensitive by: 1) generating descriptions for entities and composing each of them with confusable object descriptions; 2) generating negative captions. We visualize the gold alignment labels (ground truth) between tokens and regions for the new data. Notably, words such as tools are assigned both positive (blue block) and negative (red block) labels in alignment with the corresponding object depending on the context of the query. As such, the model requires understanding the description in order to make the correct prediction.
3.2.1 Description generation with large language models
Fine-grained descriptions could be scarce in image-caption data due to reporting bias. While this problem can be alleviated by scaling up the pre-training corpus, we show that large language models [6, 2] can be used to effectively generate the descriptions and thus enrich our training corpus.
We leverage a large language model to transform a query  $Q$  into a description-rich query  $\mathrm{LLM}(Q)$ . In this work, we only focus on generating descriptions for entities mentioned in the original query. We construct a vocabulary consisting of 10K entities appearing frequently in the pre-training corpus. For each entity, we prompt a large language model: what features should object detection models focus on for {entity}? We find that large language models give high-quality responses (see examples in Figure 1 and Figure 4). The specific prompts are included in the appendix.
3.2.2 Context-sensitive query construction
Can we simply add the description-rich data to the pre-training corpus? An intuitive idea is to append the description to the original entity to form a training prompt (e.g., "Detect: mallet. bear...")  $\rightarrow$  "Detect: mallet, a kind of tool, wooden handle... bear, a kind of animal, ..."). However, we find that models naively trained with these prompts still do not exhibit "context-sensitivity", i.e., they make predictions solely based on the entity names while ignoring other contexts (see Section 4.1 for quantitative analysis). As a result, we observe no evident benefit in incorporating descriptions during inference (Table 3). In the following, we elaborate on why the model learns to ignore the descriptions and propose two solutions.
Model learn statistical shortcuts. We first illustrate that without careful design, the model could learn two statistical shortcuts that make them insensitive to descriptive queries.
(1) Entity shortcut. The model is trained to align the entities in the query to image regions (this includes predicting "no-alignment" for entities not appearing in the image). Intuitively, if the alignment can be predicted without relying on the context information in the query, then the model is not incentivized to focus on the context information. Figure 2 illustrates this issue with an example. The left side shows the training data of GLIP, where the top query ("Detect: Mallet. Bear. Cat...") comes from detection data and the bottom query ("A toy bear holding a mallet.") comes from grounding data. The problem with such queries is that they can be grounded by only focusing on the entity names and ignoring the context. We denote the gold alignment label of regions as  $T$ , the entities in the query as  $E$ , and the non-entity part (context) of the query as  $C$ . The mutual information
$I(T; C|E, I)$  between  $C$  and  $T$  given  $E$  and the image  $I$  would be low. That is, the non-entity parts of the queries do not affect the label of the region. Training models on such data will not encourage the model to focus on the descriptions as they provide no additional information. This is similar to the "memorization overfit" issue observed in [51]: the model can simply choose to "memorize" the alignment between the entities and regions.
(2) Format shortcut (hallucination). Popularized by GLIP [25], a line of work adopts a unified view of phrase grounding and object detection: detection can be seen as language-context-free grounding while grounding can be seen as language-context-dependent detection. However, this unification is still imperfect: phrase grounding (or referring expression [52]) traditionally only concerns locating entities in a caption that always exist in the image; thus the model learns to always treat the natural-language queries (in contrast to the template-like queries) as positive and tries to ground every entity mentioned in the sentence. This will result in failure examples as illustrated in the bottom-right picture of Figure 1. Such "hallucination" can be commonly seen on models trained on language grounding data [17]; these models are almost incapable of distinguishing positive and negative "natural-language-like" queries.
Constructing context-sensitive queries. This motivates our solution of creating queries that are hard to solve without context (Figure 2 and Figure 3). We explore two strategies in this study.
(1) We construct training queries similar to the Winograd format. For example, when training on detection data, instead of "Detect: mallet, a kind of tool, ...", we remove the entity name "mallet" from the query and sample another description of a "confusable" entity that is also a kind of tool. Pairing the descriptions of the two "confusable" entities creates a strong supervision signal (the middle example in Figure 2): the alignment label (0 or 1) of the word "tool" now depends on its context. The confusable entities are obtained by prompting the large language models as well. Similarly, for training on grounding data, we prompt language models to generate confusable (hard negative) captions that differ from the original captions only by a few words (the example on the right in Figure 2). Note that the label of the word "mallet" is now affected by the context: the first "mallet" is assigned 0 as the caption ("A polar bear holding a mallet") is negative. Mixing in such hard negative captions encourages the model to focus on the context surrounding the entities, such as relations to other entities. To make the confusable caption generation process scalable for image_caption data, we first perform in-context inference and prompt GPT-3 to generate around 50K negative captions based on positive captions; then we distill this knowledge to the open-sourced LLaMA-7B [44] model that is instruction-finetuned using low-rank adaptation[14] and perform inference on large-scale image_caption data.
(2) To resolve the hallucination issue, we generalize the original grounding task: instead of always feeding the model a query that contains at least one description/caption matching the image, we allow the query contain only negative descriptions/captions (Figure 3). Thus, the model cannot blindly ground all entities mentioned in the query; implicitly, it needs to perform image-text matching [36] as well as phrase grounding. This was partly done in GLIP (see Appendix C.1 in the original GLIP paper), but the query still contains at least one positive entity.
Overview. In Figure 3, we summarize the overall data construction algorithm. Algorithm 1:  $B \in R^{N \times 4}$  are the bounding boxes of an image;  $E$  are  $M$  positive objects (entities) that appear in the image;  $V$  are the descriptions of all candidate objects in a pre-defined vocabulary;  $T \in \{0,1\}^{N \times M}$  denotes the gold alignment between boxes and entities. We first prompt LLM to generate descriptions for the positive entities and propose confusable entities and descriptions; the prompt is included in the appendix (Line 3). The original entities are removed from the descriptions with  $p_{\mathrm{drop}} = 0.5$  (DropEntity, Line 4-5). Random negative descriptions from the vocabulary are added to the candicate description set (Line 7). We then randomly subsample the descriptions and concatenate them to form a final query  $q^*$ ; this is because the total length of all the candidate descriptions is too large (Line 8). Boxes and the mapping relations between boxes and tokens are accordingly adjusted (Line 9). Algorithm 2:  $C$  is the original caption and  $E$  are  $M$  positive phrases we extracted from the caption. The last two lines of both algorithms are important: after SubSampleConcat, it is very likely that some positive sub-queries are dropped from  $Q$ ; then LabelAssign would drop boxes that are mapped to the dropped sub-queries. The output  $B$  could end with fewer boxes or even no boxes. This is
<sup>3</sup>https://github.com/tloen/alpaca-lora
Algorithm 1 Generating Queries for Detection Data
Input: B (boxes), T (alignment matrix), E (positive entities), V (vocabulary)
1: Q ← ∅
2: for i ← 1 to M do
3: q, Q-← LLM(prompt, E_i)
4: if random() < pdrop then
5: q, Q-← DropEntity(q, Q-)
6: Q ← Q ∪ {q} ∪ Q-
7: Q ← Q ∪ RandSample(V)
8: q* ← SubSampleConcat(Q)
9: q*, T, B ← LabelAssign(q*, T, E, B)
Algorithm 2 Generating Queries for Grounding Data
Input: B (boxes), T (alignment matrix), E (positive entities), V (vocabulary), C (caption)
1: if random() < pdes then
2: Q, T, B ← Algorithm1(B, T, E, V)
3: else
4: Q- ← LLM(promptneg, C)
5: Q ← {C} ∪ Q-
6: q* ← SubSampleConcat(Q)
7: q*, T, B ← LabelAssign(q*, T, C, B)
Figure 3: Algorithms for generating queries from detection data and grounding data.