SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chi Xie $^{1\dagger}$  Zhao Zhang $^{2\dagger}$  Yixuan Wu $^{3}$  Feng Zhu $^{2}$  Rui Zhao $^{2}$  Shuang Liang $^{1*}$
$^{1}$ Tongji University  $^{2}$ Sensetime Research  $^{3}$ Zhejiang University
chixie@tongji.edu.cn zzhang@mail.nankai.edu.cn shuangliang@tongji.edu.cn
Detecting objects based on language information is a popular task that includes Open-Vocabulary object Detection (OVD) and Referring Expression Comprehension (REC). In this paper, we advance them to a more practical setting called Described Object Detection (DOD) by expanding category names to flexible language expressions for OVD and overcoming the limitation of REC only grounding the pre-existing object. We establish the research foundation for DOD by constructing a Description Detection Dataset  $(\mathrm{D}^3)$ . This dataset features flexible language expressions, whether short category names or long descriptions, and annotating all described objects on all images without omission. By evaluating previous SOTA methods on  $\mathrm{D}^3$ , we find some troublemakers that fail current REC, OVD, and bi-functional methods. REC methods struggle with confidence scores, rejecting negative instances, and multi-target scenarios, while OVD methods face constraints with long and complex descriptions. Recent bi-functional methods also do not work well on DOD due to their separated training procedures and inference strategies for REC and OVD tasks. Building upon the aforementioned findings, we propose a baseline that largely improves REC methods by reconstructing the training data and introducing a binary classification sub-task, outperforming existing methods. Data and code are available at this URL and related works are tracked in this repo.
Detecting objects of interest within a scene using language is a pivotal area of focus. This field encompasses two key tasks: Open-Vocabulary object Detection (OVD) [10, 11, 20, 29, 47, 48] and Referring Expression Comprehension (REC) [21, 27, 24, 46, 52]. We present an intuitive illustration of these two settings in Fig. 1. The first task, OVD, expands the scope of object detection (OD) to any given short category name. However, these settings neglect the instances described by intricate descriptions. The second task, REC, focuses on spatially locating one target described by an expression and assumes the target must exist in the image. However, in real-world scenarios, if the described objects do not exist in the image, REC algorithms output false-positive results. Recent advancements have witnessed the joint training of bi-functional models, such as Grounding-DINO [25] and UNINEXT [44], which involve both OVD and REC data. Notwithstanding, these models still rely on separate training procedures and inference strategies for OVD and REC, and evaluate these two tasks independently.
As shown in Fig. 1, a more practical detection algorithm should be able to detect any described category, whether long or short, complex or simple, while discarding predictions in images where targets are absent. In order to address this significant yet often overlooked scenario, we propose the concept of Described Object Detection (DOD). Note that this setting is a superset of OVD and REC.
Figure 1: Examples showing the difference between REC, OVD and Described Object Detection (DOD). OVD detects arbitrary number (including zero, denoted with  $\emptyset$ ) of objects based on a category name; REC grounds one region based on a language description, whether the object truly exists or not; DOD detect all instances on each image in the dataset, based on a flexible reference.
When the language expression is limited to a short category name, it becomes OVD. When we limit the images to detect objects known to be present in the images beforehand, it downgrades to REC.
Can the existing SOTA algorithms of the community support DOD tasks? To address this inquiry, this paper establishes the research foundation of DOD tasks by constructing a dataset, scrutinizing relevant methodologies, analyzing the relevant methods, and exploring improvement space.
Motivation & real-world application of DOD. OVD is limited to categorical detection, focusing on classes rather than specific attributes or relationships. It lacks detailed contextual understanding and cannot adapt to precise detection requirements from language. REC comprehend longer descriptions for attributes or relationships, but assumes the existence of one target in the image. This leads to false positives when the target is absent, limiting its practical usability. Consider detecting individuals without helmets on a construction site using camera data: OVD can detect helmets and people but not determine their relationship. REC locate one region in any image and generate false positives frequently. Existing solutions involve using separate models for object detection then relationship classification, or REC after image classification, both resulting in inefficiency.
Hence, there is a demand for language-based object detection: a model with strong generalization capabilities that can verify the existence of described objects in images and localize them based on arbitrary expressions. The proposed DOD task addresses this need and finds practical applications in: urban security, detecting dogs without leashes in communities, clothes hung outdoors on streets, overloaded vehicles, and fallen trees on roadsides; network security, like identifying sensitive images with violence or bloodshed within large datasets; (fine-grained) photo album retrieval based on descriptions or keywords; retrieval and filtering of web image data; specific event detection in autonomous driving, such as pedestrians crossing the road.
Dataset & benchmark. For DOD, we introduce the Description Detection Dataset  $(\mathrm{D}^3, / \mathrm{dikju:b/})$  an evaluation-only benchmark containing 422 descriptions and 24,282 positive object-description pairs. Unlike previous OVD or REC datasets (see Fig. 2),  $\mathrm{D}^3$  stands out in three key aspects (see Tab. 1): 1) Complete annotation: All descriptions refer to objects annotated throughout the dataset, making  $\mathrm{D}^3$  a detection-style dataset akin to COCO [23]. 2) Unrestricted description: Annotations in  $\mathrm{D}^3$  include diverse and flexible language expressions, varying in length and complexity. 3) Absence expression: We include descriptions regarding absence of concepts, such as a person without a safety helmet, addressing an often-overlooked detection requirement. The details of  $\mathrm{D}^3$  is elaborated in Sec. 3. We evaluate state-of-the-art methods on  $\mathrm{D}^3$ : OWL-ViT [29]/CORA [42] (OVD), OFA (REC) [39], and UNINEXT [44]/Grounding-DINO [25] (bi-functional) to provide a reference for the community. This benchmark may serve as a starting point for the DOD task.
Findings & improvements. The experimental analysis for different methods on  $\mathrm{D}^3$  yields some findings for future research (see Sec. 5): 1) Existing REC methods perform poorly, lacking confidence scores and the ability to reject negatives, and struggling with multi-target situations. This is due to their task formulation of grounding, i.e., matching between text and image region and not distinguishing positive and negatives. 2) OVD methods excel REC ones on DOD, though lengthy descriptions, which is not available in their training data, limit their performance. 3) Bi-functional methods, while superior to REC and OVD ones, share similar challenges with REC methods. Sometimes they are
Table 1: Comparison between the proposed dataset and previous REC datasets and OVD datasets.
Dataset	annotation completeness	unrestricted description	absence expression	instance-level annotation
RefCOCO	image-wise	✓	✗	✓
COCO	dataset-wise	✗	✗	✓
GRD	group-wise	✓	✗	✗
Ours	dataset-wise	✓	✓	✓
surpassed by OVD models, indicating they have not fully benefited from REC and OVD. Based on these findings, we propose a baseline OFA-DOD that greatly improves a REC method, and outperforms current SOTAs. Its abilities to handle multiple targets and reject negative instances are improved by simple data reconstruction and an auxiliary sub-task. It is still far from a strong DOD method, but may provide some insights for research in the future.
2.1 Relevant datasets and benchmarks
Object detection datasets. A variety of datasets have been proposed for object detection. Some have become standard benchmarks, like PASCAL VOC [9] and COCO [23]; while others are more frequently used for pretraining [35, 16, 2]. A few works have focused on special settings, such as LVIS [12] for long-tailed detection and ODinW [19] for zero-shot evaluation in the wild. Recently, V3Det [38] facilitates object detection with an extremely large vocabulary. Some are re-splitted and frequently used in OVD as well, like COCO and LVIS. As explained in Sec. 1, these datasets are all annotated with simple category labels rather than flexible language expressions like  $\mathbf{D}^3$ .
Referring expression comprehension datasets. Several datasets have been introduced to evaluate REC methods, including RefClef [15], RefCOCO [46], RefCOCO+ [46], RefCOCOg [28], Visual Genome [17], and PhraseCut [40]. Some [15, 46] are collected interactively, and the expressions are more concise and less diverse. RefCOCOg is collected non-interactively, resulting in more complex expressions. Comparatively, Visual Genome focuses on visual relationships. All these datasets only annotate a few positive images for each category and leave other images unknown, which makes them unsuitable for the detection task.
Other related tasks and datasets. Several related tasks and benchmarks exist, but they differ significantly from DOD. Phrase Detection [31] lacks explicit negative labels as negative instances are unlabeled, and does not constitute a true detection task. Additionally, its references are simply phrases. In contrast, DOD ensures exhaustive annotation of positive and negative labels, and its references can be words, phrases, or sentences. Cops-Ref benchmark [6] focuses on evaluating the grounding capability of REC methods in difficult negative regions with related and distracting targets. It provides explicit negative certificates for only a limited set of images. In  $\mathbf{D}^3$ , negative certificates are available across the entire dataset. Zero-shot grounding [34] centers on locating concepts not in the training set. It assumes the existence of the object referred by a reference in a image, and locates a single target per image, with a short phrase, while DOD makes no assumptions about the existence of the target, and locates zero to multiple targets, with varied expressions.
2.2 Current methods
Open-vocabulary object detection methods. Open-vocabulary detection is currently receiving increased attention. It aims to detect arbitrary classes using language for generalization, even when trained on a limited set of classes. The first approach, OVR-CNN [48], utilizes image-caption pairs for pretraining the visual encoder to enhance its zero-shot generalization capabilities. With the introduction of CLIP [33], models such as Detric [51], DetCLIP [45], RegionCLIP [50], and OVDETR [47] have further advanced image and language embeddings pretrained using CLIP. ViLD [11] further distills knowledge from CLIP to inherit language semantics for recognizing novel classes. GLIP [20, 49] formulates object detection as a phrase grounding problem [32] and utilizes additional phrase grounding data to facilitate vision-language alignment.
(a) Complete annotation.
(b) Unrestricted reference.
(c) Absence expression.
Figure 2: Some examples from previous datasets and the proposed  $\mathrm{D}^3$  dataset for DOD. (a) Our dataset for DOD is completely annotated for detection, while REC datasets like RefCOCO are not. (b) Our dataset has unrestricted reference, while OVD datasets like COCO are not. (c) Our dataset not only provides traditional presence descriptions, but also absence descriptions.
Referring expression comprehension methods. Existing works [7, 36, 21, 24, 39] can be divided into three categories. (1) Specialist models. Previously, two-staged works [13, 46] reformulate this as a ranking task. More recently, one-stage approaches [52, 37] speed up the inference process. (2) Multi-task models [53, 21, 24]. They usually design a unified formulation for a few closely related tasks. For example, SeqTR [53] unifies REC and RES (Referring Expression Segmentation) as a point prediction problem. (3) Multi-modal pre-training models [5, 26, 39]. Unified-IO [26] and OFA [39] propose unified sequence-to-sequence frameworks that can handle a variety of vision, language, and multi-modal tasks. Currently, OFA holds the SOTA among REC methods.
Bi-functional models for REC and OVD/OD. Some recent works [14, 8, 18, 25, 44] aim to handle tasks such as OVD (or OD) and REC concurrently within a single model. They typically restructure the training approach for these tasks, enabling a single model to learn from datasets related to both tasks. However, the inference process for each task remains distinct and independent of the other. FIBER [8] employs a two-stage pretraining strategy, separately utilizing image-text and image-text-box data to enhance data efficiency. More recently, Grounding-DINO [25] extends a closed-set detector by performing vision-language fusion at multiple stages and evaluating its performance on REC datasets. UNINEXT [44] reformulates various image and video tasks into a unified object discovery and retrieval paradigm. Despite these models sharing knowledge between detection and REC through pretraining, they are still treated as distinct tasks in these bi-functional models.
Methods with potential for DOD are continuously emerging and we will update them in this list.
3.1 Dataset highlight
The proposed dataset is re-annotated on GRD [43], a dataset for RES [53, 24] and its variants [41, 22]. As briefly introduced in Sec. 1, it contains three major characteristics. In Fig. 2, we show some examples from previous datasets and  $\mathbf{D}^3$  to highlight them. Here we elaborate on them with a few other characteristics:
The first is complete annotation. For REC, the instances referred to by one description are only annotated in a few images. For other images without the annotation of this description, it is unknown whether the corresponding instance exists or not. That is to say, their annotations are not complete. Contrarily, as shown in Fig. 2a, in  $\mathbf{D}^3$ , the objects referred to in all images by any description are annotated, as are the negative samples, like traditional object detection datasets.
The second is unrestricted language description. As shown in Fig. 2b, unlike (open vocabulary) object detection that retrieves objects with category names, we retrieve objects with language expressions, which is rather flexible. As is shown in Fig. 3d, the lengths of descriptions in  $\mathrm{D}^3$  vary a lot. The
(b)
Figure 3: Distribution of (a) number of positive images for a description in the dataset, (b) number of positive instances for a description, (c) number of instances in a positive image for a description, and (d) lengths of descriptions.
shortest descriptions have one or two words, where the DOD task downgrades to OVD, while the longest may have 15 or more words, resulting in rather complex language expressions.
The third is absence expression. Current datasets with language description, like RefCOCO series for REC, usually describe objects with certain features. They usually focus on the ability to discover the existence of concepts but neglect their absence. Noticing the missing ability to verify such capability, we also annotate objects lacking a certain attribute. Fig. 2c shows an example with presence description and another with absence description from  $\mathrm{D}^3$ . Such absence description makes up about one quarter of the references in this dataset. This is a first for existing benchmarks.
The fourth is instance-level annotation, a characteristic not held by GRD as it is intended for RES.
The fifth is one description can refer to multiple instances in an image, as in Fig. 3c. This is not true for REC datasets. If we regard category names as references, then OD datasets do have this feature.
In summary, the proposed dataset differs from the REC dataset primarily in terms of characteristics 1st, 3rd, and 5th. In contrast, when compared to OD datasets, the proposed dataset showcases disparities in the 2nd and 3rd characteristics, and when compared with GRD, in the 2nd, 3rd, 4th, and 5th characteristics. We refer the readers to the supplementary materials for more information about the characteristics of  $\mathrm{D}^3$  and more examples.
3.2 Annotation process
We utilize the GRD dataset [43] as the source for images, along with its original annotations. Originally, it is divided into multiple groups, each containing several references, with positive and negative samples annotated only within each group. We extend the annotations in three aspects:
Adding instance-level annotation. GRD is designed for RES, where each reference corresponds to one semantic mask across one image. However, for the DOD task, which requires the recognition and localization of individual instances, we annotate each instance referred to by a description with an individual bounding box (along with an instance mask). This is the basic step to adapt the dataset for instance localization.
Adding complete annotations. In addition to the intra-group annotation in GRD, we further annotate the positive and negative samples for each reference across the entire dataset. With complete dataset-wise annotations, the division into groups becomes unnecessary for evaluation, serving only as a means to organize references by scenarios. This enhancement makes the dataset suitable for detection tasks, significantly increasing the number of positive and negative samples.
Note that we use the complete annotation similar to COCO [23], i.e., explicit positive and negative certificates for all categories on all images, rather than federated annotation [12, 16]. This allows using mAP (mean Average Precision) as the evaluation metric, which is elaborated in Sec. 3.4.
Adding annotations for absence expressions. We have designed many absence descriptions based on the scenarios within the dataset, in addition to the traditional presence expressions in GRD. We annotate the instances in the images across the entire dataset with these absence expressions. This step increases the difficulty level of the proposed benchmark and enables the evaluation of existing models' ability to comprehend the absence of concepts.
We present a concise overview of the overall annotation process here. We organize groups of images and references (both for presence and absence). For each image, the references in its group are used. References from other groups may also appear, but with lower probability. We employ CLIP [33] to select a large number of candidates from these references in other groups. We manually check and adjust the hyper-parameters to make sure that such CLIP filtering usually do not miss positive refs. Subsequently, annotators select the positive references from these candidates (rather than from all references in the dataset) and add bounding boxes to the images. For more detailed information regarding the annotation process, please refer to supplementary materials.
3.3 Dataset statistics
GRD statistics. It has 10,578 images collected online, divided into 106 groups. Each group has around 100 images and 3 expressions referring to segmentation masks in this group, resulting in 316 references, 9,323 positive image-text pairs and 22,201 negative pairs. Note that it only annotates positive and negative samples inside each group, i.e., the annotation completeness is only group-level, so a reference will not be verified outside its group. The expressions have an average length of 5.9 words. We refer the reader to the original paper for specific statistics of GRD.
$\mathbf{D}^3$  statistics. The proposed  $\mathrm{D}^3$  has 10,578 images, all from GRD. It has 422 well-designed expressions, including 316 expressions from GRD and 106 absence expressions we added (one for each scenario). The instance-level annotation results in 18,514 boxes.
Due to the effort in complete annotation, for a reference, each image in the dataset is annotated for possible positive and negative samples, i.e., the annotation completeness is dataset-level. Thus, there are 24,282 positive object-text pairs and 7,788,626 negative pairs, orders of magnitude larger than GRD. Among them, those with images and texts from the same scenario are probably more difficult, which includes 20,279 positive and 53,383 negative pairs. The average length of expressions is 6.3 words, due to the relative longer absence expressions. More statistics and examples of  $\mathrm{D}^3$  are available in supplementary materials.
3.4 Evaluation metrics
The classification of instances in  $\mathbf{D}^3$  is multi-label. Each description corresponds to a category. Naturally, there can be relationships between categories, such as parent-child hierarchies, synonyms, and partial overlap. When designing categories, we intentionally reduce parent-child or synonym relationships to ensure greater diversity and challenge. However, there exists partial overlap between categories. Therefore, in  $\mathbf{D}^3$ , one instance may correspond to multiple descriptions, and the classification in  $\mathbf{D}^3$  is multi-label [12] rather than single-label [23], making it suitable for categories with relationships. An effective detector should assign all relevant positive categories (e.g., dog not led by rope outside and clothed dog for a clothed dog not led by rope outside) for an instance.
We use standard mAP for evaluation. Given the multi-label setting and the exhaustive annotation (all positive and negative labels are known for an instance) of  $\mathbf{D}^3$ , category relationships will not affect the evaluation, so we can use consistent evaluation for each category across all images. We describe the evaluation process here. For inference, an instance predicted with category A and B is regarded as an instance for category A and an instance for B. The AP for each category is computed as follows: Predictions for each category across all images are sorted by score in descending order, and those with a ground truth IoU exceeding a threshold are counted as TP (and the ground truth is marked as taken), while the rest are counted as false positives. With these TP and FP instances, we calculate the precision, recall, and AP. The mAP is calculated by averaging the AP across all categories.
We use FULL, PRES, and ABS to denote evaluation on all descriptions, presence descriptions only, and absence descriptions only. If not noted explicitly, the FULL setting is adopted. The specific metrics for  $\mathrm{D}^3$  include: Intra-scenario mAP: For this metric, we perform evaluation on each image with only the descriptions from the image's scenario. The final metric is the mAP averaged on different IoU thresholds from 0.5 to 0.95, following COCO [23]. This is used as the default metric in our experimental settings. Inter-scenario mAP: It is similar to the intra-scenario mAP described above, except that for each image, we detect the possible instances with all 422 references. This is aligned with the common mAP in object detection datasets [23] and is much more challenging than the intra-scenario mAP.
4.1 Existing baselines from different tasks
We choose multiple advanced methods to verify on  $\mathrm{D}^3$  from OVD, REC to bi-functional methods. More details of these methods and their inference process are in our supplementary materials.