Title: Described Object Detection: Liberating Object Detection with Flexible Expressions
Venue: NeurIPS 2023
Accepted: yes
Keywords: open-vocabulary object detection

referring expression comprehension

multi-modal detection
Abstract: Detecting objects based on language information is a popular task that includes Open-Vocabulary object Detection (OVD) and Referring Expression Comprehension (REC). In this paper, we advance them to a more practical setting called *Described Object Detection* (DOD) by expanding category names to flexible language expressions for OVD and overcoming the limitation of REC only grounding the pre-existing object. We establish the research foundation for DOD by constructing a *Description Detection Dataset* ($D^3$). This dataset features flexible language expressions, whether short category names or long descriptions, and annotating all described objects on all images without omission. By evaluating previous SOTA methods on $D^3$, we find some troublemakers that fail current REC, OVD, and bi-functional methods. REC methods struggle with confidence scores, rejecting negative instances, and multi-target scenarios, while OVD methods face constraints with long and complex descriptions. Recent bi-functional methods also do not work well on DOD due to their separated training procedures and inference strategies for REC and OVD tasks. Building upon the aforementioned findings, we propose a baseline that largely improves REC methods by reconstructing the training data and introducing a binary classification sub-task, outperforming existing methods. Data and code are available at https://github.com/shikras/d-cube and related works are tracked in https://github.com/Charles-Xie/awesome-described-object-detection.

Described Object Detection: Liberating Object Detection with Flexible Expressions

Abstract

Detecting objects based on language information is a popular task that includes Open-Vocabulary object Detection (OVD) and Referring Expression Comprehension (REC). In this paper, we advance them to a more practical setting called Described Object Detection (DOD) by expanding category names to flexible language expressions for OVD and overcoming the limitation of REC only grounding the pre-existing object. We establish the research foundation for DOD by constructing a Description Detection Dataset (D 3 ). This dataset features flexible language expressions, whether short category names or long descriptions, and annotating all described objects on all images without omission. By evaluating previous SOTA methods on D 3 , we find some troublemakers that fail current REC, OVD, and bi-functional methods. REC methods struggle with confidence scores, rejecting negative instances, and multi-target scenarios, while OVD methods face constraints with long and complex descriptions. Recent bi-functional methods also do not work well on DOD due to their separated training procedures and inference strategies for REC and OVD tasks. Building upon the aforementioned findings, we propose a baseline that largely improves REC methods by reconstructing the training data and introducing a binary classification sub-task, outperforming existing methods. Data and code are available at this URL and related works are tracked in this repo.

1 Introduction

Detecting objects of interest within a scene using language is a pivotal area of focus. This field encompasses two key tasks: Open-Vocabulary object Detection (OVD) [10, 11, 20, 29, 47, 48] and Referring Expression Comprehension (REC) [21, 27, 24, 46, 52]. We present an intuitive illustration of these two settings in Fig. 1. The first task, OVD, expands the scope of object detection (OD) to any given short category name. However, these settings neglect the instances described by intricate descriptions. The second task, REC, focuses on spatially locating one target described by an expression and assumes the target must exist in the image. However, in real-world scenarios, if the described objects do not exist in the image, REC algorithms output false-positive results. Recent advancements have witnessed the joint training of bi-functional models, such as GroundingDINO [25] and UNINEXT [44], which involve both OVD and REC data. Notwithstanding, these models still rely on separate training procedures and inference strategies for OVD and REC, and evaluate these two tasks independently.

As shown in Fig. 1, a more practical detection algorithm should be able to detect any described category, whether long or short, complex or simple, while discarding predictions in images where targets are absent. In order to address this significant yet often overlooked scenario, we propose the concept of Described Object Detection (DOD) . Note that this setting is a superset of OVD and REC.

OVD

REC

DOD

oven

dog

a cat lying on the sofa

dog not led by ropes outside

a big blue dragon

a dog on the left of the image

person

person right next to a dog

individuals connected by ropes

∅

OVD

REC

Described Object

Detec1on

(DOD)

∅

Rela*onships

When the language expression is limited to a short category name, it becomes OVD. When we limit the images to detect objects known to be present in the images beforehand, it downgrades to REC.

Can the existing SOTA algorithms of the community support DOD tasks? To address this inquiry, this paper establishes the research foundation of DOD tasks by constructing a dataset, scrutinizing relevant methodologies, analyzing the relevant methods, and exploring improvement space.

Motivation & real-world application of DOD. OVD is limited to categorical detection, focusing on classes rather than specific attributes or relationships. It lacks detailed contextual understanding and cannot adapt to precise detection requirements from language. REC comprehend longer descriptions for attributes or relationships, but assumes the existence of one target in the image. This leads to false positives when the target is absent, limiting its practical usability. Consider detecting individuals without helmets on a construction site using camera data: OVD can detect helmets and people but not determine their relationship. REC locate one region in any image and generate false positives frequently. Existing solutions involve using separate models for object detection then relationship classification, or REC after image classification, both resulting in inefficiency.

Hence, there is a demand for language-based object detection: a model with strong generalization capabilities that can verify the existence of described objects in images and localize them based on arbitrary expressions. The proposed DOD task addresses this need and finds practical applications in: urban security, detecting dogs without leashes in communities, clothes hung outdoors on streets, overloaded vehicles , and fallen trees on roadsides ; network security, like identifying sensitive images with violence or bloodshed within large datasets; (fine-grained) photo album retrieval based on descriptions or keywords; retrieval and filtering of web image data; specific event detection in autonomous driving, such as pedestrians crossing the road .

Dataset & benchmark. For DOD, we introduce the Description Detection Dataset (D 3 , /dikju:b/), an evaluation-only benchmark containing 422 descriptions and 24,282 positive object-description pairs. Unlike previous OVD or REC datasets (see Fig. 2), D 3 stands out in three key aspects (see Tab. 1): 1) Complete annotation : All descriptions refer to objects annotated throughout the dataset, making D 3 a detection-style dataset akin to COCO [23]. 2) Unrestricted description : Annotations in D 3 include diverse and flexible language expressions, varying in length and complexity. 3) Absence expression : We include descriptions regarding absence of concepts, such as a person without a safety helmet , addressing an often-overlooked detection requirement. The details of D 3 is elaborated in Sec. 3. We evaluate state-of-the-art methods on D 3 : OWL-ViT [29]/CORA [42] (OVD), OFA (REC) [39], and UNINEXT [44]/Grounding-DINO [25] (bi-functional) to provide a reference for the community. This benchmark may serve as a starting point for the DOD task.

Findings & improvements. The experimental analysis for different methods on D 3 yields some findings for future research (see Sec. 5): 1) Existing REC methods perform poorly, lacking confidence scores and the ability to reject negatives, and struggling with multi-target situations. This is due to their task formulation of grounding, i.e., matching between text and image region and not distinguishing positive and negatives. 2) OVD methods excel REC ones on DOD, though lengthy descriptions, which is not available in their training data, limit their performance. 3) Bi-functional methods, while superior to REC and OVD ones, share similar challenges with REC methods. Sometimes they are

surpassed by OVD models, indicating they have not fully benefited from REC and OVD. Based on these findings, we propose a baseline OFA-DOD that greatly improves a REC method, and outperforms current SOTAs. Its abilities to handle multiple targets and reject negative instances are improved by simple data reconstruction and an auxiliary sub-task. It is still far from a strong DOD method, but may provide some insights for research in the future.

2 Related Work

2.1 Relevant datasets and benchmarks

Object detection datasets. A variety of datasets have been proposed for object detection. Some have become standard benchmarks, like PASCAL VOC [9] and COCO [23]; while others are more frequently used for pretraining [35, 16, 2]. A few works have focused on special settings, such as LVIS [12] for long-tailed detection and ODinW [19] for zero-shot evaluation in the wild. Recently, V3Det [38]facilitates object detection with an extremely large vocabulary. Some are re-splitted and frequently used in OVD as well, like COCO and LVIS. As explained in Sec. 1, these datasets are all annotated with simple category labels rather than flexible language expressions like D 3 .

Referring expression comprehension datasets. Several datasets have been introduced to evaluate REC methods, including RefClef [15], RefCOCO [46], RefCOCO+ [46], RefCOCOg [28], Visual Genome [17], and PhraseCut [40]. Some [15, 46] are collected interactively, and the expressions are more concise and less diverse. RefCOCOg is collected non-interactively, resulting in more complex expressions. Comparatively, Visual Genome focuses on visual relationships. All these datasets only annotate a few positive images for each category and leave other images unknown, which makes them unsuitable for the detection task.

Other related tasks and datasets. Several related tasks and benchmarks exist, but they differ significantly from DOD. Phrase Detection [31] lacks explicit negative labels as negative instances are unlabeled, and does not constitute a true detection task. Additionally, its references are simply phrases. In contrast, DOD ensures exhaustive annotation of positive and negative labels, and its references can be words, phrases, or sentences. Cops-Ref benchmark [6] focuses on evaluating the grounding capability of REC methods in difficult negative regions with related and distracting targets. It provides explicit negative certificates for only a limited set of images. In D 3 , negative certificates are available across the entire dataset. Zero-shot grounding [34] centers on locating concepts not in the training set. It assumes the existence of the object referred by a reference in a image, and locates a single target per image, with a short phrase, while DOD makes no assumptions about the existence of the target, and locates zero to multiple targets, with varied expressions.

2.2 Current methods

Open-vocabulary object detection methods. Open-vocabulary detection is currently receiving increased attention. It aims to detect arbitrary classes using language for generalization, even when trained on a limited set of classes. The first approach, OVR-CNN [48], utilizes image-caption pairs for pretraining the visual encoder to enhance its zero-shot generalization capabilities. With the introduction of CLIP [33], models such as Detic [51], DetCLIP [45], RegionCLIP [50], and OVDETR [47] have further advanced image and language embeddings pretrained using CLIP. ViLD [11] further distills knowledge from CLIP to inherit language semantics for recognizing novel classes. GLIP [20, 49] formulates object detection as a phrase grounding problem [32] and utilizes additional phrase grounding data to facilitate vision-language alignment.

'the dog on the le+'

'a baseball player

ready to bat'

'an oversized glove worn on the le+

hand of a baseball player'

refer

miss

refer

refer

refer

refer

REC

✗

DOD

✓

(a) Complete annotation.

'leather bag

with glossy

surface'

'backpack'

refer

refer

'wine glass'

'bo7le'

refer

OVD

✗

DOD

✓

refer

(b) Unrestricted reference.

'well cover

without

text

on the

surface'

refer

refer

'well cover

with text on

the surface'

DOD

✓

DOD

✓

(c) Absence expression.

Referring expression comprehension methods. Existing works [7, 36, 21, 24, 39] can be divided into three categories. (1) Specialist models. Previously, two-staged works [13, 46] reformulate this as a ranking task. More recently, one-stage approaches [52, 37] speed up the inference process. (2) Multi-task models [53, 21, 24]. They usually design a unified formulation for a few closely related tasks. For example, SeqTR [53] unifies REC and RES (Referring Expression Segmentation) as a point prediction problem. (3) Multi-modal pre-training models [5, 26, 39]. Unified-IO [26] and OFA [39] propose unified sequence-to-sequence frameworks that can handle a variety of vision, language, and multi-modal tasks. Currently, OFA holds the SOTA among REC methods.

Bi-functional models for REC and OVD/OD. Some recent works [14, 8, 18, 25, 44] aim to handle tasks such as OVD (or OD) and REC concurrently within a single model. They typically restructure the training approach for these tasks, enabling a single model to learn from datasets related to both tasks. However, the inference process for each task remains distinct and independent of the other. FIBER [8] employs a two-stage pretraining strategy, separately utilizing image-text and image-textbox data to enhance data efficiency. More recently, Grounding-DINO [25] extends a closed-set detector by performing vision-language fusion at multiple stages and evaluating its performance on REC datasets. UNINEXT [44] reformulates various image and video tasks into a unified object discovery and retrieval paradigm. Despite these models sharing knowledge between detection and REC through pretraining, they are still treated as distinct tasks in these bi-functional models.

Methods with potential for DOD are continuously emerging and we will update them in this list.

3 Dataset

3.1 Dataset highlight

The proposed dataset is re-annotated on GRD [43], a dataset for RES [53, 24] and its variants [41, 22]. As briefly introduced in Sec. 1, it contains three major characteristics. In Fig. 2, we show some examples from previous datasets and D 3 to highlight them. Here we elaborate on them with a few other characteristics:

The first is complete annotation . For REC, the instances referred to by one description are only annotated in a few images. For other images without the annotation of this description, it is unknown whether the corresponding instance exists or not. That is to say, their annotations are not complete. Contrarily, as shown in Fig. 2a, in D 3 , the objects referred to in all images by any description are annotated, as are the negative samples, like traditional object detection datasets.

The second is unrestricted language description . As shown in Fig. 2b, unlike (open vocabulary) object detection that retrieves objects with category names, we retrieve objects with language expressions, which is rather flexible. As is shown in Fig. 3d, the lengths of descriptions in D 3 vary a lot. The

0

50

100

150

200

Num. of images

0

20

40

60

80

Count

(a)

0

200

400

Num. of instances

0

25

50

75

100

Count

(b)

5

10

15

Num. of instances on a image

0

2500

5000

7500

10000

Count

(c)

5

10

15

Lengths of descriptions

0

20

40

Count

(d)

shortest descriptions have one or two words, where the DOD task downgrades to OVD, while the longest may have 15 or more words, resulting in rather complex language expressions.

The third is absence expression . Current datasets with language description, like RefCOCO series for REC, usually describe objects with certain features. They usually focus on the ability to discover the existence of concepts but neglect their absence. Noticing the missing ability to verify such capability, we also annotate objects lacking a certain attribute. Fig. 2c shows an example with presence description and another with absence description from D 3 . Such absence description makes up about one quarter of the references in this dataset. This is a first for existing benchmarks.

The fourth is instance-level annotation , a characteristic not held by GRD as it is intended for RES.

The fifth is one description can refer to multiple instances in an image, as in Fig. 3c. This is not true for REC datasets. If we regard category names as references, then OD datasets do have this feature.

In summary, the proposed dataset differs from the REC dataset primarily in terms of characteristics 1st, 3rd, and 5th. In contrast, when compared to OD datasets, the proposed dataset showcases disparities in the 2nd and 3rd characteristics, and when compared with GRD, in the 2nd, 3rd, 4th, and 5th characteristics. We refer the readers to the supplementary materials for more information about the characteristics of D 3 and more examples.

3.2 Annotation process

We utilize the GRD dataset [43] as the source for images, along with its original annotations. Originally, it is divided into multiple groups, each containing several references, with positive and negative samples annotated only within each group. We extend the annotations in three aspects:

Adding instance-level annotation. GRD is designed for RES, where each reference corresponds to one semantic mask across one image. However, for the DOD task, which requires the recognition and localization of individual instances, we annotate each instance referred to by a description with an individual bounding box (along with an instance mask). This is the basic step to adapt the dataset for instance localization.

Adding complete annotations. In addition to the intra-group annotation in GRD, we further annotate the positive and negative samples for each reference across the entire dataset. With complete datasetwise annotations, the division into groups becomes unnecessary for evaluation, serving only as a means to organize references by scenarios. This enhancement makes the dataset suitable for detection tasks, significantly increasing the number of positive and negative samples.

Note that we use the complete annotation similar to COCO [23], i.e., explicit positive and negative certificates for all categories on all images, rather than federated annotation [12, 16]. This allows using mAP (mean Average Precision) as the evaluation metric, which is elaborated in Sec. 3.4.

Adding annotations for absence expressions. We have designed many absence descriptions based on the scenarios within the dataset, in addition to the traditional presence expressions in GRD. We annotate the instances in the images across the entire dataset with these absence expressions. This step increases the difficulty level of the proposed benchmark and enables the evaluation of existing models' ability to comprehend the absence of concepts.

We present a concise overview of the overall annotation process here. We organize groups of images and references (both for presence and absence). For each image, the references in its group are used. References from other groups may also appear, but with lower probability. We employ CLIP [33] to select a large number of candidates from these references in other groups. We manually check and adjust the hyper-parameters to make sure that such CLIP filtering usually do not miss positive refs. Subsequently, annotators select the positive references from these candidates (rather than from all references in the dataset) and add bounding boxes to the images. For more detailed information regarding the annotation process, please refer to supplementary materials .

3.3 Dataset statistics

GRD statistics. It has 10,578 images collected online, divided into 106 groups. Each group has around 100 images and 3 expressions referring to segmentation masks in this group, resulting in 316 references, 9,323 positive image-text pairs and 22,201 negative pairs. Note that it only annotates positive and negative samples inside each group, i.e., the annotation completeness is only group-level , so a reference will not be verified outside its group. The expressions have an average length of 5.9 words. We refer the reader to the original paper for specific statistics of GRD.

D 3 statistics. The proposed D 3 has 10,578 images, all from GRD. It has 422 well-designed expressions, including 316 expressions from GRD and 106 absence expressions we added (one for each scenario). The instance-level annotation results in 18,514 boxes.

Due to the effort in complete annotation , for a reference, each image in the dataset is annotated for possible positive and negative samples, i.e., the annotation completeness is dataset-level . Thus, there are 24,282 positive object-text pairs and 7,788,626 negative pairs, orders of magnitude larger than GRD. Among them, those with images and texts from the same scenario are probably more difficult, which includes 20,279 positive and 53,383 negative pairs. The average length of expressions is 6.3 words, due to the relative longer absence expressions. More statistics and examples of D 3 are available in supplementary materials .

3.4 Evaluation metrics

The classification of instances in D 3 is multi-label . Each description corresponds to a category. Naturally, there can be relationships between categories, such as parent-child hierarchies, synonyms, and partial overlap. When designing categories, we intentionally reduce parent-child or synonym relationships to ensure greater diversity and challenge. However, there exists partial overlap between categories. Therefore, in D 3 , one instance may correspond to multiple descriptions, and the classification in D 3 is multi-label [12] rather than single-label [23], making it suitable for categories with relationships. An effective detector should assign all relevant positive categories (e.g., dog not led by rope outside and clothed dog for a clothed dog not led by rope outside) for an instance.

We use standard mAP for evaluation. Given the multi-label setting and the exhaustive annotation (all positive and negative labels are known for an instance) of D 3 , category relationships will not affect the evaluation, so we can use consistent evaluation for each category across all images. We describe the evaluation process here. For inference, an instance predicted with category A and B is regarded as an instance for category A and an instance for B. The AP for each category is computed as follows: Predictions for each category across all images are sorted by score in descending order, and those with a ground truth IoU exceeding a threshold are counted as TP (and the ground truth is marked as taken), while the rest are counted as false positives. With these TP and FP instances, we calculate the precision, recall, and AP. The mAP is calculated by averaging the AP across all categories.

We use FULL , PRES , and ABS to denote evaluation on all descriptions, presence descriptions only, and absence descriptions only. If not noted explicitly, the FULL setting is adopted. The specific metrics for D 3 include: Intra-scenario mAP: For this metric, we perform evaluation on each image with only the descriptions from the image's scenario. The final metric is the mAP averaged on different IoU thresholds from 0.5 to 0.95, following COCO [23]. This is used as the default metric in our experimental settings. Inter-scenario mAP: It is similar to the intra-scenario mAP described above, except that for each image, we detect the possible instances with all 422 references. This is aligned with the common mAP in object detection datasets [23] and is much more challenging than the intra-scenario mAP.

4 Baselines

4.1 Existing baselines from different tasks

We choose multiple advanced methods to verify on D 3 from OVD, REC to bi-functional methods. More details of these methods and their inference process are in our supplementary materials .

REC methods. We employ the state-of-the-art REC method, OFA [39], with two variants. OFA is based on an encoder-decoder, sequence-to-sequence framework. It is a multi-modal multi-task generalist that deals with different tasks together and is trained on various tasks, including language tasks (masked language modeling), image-to-text tasks (image captioning and Visual Question Answering (VQA)), and localization tasks (REC). Notably, although it is trained with a detection dataset, it is not evaluated on object detection and achieves poor performance if we do. Currently, it holds the SOTA performance on standard REC benchmarks like the RefCOCO series.

OVD methods. We evaluate OWL-ViT [29] with two variants and CORA [42]. They are the SOTA methods on OVD tasks, with a vision transformer as well as a language transformer. They are pretrained with image-text contrastive learning and then fine-tuned on detection dataset.

Bi-functional methods utilizing both REC and OVD data. Methods falling into this category are not many but emerging fast recently. We test two methods: Grounding-DINO [25] and UNINEXT [44], each with two variants. Both of them are based on DETR [3]. They are pretrained on multiple datasets, including detection and REC datasets, and then evaluated with different strategies for different tasks.

4.2 A proposed baseline

D 3 is very challenging for existing works, as we will demonstrate in Sec. 5.1. We have selected one of these works for adjustment to provide a better baseline. The chosen work should (1) be capable of understanding text of various lengths; (2) excel in their original tasks; (3) have a framework with a rather simple technical design, allowing us to modify its components easily. We have chosen OFA because it (1) is a multi-modal multi-task framework with MLM (Masked Language Modeling) and image-to-text pretraining; (2) achieves SOTA on REC; (3) has a simple seq2seq framework.

However, OFA faces several problems that make it unsatisfactory for this task, as discussed in Sec. 5. First, forcing multiple tasks of different modalities into one seq2seq framework adversely affects the performance of specific tasks, especially tasks related to localization. Second, training on the grounding task results in poor ability to handle multiple instances. We evaluated the model on COCO detection, and it achieved less than 10 mAP. Thirdly, its REC paradigm also makes it predict only one instance, making it unable to reject negative images and irrelevant descriptions.

Therefore, we have made some modifications to OFA to make it more suitable for this task. The first modification is granularity decomposition to make it more suitable for localization. We have divided the pretraining tasks of OFA into two different granularities: global tasks (related to language modeling, such as captioning, VQA, MLM, etc.) and local tasks (related to localization, such as detection and REC). We have added an additional decoder parallel to the original decoder in OFA that handles the local tasks, while the original decoder focuses on the global tasks. This alleviates conflicts between different tasks and enhances localization.

The second modification is reconstructed data for pretraining on REC, aiming to improve multitarget localization. We have reconstructed the data for REC to ensure that (1) multiple references are input for an image, and (2) a reference does not necessarily correspond to one object, but zero or multiple. This results in a unified data format for detection and REC, although the labels may be noisy since they were not initially prepared for DOD.

The third modification is task decomposition to empower the model with the ability to reject false positives. We have reformulated the training on reconstructed data into two tasks: REC (for locating a region based on a reference) and VQA (for determining if a region and a reference match each other, essentially a binary classification). The second step is responsible for rejecting false positives.

We refer to the model with all three modifications as OFA-DOD . More details on the proposed improvements can be found in the supplementary materials . It is important to note that this model is far from perfect for the complex D 3 benchmark. As we will show in Sec. 5.1, although it outperforms existing methods, it serves as a baseline for future tasks on D 3 .

0

25

50

TP Count

TP Scores

3

2

1

0

FP Scores

0

200

400

FP Count

(a) OFA

0

500

1000

TP Count

TP Scores

0.2

0.4

0.6

0.8

FP Scores

0

5000

10000

FP Count

(b) OWL-ViT

0

50

100

TP Count

TP Scores

0.2

0.4

0.6

0.8

1.0

FP Scores

0

100

200

FP Count

(c) Grounding-DINO

0

500

TP Count

TP Scores

0.6

0.4

0.2

0.0

FP Scores

0

500

FP Count

(d) OFA-DOD

5 Experimental Analyses

5.1 Comparison of baselines on our metrics

We make comparisons on the baselines introduced in Sec. 4, mainly with the intra-scenario setting. Unless explicitly noted, this is the default setting, instead of the more difficult inter-scenario.

Existing SOTAs are insufficient for DOD, and bi-functional models outperform others. As demonstrated in Tab. 2, existing methods, while achieving SOTA performance on their original benchmarks, fall short in delivering strong performance on D 3 . Among them, recent bi-functional methods [25, 44] are notably superior to others, and currently, OVD methods outperform REC. The inferiority of REC methods is likely due to their impractical setting described in Sec. 1, which involves predicting one and only one instance for each reference. We will delve into this further.

Rejecting irrelevant references are difficult, which REC are naturally incapable of. In contrast to intra-scenario evaluation, the inter-scenario setting assesses all references in the dataset for each image. Since references from other scenarios are likely not semantically relevant to the images, this necessitates the ability to reject irrelevant references for an image. This aligns with the evaluation in standard detection tasks. From Tab. 2, it is evident that OFA, a REC method, almost completely fails in this setting. This is caused by its prediction of a region for every reference, resulting in a large number of false positives when there are numerous candidate references. This underscores the importance of empowering REC methods with the ability to reject false positives. We find that none of the verified methods achieve good performance under the inter-scenario setting, indicating that existing methods are far from being capable of DOD. This highlights the challenge of D 3

The proposed baseline outperforms existing methods. The proposed baseline is based on OFA, but our improvements significantly enhance its performance. It outperforms all existing methods in the intra-scenario setting and surpasses them by a wider margin in the inter-scenario setting. This may suggest that the proposed baseline has a stronger ability to reject irrelevant references. Nonetheless, the proposed method is far from perfect and can only serve as a baseline for future research.

5.2 Further analysis

Absence descriptions are more difficult for most methods. As shown in Tab. 2, the performance of baseline methods on PRES (presence descriptions) is consistently superior to that on ABS (absence descriptions). This suggests that existing methods may not effectively differentiate between the presence and absence of attributes in a language description.

REC methods fail to provide good confidence scores. We visualized the score distributions from baselines for TPs and FPs, to assess their capabilities in classification and confidence estimation. As in Fig. 4, the confidence scores from OFA do not exhibit a clear distinction between TP and FP cases. This can be attributed in part to the seq2seq framework in OFA, which does not directly yield confidence scores, and in part to the grounding formulation of REC, which identifies the image region most similar to the text description without distinguishing between positive and negative.

With a task decomposition step to enhance binary classification performance, our OFA-DOD demonstrates a significant disparity between TP and FP score distributions, yielding more reliable classification results. Note that this improvement does not necessitate modifications to the model framework or training datasets; rather, it is attributed to a more appropriate task formulation.

Multi-instance detection is challenging for methods other than OVD. For each image, D 3 can have zero to multiple instances for a single description . To assess how current methods handle varying numbers of instances, we conducted evaluations under three different settings: no-instance , where for a reference, evaluations are limited to images without any referred instance; one-instance , for images with a single instance; and multi-instance , for images with multiple instances. As shown in Tab. 3, OVD methods outperform others when multiple instances are referred by the description, although they may not be as competitive on the entire dataset or images with few instances. Notably, OWL-ViT maintains consistent performance even as the number of instances increases, which sets it apart from other methods. In contrast, REC and current bi-functional methods struggle in multi-instance scenarios. This highlights the strength of OVD methods in multi-target detection, while REC and current bi-functional approaches are less robust in such situations.

REC and bi-functional methods lack the ability to reject negative instances. In the no-instance column of Tab. 3, we do not report mAP since there are no positive instances in GT for the corresponding reference, making AP inapplicable. Predictions on such images are FPs, so we measure the ratio of images where FPs are produced to the total number of no-instance images for a given reference, namely False Positives Per Category (FPPC). We report the average FPPC over all references. We observe that most baselines are incapable of determining whether an image contains the referred target or not, yet they still produce predictions. This behavior is expected for REC methods. Bi-functional methods, trained and inferred with the REC task formulation, also exhibit this issue. Only the OVD method and our proposed baseline can effectively reject such negative image-text pairs.

OVD methods suffer from long descriptions greatly while others do not. We partition the references according to their lengths and then evaluate on these partitions. The results are shown in Tab. 4, where short , middle , long and very long corresponding to references with 1~3, 4~6, 7~9, and more than 9 words. For short descriptions, which is close to OVD setting, OVD and bi-functional methods obtain similar performance. However, as the length of references increases, the performance of OVD methods decrease fast, while REC and bi-functional methods suffer less from this. We can see that OVD methods are sensitive to long references, as expected, while other two types do not.

More experiments and additional qualitative results are available in supplementary materials .

5.3 Ablation on the proposed baseline

Method components. In Tab. 5, we perform ablation on the proposed improvements in our baseline, step-by-step from OFA to OFA-DOD, to see how they affect the performance. Granularity decomposition (GD) makes the method more suitable for localization task. It disentangle tasks of global or local granularity by handling them with 2 separated branch. Reconstructed data (RD) uniforms REC and OD data into the same form, and prepares multi-instance samples with both short and long references. Task decomposition (TD) is proposed to help rejecting FPs. It breaks down the DOD task into a REC step followed by a VQA step. All three of them improve the performance obviously.

Training tasks. We also perform a drop-one-out ablation on the multi-modal multi-task training data, in Tab. 5b. Detection data provides samples for localization, especially multi-instance situation. It is instinctively important for learning to localize, and indeed matters for performance. I2T (image-totext, like image captioning and visual question answering) often helps the generalization and zero-shot performance of multi-modal methods. We find that it does affect the zero-shot performance on D 3 greatly. MLM is theoretically important for language understanding and generalization. However, we find it actually is not. Removing the MLM task has no significant effect on the performance. We surmise that the generalization ability of OFA-DOD on D 3 mainly comes from I2T.

6 Conclusion and Limitation

In this paper, we bring the Described Object Detection (DOD) task to the foreground. For this task, we introduce a dataset called D 3 , which annotates described objects without omission and features flexible language expressions, whether long or short, complex or simple. Our evaluation of SOTA methods from REC or OVD on D 3 reveals challenges faced by REC, OVD, and bi-functional approaches. Based on these observations, we propose a baseline that largely improves REC methods for DOD task. We believe that the dataset and findings will contribute to advancing the understanding and development of DOD methods, facilitating future research in this area.

Limitation and broader impact. This work does have some limitations. Due to the significant annotation cost brought by our complete annotation process, we are unable to propose a huge dataset with millions or billions of images. Besides, the evaluation and findings in this work may be dependent on the choice of descriptions and the image sources. This work only serves as a starting point for DOD and we hope there will be other DOD datasets with larger scales. In the broader community, compared to traditional detection algorithms, DOD models have a lower customization threshold, enabling users to specify the detection target using language. This may lead to potential abuse.

Future work. During peer-review process, some new works with potential for DOD emerges, including Shikra [4], Kosmos-2 [30] and Qwen-VL [1]. We will continue to investigate such methods for DOD and update them in this list.

Acknowledgments. This work was supported in part by the National Natural Science Foundation of China under Grant 62076183, 61936014 and 61976159, in part by the Natural Science Foundation of Shanghai under Grant 20ZR1473500, in part by the Shanghai Science and Technology Innovation Action Project under Grant 20511100700 and 22511105300, in part by the Shanghai Municipal Science and Technology Major Project under Grant 2021SHZDZX0100, and in part by Hetao Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone (HZQB-KCZYZ2021045). The authors would also like to thank the anonymous reviewers for their careful work and valuable suggestions.

References

J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 , 2023.

L. Cai, Z. Zhang, Y. Zhu, L. Zhang, M. Li, and X. Xue. Bigdetection: A large-scale benchmark for improved object detector pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4777-4787, 2022.

N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16 , pages 213-229. Springer, 2020.

K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao. Shikra: Unleashing multimodal llm's referential dialogue magic. arXiv preprint arXiv:2306.15195 , 2023.

Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Universal image-text representation learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX , pages 104-120. Springer, 2020.

Z. Chen, P. Wang, L. Ma, K.-Y. K. Wong, and Q. Wu. Cops-ref: A new dataset and task on compositional referring expression comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10086-10095, 2020.

J. Deng, Z. Yang, T. Chen, W. Zhou, and H. Li. Transvg: End-to-end visual grounding with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1769-1779, 2021.

Z.-Y. Dou, A. Kamath, Z. Gan, P. Zhang, J. Wang, L. Li, Z. Liu, C. Liu, Y. LeCun, N. Peng, et al. Coarse-to-fine vision-language pre-training with fusion in the backbone. Advances in neural information processing systems , 35:32942-32956, 2022.

M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 2010.

G. Ghiasi, X. Gu, Y. Cui, and T.-Y. Lin. Scaling open-vocabulary image segmentation with image-level labels. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI , pages 540-557. Springer, 2022.

X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations , 2022.

A. Gupta, P. Dollar, and R. Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5356-5364, 2019.

R. Hu, M. Rohrbach, J. Andreas, T. Darrell, and K. Saenko. Modeling relationships in referential expressions with compositional modular networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1115-1124, 2017.

A. Kamath, M. Singh, Y. LeCun, G. Synnaeve, I. Misra, and N. Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1780-1790, 2021.

S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP , pages 787-798, 2014.

I. Krasin, T. Lin, T. Duerig, P. Krähenbühl, A. Gupta, C. Burgess, and V. Ferrari. OpenImages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://github.com/openimages , 2017.

R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. IJCV , 123:32-73, 2017.

W. Kuo, F. Bertsch, W. Li, A. Piergiovanni, M. Saffar, and A. Angelova. Findit: Generalized localization with natural language queries. In European Conference on Computer Vision . Springer, 2022.

C. Li, H. Liu, L. Li, P. Zhang, J. Aneja, J. Yang, P. Jin, H. Hu, Z. Liu, Y. J. Lee, et al. Elevater: A benchmark and toolkit for evaluating language-augmented visual models. Advances in Neural Information Processing Systems , 35:9287-9301, 2022.

L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J.-N. Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10965-10975, 2022.

M. Li and L. Sigal. Referring transformer: A one-step approach to multi-task visual grounding. Advances in neural information processing systems , 34:19652-19664, 2021.

X. Li, J. Wang, X. Xu, X. Li, B. Raj, and Y. Lu. Robust referring video object segmentation with cyclic structural consensus. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22236-22245, 2023.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common objects in context. In ECCV , pages 740-755. Springer, 2014.

J. Liu, H. Ding, Z. Cai, Y . Zhang, R. K. Satzoda, V . Mahadevan, and R. Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. In CVPR , 2023.

S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 , 2023.

J. Lu, C. Clark, R. Zellers, R. Mottaghi, and A. Kembhavi. UNIFIED-IO: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations , 2023.

G. Luo, Y. Zhou, X. Sun, L. Cao, C. Wu, C. Deng, and R. Ji. Multi-task collaborative network for joint referring expression comprehension and segmentation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages 10034-10043, 2020.

J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 11-20, 2016.

M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Mahendran, A. Arnab, M. Dehghani, Z. Shen, et al. Simple open-vocabulary object detection. In European Conference on Computer Vision , pages 728-755. Springer, 2022.

Z. Peng, W. Wang, L. Dong, Y. Hao, S. Huang, S. Ma, and F. Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824 , 2023.

B. A. Plummer, K. J. Shih, Y. Li, K. Xu, S. Lazebnik, S. Sclaroff, and K. Saenko. Revisiting imagelanguage networks for open-ended phrase detection. IEEE transactions on pattern analysis and machine intelligence , 44(4):2155-2167, 2020.

B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision , pages 2641-2649, 2015.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

A. Sadhu, K. Chen, and R. Nevatia. Zero-shot grounding of objects from natural language queries. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4694-4703, 2019.

S. Shao, Z. Li, T. Zhang, C. Peng, G. Yu, X. Zhang, J. Li, and J. Sun. Objects365: A large-scale, highquality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision , pages 8430-8439, 2019.

S. Song, X. Lin, J. Liu, Z. Guo, and S.-F. Chang. Co-grounding networks with semantic attention for referring expression comprehension in videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1346-1355, 2021.

S. Subramanian, W. Merrill, T. Darrell, M. Gardner, S. Singh, and A. Rohrbach. Reclip: A strong zero-shot baseline for referring expression comprehension. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics , Dublin, Ireland, May 2022. Association for Computational Linguistics.

J. Wang, P. Zhang, T. Chu, Y. Cao, Y. Zhou, T. Wu, B. Wang, C. He, and D. Lin. V3det: Vast vocabulary visual detection dataset. In The IEEE International Conference on Computer Vision (ICCV) , October 2023.

P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning , pages 23318-23340. PMLR, 2022.

C. Wu, Z. Lin, S. Cohen, T. Bui, and S. Maji. Phrasecut: Language-based image segmentation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10216-10225, 2020.

J. Wu, X. Li, X. Li, H. Ding, Y. Tong, and D. Tao. Towards robust referring image segmentation. arXiv preprint arXiv:2209.09554 , 2022.

X. Wu, F. Zhu, R. Zhao, and H. Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7031-7040, 2023.

Y. Wu, Z. Zhang, C. Xie, F. Zhu, and R. Zhao. Advancing referring expression segmentation beyond single image. In International Conference on Computer Vision (ICCV) , 2023.

B. Yan, Y. Jiang, J. Wu, D. Wang, P. Luo, Z. Yuan, and H. Lu. Universal instance perception as object discovery and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15325-15336, 2023.

L. Yao, J. Han, Y. Wen, X. Liang, D. Xu, W. Zhang, Z. Li, C. Xu, and H. Xu. Detclip: Dictionaryenriched visual-concept paralleled pre-training for open-world detection. Advances in Neural Information Processing Systems , 35:9125-9138, 2022.

L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 , pages 69-85. Springer, 2016.

Y. Zang, W. Li, K. Zhou, C. Huang, and C. C. Loy. Open-vocabulary detr with conditional matching. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX , pages 106-122. Springer, 2022.

A. Zareian, K. D. Rosa, D. H. Hu, and S.-F. Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1439314402, 2021.

H. Zhang, P. Zhang, X. Hu, Y.-C. Chen, L. Li, X. Dai, L. Wang, L. Yuan, J.-N. Hwang, and J. Gao. Glipv2: Unifying localization and vision-language understanding. Advances in Neural Information Processing Systems , 35:36067-36080, 2022.

Y. Zhong, J. Yang, P. Zhang, C. Li, N. Codella, L. H. Li, L. Zhou, X. Dai, L. Yuan, Y. Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16793-16803, 2022.

X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra. Detecting twenty-thousand classes using image-level supervision. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX , pages 350-368. Springer, 2022.

Y. Zhou, R. Ji, G. Luo, X. Sun, J. Su, X. Ding, C.-W. Lin, and Q. Tian. A real-time global inference network for one-stage referring expression comprehension. IEEE Transactions on Neural Networks and Learning Systems , 2021.

C. Zhu, Y. Zhou, Y. Shen, G. Luo, X. Pan, M. Lin, C. Chen, L. Cao, X. Sun, and R. Ji. Seqtr: A simple yet universal network for visual grounding. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV , pages 598-615. Springer, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new multi-modal computer vision task, called Described Object Detection, which is a superset of existing OVD and REC tasks. In particular, the DOD task seeks to create models which can detect multiple instances of something in images, from textual descriptions, which could include describing the absence or presence of something. The paper then goes on to create a new dataset for this task, based on an existing one, called $D^3$. It then shows how existing OVD, REC, Bi-Functional, and proposed a baseline method perform on the data set.\n\nSTRENGTHS:\nThe paper is attacking a significant problem, is of high experimental quality, and is novel. The idea of having free-text descriptors for finding things in images is extremely important to the adoption of ML for various important, real-world tasks. In fact, I was surprised to learn that given the importance of this concept, something like DOD had not been proposed before. The paper is also of high quality in that it not only identifies this shortcoming of real-world, open-set detection tasks (i.e. DOD), but also creates a dataset, tests it across a range of existing methods, and proposes a new baseline. Because the paper does propose a new formal problem definition for something that is desired in the real-world performance of ML systems, and therefore realizes how current tasks like REC and OVD fall short of what is actually needed for things like open-set, zero-shot object detection it is a novel look at the problem domain. Finally, it is also worth mentioning that the inclusion of the absence examples is a novel idea, which shows in the poor performance of models on absence instances in the data set.\n\nWEAKNESSES:\nThe only major weakness of the paper is its clarity. This lack of clarity manifests primarily in two places. First, the description of the creation of the data set needs more detail. Specifically, how was CLIP utilized for adding complete annotations? Were full images given to CLIP with all of the possible annotations and then the most probable selected as an annotation for the image? And, if so, how did you deal with multiple annotations being present in the same images? Additionally, how were the negative image annotations created? Were these done by hand, and, if so, by how many people? Similarly, how many people were involved with creating multiple-instance bounding boxes? The annotation section might benefit from a diagram to better explain this process. \n\nSecond, the baseline method is poorly described. It's not clear from the main paper or the semantic similarity how the different proposed components come together into the OFA-DOD model. The paper would greatly benefit from a figure displaying the OFA model and the OFA-DOD model to better understand the architecture of the OFA-DOD model. As it stands now, a reader would have a hard time recreating the OFA-DOD model from the given text.\n\nFinally, there are some minor typo issues that need to be addressed. For example, line 130 page 4 repeats “their” twice, and line 212 on page 8 has the wrong subject-verb agreement (i.e. “fails” instead of “fail”). The paper could use another proofread for typos and grammar.\n\nQUESTIONS:\nPlease see the weaknesses section on the questions related to the annotation process and the OFA-DOD model.\n\nLIMITATIONS:\nThe paper adequately addresses its limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose the definition, dataset, evaluation metrics and benchmark results of a new task - Described Object Detection (DOD). DOD is designed to detect objects aligned with full, presence or absence langauge descriptions. The proposed evaluation metrics have three groups based on full, presence or absence descriptions. The authors also propose the Intra-scenario mAP that only detect the categories appear in the image, and Inter-scenario mAP that detecting all the categories. The authors provide the benchmarks results on the proposed task with a new OFA-DOD baseline.\n\nSTRENGTHS:\n- The authors analysize the limitation of previous tasks\n- The authors provide comprehensive benchmark results on the proposed new settings\n\nWEAKNESSES:\nFirst, I want to clarify two different tasks: category-level object detection and visual grounding. The category-level tasks like OVD focus on recognizing pre-defined C categories, e.g., \"cat,\" \"dog,\" etc. These categories are commonly mutually exclusive (COCO) or share a parent-child relationship (LVIS). The OVD task will divide the C categories into base and novel two disjoint groups and the models are required to train on base classes and test on novel classes to verify the generalization ability. Instead of detecting C classes, the grounding tasks like REC, aim to align each region to a text phrase. The REC datasets like RefCOCO do not pre-defined a set of categories.\nBased on my personal understanding on the OVD and REC tasks, I think this paper has the following weaknesses:\n\n1. The authors argue that DOD is a superset of OVD (#35 and the right part of Figure 1). But I disagree with this argument. The OVD task focuses on evaluating the generalization ability. However, for the DOD task, the proposed dataset evaluation metrics do not consider dividing base/novel groups. Thus, OVD is suitable to evaluate the generalization ability, but DOD is not suitable. If the authors think that the DOD task does not focus on the generalization ability, they should not argue that \"superset of OVD\".\n  - 1.1 For the proposed D3 datasets for DOD, the authors pre-defined 422 categories. What happens to the other expressions that are not pre-defined? The authors may add some novel classes in the test set that are not available during training to simulate this situation. I think the base/novel setting is one advantage of OVD over DOD. If the proposed D3 dataset has already some novel expression in the test set, the authors may consider adding evaluation results on base/novel groups.\n  - 1.2 The left part of Figure 1 is the definition of object detection rather than open-vocabulary object detection with the base/novel setting.\n  - 1.3 From my perspective DOD is focused on fine-grained language understanding (e.g., the unrestricted language description in #123-127), which is not the main focus of OVD.\n\n2. #188-122: For the REC task, the model is required to align the region with the input language expression and the datasets of the REC task. In my opinion, I don't consider \"complete annotation\" to be a disadvantage of REC. For instance, REC datasets such as RefCOCO & RefCOCO+ consist of 141,564 language queries, which is significantly larger than the proposed D3 dataset with only 422 language queries. Can we conclude that the vocabulary size of REC datasets is larger than that of DOD? Can we argue that due to the cost of annotation, the \"complete annotation\" requirement restricts the vocabulary size?\n\nQUESTIONS:\n3. In the proposed DOD task, the category-level definition is vague, and let me feel confused. The authors pre-defined 422 expressions or categories and use these 422 categories for evaluating the Inter-scenario mAP. But are these categories mutually exclusive or overlapped? For example, I guess some short expressions (e.g., 'backpack') may be a parent node of other long expressions (e.g., 'backpack with yellow color'). The authors may consider this point when designing their evaluation metrics, like adopting the positive/negative labels in the OpenImage and LVIS datasets.\n- 3.1 Figure 3 (b) and (c) use the \"number of instances\", does it mean \"number of categories\"? \n\n4. The motivation for the new DOD task is not clear to me. Can the authors provide straightforward application scenarios or real-world examples that previous tasks are limited while the proposed new DOD task offers broader possibilities?\n\nLIMITATIONS:\nIn #341-345, the authors have discussed their limitations regarding potential abuse and carbon emissions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new Described Object Detection (DOD) task, which extends the existing Open-Vocabulary Object Detection (OVD) and Referring Expression Comprehension (REC) tasks into a more general paradigm. For this new task, the authors build a Description Detection Dataset (D3), and find the troublemakers that currently hinder current REC, OVD, and bi-functional methods. They further propose a baseline that outperforms existing methods on the DOD task.\n\nSTRENGTHS:\n1.\tFocusing on language-driven object detection, the authors propose a new DOD task, which they argue is more practical and presents more challenges than the current OVD and REC tasks. They also introduce a new dataset D3 for this task. \n2.\tOn the DOD task, the authors thoroughly investigate the challenges faced by current REC, OVD, and bi-functional approaches, and they put forth a baseline method with state-of-the-art performance on this task.\n\nWEAKNESSES:\n1.\tWhile the authors introduce this new DOD task, instead of constructing a new training dataset or setting, they focus on analyzing the performance of models trained on the old OVD or REC tasks for this new task. This may not fairly represent the performance of existing methods for the new task.\n2.\tAs the DOD task can be seen as a more generalized version of the OVD task, where category names are extended to language descriptions, the experimental results and conclusions for OVD methods highly depend on their training data and settings. It would be beneficial to assess OVD baselines trained following the style and setting of the DOD task. \n3.\tThe compatibility of the DOD task with the existing OVD and REC tasks is not studied. Given that the DOD task is essentially a superset of the OVD and REC tasks, it would be beneficial to evaluate the baseline OFA-DOD in comparison with other methods on the OVD and REC tasks. \n4.\tThe existing methods and the proposed baseline are evaluated and compared under a zero-shot setting. How would the performance vary when the models are fine-tuned on the D3 dataset?\n\nQUESTIONS:\nPlease refer to the weakness part.\n\nLIMITATIONS:\nThe authors have discussed the limitations and broader impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBrief Summary: The paper presents a new task Described Object Detection which extends open vocabulary object detection (OVD) to use phrases. This, in turn, extends referring expression (REC) to include objects not seen in the training data. To this end, the authors introduce a new dataset called D3 building on existing GRD dataset [44]. \n\nThe key idea behind creating D3 are to have complete annotation (i.e. each referring expression has a bounding box in each image if present), have natural language (extends OVD), include absence expression (i.e. objects NOT having a particular feature / attribute such as blackboard with no signs), and one expression referring to more than one instance. \n\nThe authors experiment on the provided D3 dataset and provide a detailed benchmark with multiple baselines ranging from REC methods like OFA, OVD methods like OWL-ViT and bifunctional method like Grounding-DINO. The authors further propose a new baseline OFA-DOD which changes some pre-training schemes such as including additional localization tasks and find that it outperforms competitive baselines.\n\nSTRENGTHS:\nPros:\n\n1. Dataset contribution is always welcome. It is clear the authors have put thought into the dataset construction. In particular, absence expression are quite interesting.\n\n2. A number of baselines are considered and the takeaways are quite interesting that REC methods fail at this task due to being unable to effectively choose more than one boxes. While OVD methods  perform better, there is a very large gap compared to bifunctional methods like grounding-dino. The proposed baseline OFA-DOD makes sense and it is good to see that it outperforms other baselines. \n\n3. Visualization of the dataset as well as results in suppl. are very useful. Ablative studies on the baseline such as effect of training data (Table 5b) are interesting.\n\nWEAKNESSES:\nCons:\n\n1. The authors should compare their work with zero-shot grounding [Ref1] which also extends REC to new objects. The obvious difference is that DOD can have more than one instances but a clear distinction would be helpful.\n\n2. The main idea behind DOD is to encompass both OVD and REC. The authors need to motivate this setting more. In my opinion, having the two cases separate can be much more revealing than trying to combine the two. OVD is strictly object detection (with phrases) while REC explicitly requires disambiguation between different objects. For instance, \"oversized glove on left-hand\" in suppl fig 2 (last row), is simply object detection of glove and doesn't require reasoning whether it is \"oversized\" or on \"left-hand\". To me a more natural setting is to separate the two. I would like to know about the author's motivation for the task.\n\n3. It is unclear in the text but in OFA-DOD which OFA is chosen, base or large? To have a fair comparison with Grounding-DINO it should be base but it is not clear (in main text as well as in suppl).\n\n4. In suppl. Table 2, it seems Grounding-DINO outperforms OFA-DOD by considerable margin on Average Recall. Why is this the case? For practical use, wouldn't one prefer using Grounding-DINO?\n\n\n[Ref1]: Sadhu, Arka, Kan Chen, and Ram Nevatia. \"Zero-shot grounding of objects from natural language queries.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4694-4703. 2019.\n\nQUESTIONS:\nSee weakness section.\n\nLIMITATIONS:\nYes, the authors have discussed some limitations but more could be included such as:\n\n1. The evaluation is heavily dependent on the choice of 412 phrases and the dataset used for GRD.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a task called described object detection which involves detecting objects through free form text queries, encompassing referring expression comprehension as well as open vocabulary detection.\n\nSTRENGTHS:\n## Clarity\nThe paper is written quite clearly.\n\n## Originality, Quality and Significance: Please see weaknesses\n\nWEAKNESSES:\n# Major issues: \n\n## Quality [Annotating using CLIP is insufficient]\n* DOD does not provide manually annotated explicit negative certificates for the images that are deemed as a negative for a given text query. The negatives are extracted using a CLIP matching score for each image with all the possible text queries. It has been demonstrated in several works that CLIP has no fine-grained understanding of the image [1,2,3], is largely incapable\nof performing spatial reasoning off-the-shelf, obtaining chance performance even on simple synthetic images [1] and also behaves as a bag of words when understanding the text [2]. This implies that any complex query that is more than just a category name, cannot be appropriately distinguished as being relevant or not for a given image, and using this as a step in the annotation pipeline is sure to introduce errors. Using any sort of image-text model during the annotation process inherits the biases of the underlying model and in my opinion is not a viable approach for constructing an evaluation benchmark. For a benchmark that is characterized as a detection dataset, having accurate negatives is paramount, and the authors have not demonstrated that this criteria can be met using a CLIP matching in the annotation process. \n* Further, testing the ability of models to distinguish the presence or absence of a textual query, and localize it, would be truly tested in the cases of having unlikely phrases or combinations of objects and attributes or unlikely relations. Using CLIP in the annotation process would completely fail in these cases, as it has been shown that CLIP has a strong Concept Association Bias [3], frequently giving the highest matching score to the most likely completion, without paying attention to the image. This would further exacerbate the difficulty in accurately evaluating models on examples that might be especially hard and interesting (especially relevant to the more challenging \"absence\" type of queries).\n\n## Originality, Significance [Proposed task is equivalent to existing benchmarks]\n* As far as I can tell, the proposed task does not differ from the Phrase Detection task proposed by [5] which addresses the problem of both identifying whether the phrase is relevant to an image and also localizing the phrase, across a whole dataset. Departing from referring expression comprehension, they allow prediction of multiple boxes per phrase, and different from object detection, they evaluate text prompts that are longer than simple category names. Missing this line of literature completely is quite a red flag given that it has been around for quite some years now. \n* Another benchmark, \"COPS-Ref\" has also been proposed [6] that focuses on referring expressions with varying degrees of complexity and in which the localization must be done across multiple images, also containing distractors. This work also does not acknowledge or differentiate from COPS-Ref. \n\n\n[1] ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension. Sanjay Subramania et al, 2022\n\n[2] When and why vision-language models behave like bags-of-words, and what to do about it? Mert Yuksekgonul et al, 2023. \n\n[3] When are Lemons Purple? The Concept Association Bias of CLIP. Yutaro Yamada et al, 2022\n\n[4] Revisiting Image-Language Networks for Open-ended Phrase Detection. Plummer et al, 2020\n\n[5] Cops-Ref: A new Dataset and Task on Compositional Referring Expression Comprehension. Zhenfang Chen, 2020\n\nQUESTIONS:\n## Suggestions \n* Line 103: \"Currently, OFA holds the SOTA among REC methods.\" I believe this comment is a quite outdated and can be updated (joint OVD & REC methods such as FIBER [1] outperform it). \n* For the rebuttal, the authors could explain the difference between the proposed task and phrase detection.\n\n[1]  Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone. Dou et al, 2022\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-11"}