Described Object Detection: Liberating Object Detection with Flexible Expressions
Authors: Chi Xie
Zhao Zhang
Yixuan Wu
Feng Zhu
Rui Zhao
Shuang Liang
Venue: NeurIPS 2023 poster (2023)
Keywords: open-vocabulary object detection
referring expression comprehension
multi-modal detection
TL;DR: A novel and realistic setting that generalizes OVD and REC tasks, a corresponding benchmark with complete annotation and free-form references, an in-depth analysis over existing methods, and an improved baseline following the analysis findings.

Abstract: Detecting objects based on language information is a popular task that includes Open-Vocabulary object Detection (OVD) and Referring Expression Comprehension (REC). In this paper, we advance them to a more practical setting called *Described Object Detection* (DOD) by expanding category names to flexible language expressions for OVD and overcoming the limitation of REC only grounding the pre-existing object. We establish the research foundation for DOD by constructing a *Description Detection Dataset* ($D^3$). This dataset features flexible language expressions, whether short category names or long descriptions, and annotating all described objects on all images without omission. By evaluating previous SOTA methods on $D^3$, we find some troublemakers that fail current REC, OVD, and bi-functional methods. REC methods struggle with confidence scores, rejecting negative instances, and multi-target scenarios, while OVD methods face constraints with long and complex descriptions. Recent bi-functional methods also do not work well on DOD due to their separated training procedures and inference strategies for REC and OVD tasks. Building upon the aforementioned findings, we propose a baseline that largely improves REC methods by reconstructing the training data and introducing a binary classification sub-task, outperforming existing methods. Data and code are available at https://github.com/shikras/d-cube and related works are tracked in https://github.com/Charles-Xie/awesome-described-object-detection.

Described Object Detection: Liberating Object Detection with Flexible Expressions
Chi Xielt Zhao Zhang2t
Yixuan Wu
Feng Zhu
Rui Zhao?
Shuang Liang"*
'Tongii University
Sensetime Research
*Zhejiang University
chixie@tongji. edu. en
zzhanggmail nankai. .dd. cn
shuangliang@tongji edu
Abstract
Detecting objects based language information is popular task that includes Open-Vocabulayy object Detection (OVD) and Referring Expression Comprehen- sion (REC). this paper, we advance them to a more practical setting called Described Object Detection (DOD) by expanding category names to flexible lan- guage expressions for OVD and overcoming the limitation of REC only grounding the pre-existing object. We establish the research foundation for DOD by construct- ing Description Detection Dataset (D²). This dataset features flexible language expressions, whether short category names or long descriptions, and annotating all described objects on all images without omission. By evaluating previous SOTA methods on D3 we find some troublemakers that fail current REC, OVD, and bi-functional methods. REC methods struggle with confidence scores, rejecting negative instances, and multi-target scenarios, while OVD methods face constraints with long and complex descriptions. Recent bi- bi-functional methods also not work well on DOD due to their separated training procedures and inference strategies for REC and OVD tasks. Building upon the aforementioned findings, we propose a baseline that largely improves REC methods by reconstructing the training data and introducing a binary classification sub-task, outperforming existing methods. Data and code available at this URL and related works are tracked in this repo
Introduction
Detecting objects of interest within a scene using language is a pivotal area of focus. This field encompasses two key tasks: Open-Vocabulary Vocabulary object Detection (OVD)
[0
20
and Referring Expression Comprehension (REC) [1 illustration of these two settings in Fig. [ The first task, OVD, expands the scope of object detection (OD) to any given short category name. However, these settings neglect the instances described by intricate descriptions. The second task, REC, focuses on spatially locating one target described by an expression and assumes the target must exist in the image. However, in real-world scenarios, the described objects do not exist in the image, REC algorithms output calse-positive results. Recent advancements have witnessed the joint training of bi-functional models, such Grounding- DINO 25 and UNINEXT [4), which involve both OVD and REC data. Notwithstanding, these models still rely separate training procedures and inference strategies for OVD and REC, and evaluate these two tasks independently.
27
24
46
52
We present an intuitive
As shown in Fig. category, whether long or short, complex or simple, while discarding predictions in images where targets are absent. In order to address this significant yet often overlooked scenario, we propose the concept of Described Object Detection (DOD). Note that this setting is a superset of OVD and REC.
more practical detection algorithm should be able to detect any described
Conresponding author. "Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

blue dragon
p- a lying on sofa
person
person right next to dog
individuals connected by ropes
OVD
REC
Described Object Detection (DOD)
dog
a dog on the left the image
dog not led by ropes outside
OVD
REC
DOD
Relationships
Figure 1: Examples showing the difference between REC, OVD and Described Object Detection (DOD). OVD detects arbitrary number (including zero, denoted with 0) of objects based on category name; REC grounds one region based on a language description, whether the object truly exists or not; DOD detect all instances each image in the dataset, based a flexible reference.
When the language expression is limited to a short category name, it becomes OVD. When we limit the images detect objects known be present in the images beforehand, downgrades to REC.
Can the existing SOTA algorithms of the community support DOD tasks? To address this inquiry, this paper establishes the research foundation of DOD tasks by constructing dataset, scrutinizing relevant methodologies, analyzing the relevant methods, and exploring improvement space.
Motivation & real-world application of DOD. OVD is limited to categorical detection, focusing on classes rather than specific attributes or relationships. lacks detailed contextual understanding and cannot adapt to precise detection requirements from language. REC comprehend longer descriptions for attributes or relationships, but assumes the existence of one target the image. This leads false positives when the target is absent, limiting its practical usability. Consider detecting individuals without helmets on a construction site using camera data: OVD can detect helmets and people but not determine their relationship. REC locate one region in any image and generate false positives frequently. Existing solutions involve using separate models for object detection then relationship classification, or REC after image classification, both resulting in inefficiency.
Hence, there is demand for language- object detection: model with strong generalization capabilities that can verify the existence of described objects in images and localize them based on arbitrary expressions. The proposed DOD task addresses this need and finds practical applications in: urban security, detecting dogs without leashes in communities, clothes hung outdoors on streets, overloaded vehicles, and fallen trees on roadsides; network security, like identifying sensitive images with violence bloodshed within large datasets; (fine-grained) photo album retrieval based on descriptions or keywords; retrieval and filtering of web image data; specific event detection in autonomous driving, such pedestrians crossing the road.
Dataset & benchmark. For DOD, we introduce the Description Detection Dataset (D*3, /dikju:b/) an evaluation-onll benchmark containing 422 descriptions and 24,282 positive object-description pairs. Unlike previous OVD or REC datasets (see Fig. 2). D3 stands out three key aspects (see Tab. [ 1) Complete annotation: All descriptions refer to objects annotated throughout the dataset, making D a detection-ste -style dataset akin COCO 23). 2) Unrestricted description: Annotations in D include diverse and flexible language expressions, varying in length and complexity. 3) Absence expression: We include descriptions regarding absence of concepts, such as a person wi thout safety helmet, addressing an often-overlooked detection requirement. The details of D is claborated in Sec. We evaluate sate--o--hhearr art methods on D OWL-ViT /29/CORA [42 (OVD), OFA (REC) [39, and UNINEXT [44/Grounding DINO (25) (bi- bbiffunctiona) to provide a reference for the community. This benchmark may serve as a starting point for the DOD task.
Findings & improvements. The experimental analysis for different methods on D yields some findings for future research (see Sec. Sec.55b 1) Existing REC methods perform poorly, lacking confidence scores and the ability to reject negatives, and struggling with multi- situations. This is due to their task formulation of grounding, i.e., matching between text and image region and not distinguishing positive and negatives. 2) OVD methods excel REC ones on DOD, though lengthy descriptions, which is not available in their training data, limit their performance. 3) Bi- Bi-functional methods, while superior to REC and OVD ones, share similar challenges with REC methods. Sometimes they

---

Table 1: Comparison between the proposed dataset and previous REC datasets and OVD datasets.
annotation completeness
unrestricted description
absence expression
instance-level
Dataset
annotation
RefCOCO
image-wise
X
COCO
dataset-wise
x
X
GRD
group-wise
X
Ours
dataset-wise
V
surpassed by OVD models, indicating they have not fully benefited from REC and OVD. Based on these findings, we propose a baseline OFA-DOD that greatly improves a REC method, and outperforms current SOTAs. Its abilities handle multiple targets and reject negative instances are improved by simple data reconstruction and an auxiliary sub-task. It is still far from a strong DOD method, but may provide some insights for research in the future.
2 Related Work
2.1 Relevant datasets and benchmarks
Object detection datasets. A variety of datasets have been proposed for object detection. Some have become standard benchmarks, like PASCAL VOC [9] and COCO [33;; while others are more frequently used for pretraining [35 16 2). A few works have focused on special settings, such as LVIS [2] for long-tailed detection and ODinW [9 for zero-shot evaluation in the wild. Recently, V3Det 38 facilitates object detection with an extremely large vocabulary. Some are re-splitted and frequently used in OVD as well, like COCO and LVIS. As explained Sec. [ these datasets are all annotated with simple category labels rather than flexible language expressions like D3
Referring expression comprehension datasets. Several datasets have been introduced evaluate REC methods, including ReiClef [15., RefCOCO [66,, RefCOCO+ [46), RefCOCOg (28) Visual Genome [17, and PhraseCut [40]. Some 15
collected interactively, and the expressions are more concise and less diverse. RefCOCOg is collected non-interactively, resulting in more complex expressions. Comparatively. Visual Genome focuses on visual relationships. All these datasets only annotate a few positive images for each category and leave other images unknown, which makes them unsuitable for the detection task.
Other related tasks and datasets. Several related tasks and benchmarks exist, but they differ significantly from DOD. Phrase Detection 31 lacks explicit negative labels negative instances are unlabeled, and does not constitute a true detection task. Additionally, its references are simply phrases. In contrast, DOD ensures exhaustive annotation of positive and negative labels, and its references can be words, phrases, or sentences. Cops-Ref benchmark [6] focuses on evaluating the grounding capability of REC methods difficult negative regions with related and distracting targets. It provides explicit negative certificates for only limited set of images. In D², negative certificates are available across the entire dataset. Zero-shot grounding [34] centers on locating concepts not in the training set. It assumes the existence of the object referred by a reference in a image, and locates a single target per image, with short phrase, while DOD makes no assumptions about the existence of the target, and locates zero to multiple targets, with varied expressions.
2.2 Current methods
Open-vocabulary object detection methods. Open-vocabulary detection is currently receiving increased attention. aims detect arbitrary classes using language for generalization, even when trained on a limited set of classes. The first approach, OVR-CNN [48), utilizes image-caption pairs for pretraining the visual encoder to enhance its zero-shot generalization capabilities. With the introduction of CLIP 33 models such as Detic [51]. DetCLIP 45), RegionCLIP [50], and OV. DETR [47 have further advanced image and language embeddings pretrained using CLIP. ViLD [I]] further distills knowledge from CLIP to inherit language semantics for recognizing novel classes.
GLIP 20 formulates object detection as phrase grounding problem [32] and utilizes additional phrase grounding data to facilitate vasion-language alignment.

---

REC
OVD X
DOD
dog
"bottle"
refer
with text on
the surfoce"
refer
refer
"wire glass"
Doo
DOD /
DOD V
Ta baseball player ready
"barkpack"
refe
refer
nefer
refer
refer
leather bag
"well cover on the surface"
"an
the
glass
of player"
surfcce
(a) Complete annotation.
(b) Unrestricted reference.
(c) Absence expression.
Figure 2: Some examples from previous datasets and the proposed D dataset for DOD, (a) Our dataset for DOD is completely annotated detection, while REC datasets like RefCOCO are not. (b) Our dataset has unrestricted reference, while OVD datasets like COCO not. (c) Our dataset not only provides traditional presence descriptions, but also absence descriptions.
Referring expression comprehension methods. Existing works 7 36
21
24
39
be divided
into three categories. (1) Specialist models. Previously, two-staged works [3]
46
reformulate this
ranking task. More recently, one-stage approaches [52 [7] speed up the inference process. (2)
Multi-task models [53 24) They usually design unified formulation for few closely related
tasks. For example, Seq TR (53 unifies REC and RES (Referring Expression Segmentation)
point
prediction problem. (3) Multi-modal pre-training models 26 39]. Unified-IO (26) and OFA [39]
propose unified sequence-to-scquenee -e-uuueence frameworks that can handle variety of vision, language, and multi-modal tasks. Currently, OFA holds the SOTA among REC methods.
aim to handle tasks such as OVD (or OD) and REC concurrently within single model. They typically restructure the training approach for these tasks, enabling single model to learn from datasets related both tasks. However, the inference process for each task remains distinct and independent the other. FIBER & employs two-stage pretraining strategy, separately utilizing image-text and image-text- box data to enhance data efficiency. More recently, Grounding-DINO (5 extends closed-set detector by performing vision-language fusion multiple stages and evaluating its performance on REC datasets. UNINEXT [44] reformulates various image and video tasks into a unified object discovery and retrieval paradigm. Despite these models sharing knowledge between detection and REC through pretraining, they are still treated as distinct tasks in these bi-functional models.
Bi-functional models for REC and OVD/OD. Some recent works [14
8
18 25
Methods with potential for DOD continuously emerging and we will update them in this list
3 Dataset
3.1
Dataset highlight
The proposed dataset is re-annotated on GRD (43, a dataset for RES 53
24 and its variants [41 22] show some examples from previous datasets and D to highlight them. Here we elaborate on them with a few other characteristics:
As briefly introduced in Sec. [ it contains three major characteristics. In Fig. 2
The first is complete annotation. For REC, the instances referred to by one description are only annotated in few images. For other images without the annotation of this description, it is unknown whether the corresponding instance exists or not. That is to say, their annotations are not complete. Contrarily, shown in Fig. 22 in D the objects referred to in all images by any description annotated, as are the negative samples, like traditional object detection datasets.
The second is unrestricted language description. As shown in Fig. 2b unlike (open vocabulary) object detection that retrieves objects with category names, we retrieve objects with language expressions, which is rather flexible. As is shown in Fig. 3d the lengths of descriptions in D vary a lot. The

---

80
100
1000
Count
60
Count
7503
Count
40
2500
150
200
200 Num. instances
10
Num, images
Num. of instancess Image
Langths descriptions
(a)
(b)
(c)
(d)
Figure 3: Distribution of (a) number of positive images for description in the dataset, (b) number of positive instances for a description, (c) number of instances in positive image for description, and (d) lengths of descriptions.
shortest descriptions have one or two words, where the DOD task downgrades to OVD, while the longest may have 15 or more words, resulting in rather complex language expressions.
The third is absence expression. Current datasets with language description, like RefCOCO series for REC, usually describe objects with certain features. They usually focus on the ability to discover the existence of concepts but neglect their absence. Noticing the missing ability to verify such capability, we also annotate objects lacking a certain attribute. Fig. [c shows an example with presence description and another with absence description from Dd Such absence description makes up about one quarter of the references in this dataset. This is a first for existing benchmarks.
The fourth is instance-level annotation, a characteristic not held by GRD as is intended for RES.
The fifth is description can refer to multiple instances in an image, as in Fig. 3c This is not true for REC datasets. If we regard category names as references, then OD datasets do have this feature.
In summary, the proposed dataset differs from the REC dataset primarily in terms of characteristics 1st, 3rd, and 5th. In contrast, when compared to OD datasets, the proposed dataset showcases disparities in the 2nd and 3rd characteristics, and when compared with GRD, in the 2nd, 3rd, 4th, and 5th characteristics. We refer the readers the supplementary materials for more information about the characteristics of 3 and more examples.
3.2 Annotation process
We utilize the GRD dataset 43
as the source for images, along with original annotations. Originally, it is divided into multiple groups, each containing several references, with positive and negative samples annotated only within each group. We extend the annotations in three aspects:
Adding instance-level annotation. GRD is designed for RES, where each reference corresponds to one semantic mask across one image. However, for the DOD task, which requires the recognition and localization of individual instances, we annotate each instance referred to by a description with an individual bounding box (along with an instance mask). This is the basic step to adapt the dataset for instance localization.
Adding complete annotations. In addition to the intra-group annotation in GRD, further annotate the positive and negative samples for each reference across the entire dataset. With complete dataset- wise annotations, the division into groups becomes unnecessary for evaluation, serving only means to organize references by scenarios. This enhancement makes the dataset suitable for detection tasks, significantly increasing the number of positive and negative samples.
Note that we use the complete annotation similar to COCO 23 i.c., explicit positive and negative certificates for all categories on all images, rather than federated annotation 11, [6]. This allows using mAP (mean Average Precision) as the evaluation metric, which elaborated in Sec. 3.4
Adding annotations for absence expressions. We have designed many absence descriptions based on the scenarios within the dataset, in addition to the traditional presence expressions GRD. We annotate the instances in the images across the entire dataset with these absence expressions. This step increases the difficulty level of the proposed benchmark and enables the evaluation of existing models' ability to comprehend the absence of concepts.
5

---

We present a concise overview of the overall annotation process here. We organize groups of images and references (both for presence and absence). For each image, the references in its group are used. References from other groups may also appear, but with lower probability. We employ CLIIP [33 to select a large number of candidates from these references in other groups. We manually check and adjust the nyper-parameterr to make sure that such CLIP filtering usually do not miss positíve refs. Subsequenlly, annotators select the positive references from these candidates (rather than from all references in the dataset) and add bounding boxes to the images. For more detailed information regarding the annotation process, please refer to supplementary materials.
3.3 Dataset statisties
GRD statisties. It has 10,578 images collected online, divided into 106 groups. Each group has around 100 images and 3 expressions referring to segmentation masks in this group, resulting in 316 references. 9,323 positive image-text pairs and 22,201 negative pairs. Note that only annotates positive and negative samples inside each group. i.c., the annotation completeness is only group-level, so a reference will not be verified outside its group. The expressions have an average length of 5.9 words. We refer the reader to the original paper for specific statistics of GRD.
D statistics. The proposed D has 10,578 images, all from GRD. It has 422 well-designed expres- sions, including 316 expressions from GRD and 106 absence expressions we added (one for each scenario). The instance-level annotation results in 18,514 boxes.
Due to the effort in complete annotation, for a reference, each image in the dataset annotated for possible positive and negative samples, i.e., the annotation completeness is dataset-level. Thus, there are 24,282 positive object-text pairs and 7.788,626 negative pairs, orders of magnitude larger than GRD. Among them, those with images and texts from the same scenario are probably more difficult, which includes 20,279 positive and 53,383 negative pairs. The average length of expressions is 6.3 words, due the relative longer absence expressions. More statistics and examples of D3 are available in supplementary materials.
3.4 Evaluation metries
The classification of instances in D is multi-label. Each description corresponds to category. Naturally, there can be relationships between categories, such as parent-child hierarchies, synonyms, and partial overlap. When designing categories, we intentionally reduce parent-child or synonym relationships to ensure greater diversity and challenge. However, there exists partial overlap between categories. Therefore, in Dd, one instance may correspond to multiple descriptions, and the classifi- cation in Dd is multi-label [12 rather than single-label [23], making it suitable for categories with relationships. An effective detector should assign all relevant positive categories (e.g., dog not led by rope outside and clothed dog for clothed dog not led by rope outside) for instance.
We use standard mAP for evaluation. Given the multi-label setting and the exhaustive annotation (all positive and negative labels are known for an instance) of D category relationships will not affect the evaluation, so we can use consistent evaluation for each category across all images. We describe the evaluation process here. For inference, an instance predicted with category A and B is regarded as an instance for category A and an instance for B. The AP for each category computed as follows: Predictions for each category across all images are sorted by score in descending order, and those with ground truth loU exceeding a threshold are counted as TP (and the ground truth marked as taken), while the rest are counted as false positives. With these TP and FP instances, we calculate the precision, recall, and AP. The mAP is calculated by averaging the AP across all categories.
We use FULL, PRES, and ABS to denote evaluation on all descriptions, presence descriptions only. and absence descriptions only. If not noted explicitly, the FULL setting is adopted. The specific metrics for Dd include: Intra-scenario mAP: For this metric, we perform evaluation on each image with only the descriptions from the image's 's scenario. The final metric is the mAP averaged on different loU thresholds from 0.5 to 0.95, following COCO (23). This is used as the default metric in our experimental settings. Inter-scenurio mAP: It is similar to the intra-scenario mAP described above, except that for each image, we detect the possible instances with all 422 references. This is aligned with the common mAP in object detection datasets the intra-scenario mAP.
23 and is much more challenging than

---

4 Baselines
4.1 Existing baselines from different tasks
We choose multiple advanced methods verify on D from OVD, REC to bi-functional methods. More details of these methods and their inference process are in our supplementary materials.
REC methods. We employ the state-of-the-art REC method, OFA (39 with two variants. OFA is based on an encoder- decoder,
to-eeuueee framework. It is multi-modal multi-task generalist that deals with different tasks together and is trained on various tasks, including language tasks (masked language modeling). image-to-text tasks (image captioning and Visual Question Answering (VQA)), and localization tasks (REC). Notably, although is trained with a detection dataset, it is not evaluated on object detection and achieves poor performance if we do. Currently, it holds the SOTA performance on standard REC benchmarks like the RefCOCO series.
OVD methods. We evaluate OWL-ViT [29] with two variants and CORA [42]. They are the SOTA methods on OVD tasks, with vision transformer as well a language transformer. They are pretrained with image-text contrastive learning and then fine-tuned on detection dataset.
Bi-functional methods utilizing both REC and OVD data. Methods falling into this category are not many but emerging fast recently. We test two methods: Grounding -INO [25] and UNINEXT [44], each with two variants. Both of them are based on DETR 33. They are pretrained multiple datasets, including detection and REC datasets, and then evaluated with different strategies for different tasks.
4.2 A proposed baseline
Dd is very challenging for existing works, as we will demonstrate in Sec. 5.1 We have selected one of these works for adjustment to provide better baseline. The chosen work should (1) be capable of understanding text of various lengths; (2) excel in their original tasks; (3) have framework with rather simple technical design, allowing us to modify its components easily. We have chosen OFA because it (1) is multi-modal multi-task framework with MLM (Masked Language Modeling) and image-to-texs text pretraining: (2) achieves SOTA on REC; (3) has a simple seq2seq framework.
However, OFA faces several problems that make it unsatisfactory for this task, as discussed in Sec. 5 First, forcing multiple tasks of different modalities into one seq2seq framework adversely affects the performance of specific tasks, especially tasks related to localization. Second, training on the grounding task results in poor ability to handle multiple instances. We evaluated the model on COCO detection, and it achieved less than 10 mAP. Thirdly, its REC paradigm also makes it predict only one instance, making unable to reject negative images and irrelevant descriptions.
Therefore, we have made some modifications to OFA to make it more suitable for this task. The first modification is granularity decomposition make it more suitable for localization. We have divided the pretraining tasks of OFA into two different granularities; global tasks (related to language modeling, such as captioning. VQA, MLM, etc.) and local tasks (related localization, such as detection and REC). We have added an additional decoder parallel to the original decoder in OFA that handles the local tasks, while the original decoder focuses on the global tasks. This alleviates conflicts between different tasks and enhances localization.
The second modification is reconstructed data for pretraining REC, aiming to improve multi- target localization. We have reconstructed the data for REC to ensure that (1) multiple references input for an image, and (2) a reference does or multiple. This results in a unified data format for detection and REC, although the labels may be noisy since they were not initially prepared for DOD.
necessarily correspond one object, but zero
The third modification is task decomposition to empower the model with the ability to reject false positives. We have reformulated the training reconstructed data into two tasks: RBC (for locating a region based on a reference) and VQA (for determining if a region and a reference match each other, essentially binary classification). The second step is responsible for rejecting false positives.
We refer to the model with all three modifications as OFA-DOD. More details on the proposed improvements can be found in the supplementary materials. It is important to note that this model is far from perfect for the complex 3 benchmark. As we will show in See. 5.1 although outperforms existing methods, it serves baseline for future tasks on D3.
7

---

Table Comparison of different methods on the proposed dataset for different mAP metrics.
Intra-scenario
Inter-scenario PRES
Task
Method
FULL
PRES
ABS
FULL
ABS
DaAse
3.4
3.0
4.3
0.1
0.1
0.1
REC
OFAlagee
4.2
4.1
4.6
0.1
0.1
0.1
CORAR5O
6.2
6.7
5.0
2.0
2.2
1.3
OVD
OWL-ViTTbse
8.6
8.5
8.8
3.2
3.7
4.7
OWL-ViT large
9.6
10.7
6.4
2.5
2.9
2.1
UNINEXT large
17.9
18.6
15.9
2.9
3.1
2.5
UNINEXTI huge
20.0
20.6
18.1
3.3
3,9
1.6
Bi-functional
G-DINO tiny
19.2
18.5
21.2
2.3
2.5
2.1
G-DINO busee
20.7
20.1
22.5
2.7
2.4
3.5
DOD
OFA-DODhes
21.6
23.7
15.4
5.7
6.9
2.3
Scores
TP Scores
1.00
320
208+
TTP
Count
200-
3020
520
23000
200
FP Scorre
Scores
FP Scones
Scores
(a) OFA
(b) ViT
(c) Grounding-DDin DINO
(d) OFA-DOD
Figure 4: Distribution of TP and FP scores from different baseline methods.
Experimental Analyses
5.1 Comparison of baselines on our metrics
We make comparisons on the baselines introduced in Sec. mainly with the intra-scenario setting. Unless explicitly noted, this is the default setting, instead of the more difficult inter-scenario.
Existing SOTAs are insufficient for DOD, and bi-functional models outperform others. As demonstrated Tab. 2 existing methods, while achieving SOTA performance on their original benchmarks, fall short delivering strong performance on D Among them, recent bi- bi-functional methods [25] 44] are notably superior to others, and currently, OVD methods outperform REC, The inferiority of REC methods is likely due their impractical setting described in Sec. T which involves predicting one and only one instance for each reference. We will delve into this further.
Rejecting irrelevant references are difficult, which REC are naturally incapable of. In contrast aitra-ccenario evaluation, the inter-scenario setting assesses all references in the dataset for each image. Since references from other scenarios are likely not semantically relevant to the images, this necessitates the ability to reject irrelevant references for an image. This aligns with the evaluation in standard detection tasks. From Tab. [ it is evident that OFA, REC method, almost completely fails in this setting. This is caused by its prediction large number of false positives when there numerous candidate references. This underscores the importance of empowering REC methods with the ability to reject false positives. We find that none of the verified methods achieve good performance under the inter-scenario setting, indicating that existing methods are far from being capable of DOD. This highlights the challenge of 3
region for every reference, resulting in
The proposed baseline outperforms existing methods. The proposed baseline is based OFA, but our improvements significantly enhance performance. It outperforms all existing methods intra-scenario setting and surpasses them by wider margin in the inter-scenario setting. This may suggest that proposed baseline has a stronger ability to reject irrelevant references. Nonetheless, the proposed method is far from perfect and can only serve bascline for future research.

---

Table Evaluation regarding different number of instances in a image for each reference.
Method
No-instance
One-instance
Multi-instance mAP %
FPPC (%) +
mAP (%) t
2
4
4+
OFA
100.0
14.8
9.5
7.9
5.4
3.7
CORA
17.3
9.7
8.4
9.5
9.0
8.5
OWL-ViT
41.9
21.1
17.3
16.6
16.0
14.0
UNINEXT
100.0
55.7
26.2
18.6
14.4
9.0
G-DINO
100.0
63.7
28.3
19.7
15.9
10.1
OFA-DOD
35.6
56.4
19.6
12.7
10.3
7.1
Table 4: Evaluation one references with various lengths.
Method
shorl
middle
long
very long
OFA
4.9
5.4
3.0
2.1
OWL-VIT
20.7
9.4
6.0
5.3
UNINEXT
18.5
23.3
17.4
16.1
G-DINO
22.6
22.5
18.9
16.5
OFA-DOD
23.6
22.6
20.5
18.4
5.2 Further analysis
Absence descriptions are more difficult for most methods. As shown in Tab. 2 the performance of baseline methods on PRES (presence descriptions) is consistently superior to that ABS (absence descriptions). This suggests that existing methods may not effectively differentiate between the presence and absence of attributes in language description.
REC methods fail to provide good confidence scores. We visualized the score distributions from baselines for TPs and FPs, to assess their capabilities in classification and confidence estimation. As in Fig. the confidence scores from OFA do not exhibit a clear distinction between TP and FP cases. This can be attributed in part to the seq2seq framework in OFA, which does not directly yield confidence scores, and in part to the grounding formulation of REC, which identifies the image region most similar the text description without distinguishing between positive and negative.
With a task decomposition step to enhance binary classification performance, our OFA-DOD demon- strates a sigmificant disparity between TP and FP score distributions, yielding more reliable classifica- tion results. Note that this improvement does not necessitate modifications to the model framework or training datasets; rather, it is attributed to more appropriate task formulation.
Multi-instance detection is challenging for methods other than OVD. For each image, D can have zero to multiple instances for a single description. To assess how current methods handle varying numbers of instances, we conducted evaluations under three different settings: no-instance, where for a reference, evaluations are limited to images without any referred instance; one-instance, for images with single instance; and multi-instance, for images with multiple instances. As shown in Tab, 3 OVD methods outperform others when multiple instances are referred by the description, although they may not be competitive on the entire dataset or images with few instances. Notably, OWL-Vit maintains consistent performance even the number of instances increases, which sets it apart from other methods. In contrast, REC and current fi-functional methods struggle in multi-instance scenarios. This highlights the strength of OVD methods in multi-target detection, while REC and current bi- fi-fucctional approaches are less robust such situations.
REC and bi-functional methods lack the ability to reject negative instances. In the no-instance column of Tab. we do report mAP since there no positive instances in GT for the correspond- ing reference, making AP inapplicable. Predictions on such images are FPs, so we measure the ratio of images where FPs are produced the total number of no-instance images for given reference. namely False Positives Per Category (FPPC). We report the average FPPC over all references. We observe that most baselines are incapable of determining whether an image contains the referred target or not, yet they still produce predictions. This behavior is expected for REC methods. Bi-functional methods, trained and inferred with the REC task formulation, also exhibit this issue. Only the OVD method and proposed bascline effectively reject such negative image-text pairs.

---

Table Ablation on the proposed baseline for its improvement components and the training data.
(a) Method components.
(b) Training data.
OFA
GD
RD
TD
mAP(%)
REC
OD
12T
MLM
mAP(%)
X
X
3.4
V
V
/
21.6
/
x
10.5
X
/
/
16.4
/
J
X
17.2
/
X
V
14.2
/
21.6
/
J
X
20.3
OVD methods suffer from long descriptions greatly while others do not. We partition the references according to their lengths and then evaluate on these partitions. The results are shown in Tab. 1 where short, middle, long and very long corresponding to references with 1-3, 4-6, 7-9, and more than 9 words. For short descriptions, which is close to OVD setting, OVD and bi- bi-functional methods obtain similar performance. However, as the length of references increases, the performance of OVD methods decrease fast, while REC and fi-functional methods suffer less from this. We can see that OVD methods are sensitive to long references, as expected, while other two types do not.
More experiments and additional qualitative results are available in supplementary materials.
5.3 Ablation on the proposed baseline
Method components. In Tab. we perform ablation on the proposed improvements our baseline, step-by-step from OFA to OFA-DOD, to see how they affect the performance. Granularity decom- position (GD) makes the method more suitable for localization task. It disentangle tasks of global or local granularity by handling them with 2 separated branch. Reconstructed data (RD) uniforms REC and OD data into the same form, and prepares multi-instance samples with both short and long references. Task decomposition (TD) is proposed to help rejecting FPs. It breaks down the DOD task into a REC step followed by VQA step. All three of them improve the performance obviously.
Training tasks. We also perform a drop-one-out ablation on the multi-modal multi-task training data, in Tab. 5b) Detection data provides samples for localization, especially multi-instance situation. is instinctively important for learning to localize, and indeed matters for performance. 12T (image-to- text, like image captioning and visual question answering) often helps the generalization and zero-shot performance of multi-modal methods. We find that it does affect the zero-shot performance on 3 greatly. MLM is theoretically important for language understanding and generalization. However, we find it actually is not. Removing the MLM task has no significant effect on the performance. We surmise that the generalization ability of OFA-DOD on D mainly comes from 12T.
Conclusion and Limitation
In this paper, we bring the Described Object Detection (DOD) task to the foreground. For this task, we introduce a dataset called D3, which annotates described objects without omission and features flexible language expressions, whether long or short, complex simple. Our evaluation of SOTA methods from REC or OVD on D reveals challenges faced by REC, OVD, and bi-functional approaches. Based on these observations, we propose baseline that largely improves REC methods for DOD task. We believe that the dataset and findings will contribute to advancing the understanding and development of DOD methods, facilitating future research in this area.
Limitation and broader impact. This work does have some limitations. Due to the significant unable to propose huge dataset
annotation cost brought by complete annotation process, with millions or billions of images. Besides, the evaluation and findings in this work may be dependent starting point for
on the choice of descriptions and the image sources. This work only serves DOD and we hope there will be other DOD datasets with larger scales. In the broader community, compared traditional detection algorithms, DOD models have lower customization threshold, enabling users to specify the detection target using language. This may lead to potential abuse.
Future work. During peer-review process, some new works with potential for DOD emerges, including Shikra 44. Kosmos-2 (30) and Qwen-VI. [T. We will continue to investigate such methods for DOD and update them in this list.
10

---

Acknowledgments. This work was supported in part by the National Natural Science Foundation of China under Grant 62076183, 61936014 and 61976159. in part by the Natural Science Foundation of Shanghai under Grant 207R 1473500. in part by the Shanghai Science and Technology Innovation Action Project under Grant 2051 100700 and 22511 105300, in part by the Shanghai Municipal Science and Technology Major Project under Grant 2021 SHZDD200000 and in part by Hetao Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone (HZQB-KCZYZ) 2021045). The authors would also like to thank the anonymous reviewers for their careful work and valuable suggestions.
References
[1] Bai, S. Bai, Yang. S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: fronticr large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
[2] L. Cai, Zhang, Y. Zhu, L. Zhang, M. Li, X. Xue. Bigdetection: A large-scale benchmark for improved object detector pre-training- Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pages 4777-4787, 2022.
[3] N. Carion, Massa, G. Synnaeve, N. Usunier, A. Kirillov, and Zagoruyko. End-to-end object detection with transformers. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pages 2 213-229. Springer, 2020.
[4] K. Chen, Z. Zhang, W. Zeng, R. Zhang, Zhu, and Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023.
[5] Y.-C. Chen, L. Li. L. Yu, A. EI Kholy, F. Ahmed, Z. Gan, Cheng, and J. Liu. Uniter: Universal image-text representation learning, Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XXX, pages 104-120. Springer, 2020.
[6] 7. Chen, Wang, Ma, K.-Y. K. Wong, and Q. Wu. Cops-ref: A new dataset and task on compositional referring expression comprehension. Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (0086-10095, 2020.
[7] J. Deng, Z. Yang. T. Chen, W. Zhou, and H. Li. Transvg: End-to-end visual grounding with transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1769-1779, 2021,
[8] Z.-Y. Dou, A. Kamath, Z. Gan, P. Zhang, J. Wang, L. Li, Z. Liu, C. Liu, Y. LeCun, N. Peng, et al. Coarse-to-ine vision-language pre-training with fusion in the backbone. Advances in neural information processing systems, 35:32942-32992 2022.
[9] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 2010.
[10] G. Ghiasi, X. Gu. Y. Cui, and T.-Y. Lin. Scaling open- vocabulayyy image segmentation with image-level labels. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI, pages 540-557. Springer, 2022.
[11] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference Learning Representations, 2022.
[12] A. Gupta, P. Dollar, and R. Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 5356-5364, 2019.
[13] R. Hu, M. Rohrbach, J. Andreas, T. Darrell and K. Saenko. Modeling relationships in referential expressions with compositional modular networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11 1115-1124, 2017.
[14] A. Kamath, M. Singh, Y. LoCun, G. Synnaeve, I. Misra, and N. Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
[15] S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, pages 787-798, 2014.
[16] I. Krasin, T. Lin, T. Duerig, P. Krähenbühl, A. Gupta, C. Burgess, and V. Ferrari. OpenImages: public dataset for scale multi-label and multi- image classification. Dataset available from http:://gituub.comoopnniiet 2017.

---

[17] R. Krishna, Y. Zhu. O. Groth. J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, al. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. IJC,, 123:32-73, 2017.
[18] W. Kuo, F. Bertsch, W. Li, A. Piergiovanni. M. Saffar, and A. Angelova. Findit: Generalized localization with natural language queries. In European Conference on Computer Vision. Springer, 2022.
[19] C. Li, H. Liu, L. Li, P. Zhang. J. Aneja, J. Yang. P. Jin, H. Hu, Z. Liu, Y. J. Lec, et al. Elevater: A benchmark and toolkit for evaluating language- aaggugee visual models. Advances in Neural Information Processing Systems, 35:9287-9301, 2022.
[20] L. H. Li, P. Zhang. H. Zhang. J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J.-N. Hwang, et al. Grounded language-image pre-training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965-10975, 2022.
[21] Li and L. Sigal. Referring transformer: one-step approach to multi-task visual grounding. Advances in neural information processing systems, 34: 19662-19664, 2021.
[22] X. Li, J. Wang, X. Xu, Li, B. Raj, and Y. Lu. Robust referring video object segmentation with cyclic structural consensus. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22236-22245, 2023.
[23] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, Microsoft COCO: Common objects in context. In ECCV, pages 740-755. Springer, 2014.
[24] Liu, H. Ding, Cai, Y. Zhang, R. Satzoda, V. Mahadevan, and R. Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. In CVPR, 2023.
[25] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang. J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:230.005499 2023.
[26] J. Lu, C. Clark, R. Zellers, R. Mottaghi, and A. Kembhavi. UNIFIED-IO: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations, 2023.
[27] G. Luo, Y. Zhou, X. Sun, L.. Cao, C. Wu, C. Deng, and R. Ji. Multi-task collaborative network for joint referring expression comprehension and segmentation. In Proceedings of the IEEE/CVF Conference on computer vision and patlern recognition, pages 10034-10043, 2020.
[28] J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 11-20. 2016.
[29] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenbomn A. Dosovitskiy, A. Mahendran, A. Arnb,, M. Dehghani, Shen, al. Simple vecabulayy object detection. European Conference on Computer Vision, pages 728-755. Springer, 2022.
[30] Z. Peng. W. Wang. L. Dong, Y. Hao, S. Huang, S. Ma, and F. Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.1 14824, 2023.
[31] B. A. Plummer. K. J. Shih, Y. Li. K. Xu. S. Lazebnik. S. Sclaroff, and K. Saenko. Revisiting image- language networks for open-ended phrase detection. IEEE transactions on pattern analysis and machine intelligence, 44(4):2155-2166 2020.
[32] B. A. Plummer, Wang. C. M. Cervantes, C. Caicedo, J. Hockenmaier, and S. Lazebnik. Flickr30k entities: Collecting region- to -o-phaase correspondences for richer image- to-senencce models. In Proceedings of the IEEE international conference on computer vision, pages 261 1-2649, 20155
[33] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[34] A. Sadhu, K. Chen, and R. Nevatia. Zero-shot grounding of objects from natural language queries. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4694 4703, 2019.
[35] S. Shao, Z. Li, T. Zhang, C. Peng, G. Yu, X. Zhang, J. Li, and J. Sun. Objects365: A large-scale, high- quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8430 8439, 2019.
12

---

[36] S. Song, X. Lin, J. Liu, Z. Guo, and S.-F. Chang. Co-grounding networks with semantic attention for referring expression comprehension in videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1346-1355, 2021.
[37] S. Subramanian, W. Merrill, T. Darrell, M. Gardner, S. Singh, and A. Rohrbach. Reclip: A strong zero-shot baseline for referring expression comprehension. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland, May 2022, Association for Computational Linguistics.
[38] Wang, Zhang, T. Chu, Y. Cao, Y. Zhou, T. Wu, Wang, C. He, and D. Lin. V3det: Vast vocabulary visual detection dataset. In The IEEE International Conference on Computer Vision (ICCV), October 2023.
[39] P. Wang. Yang, R. Men, J. Lin, Bai, Z. Li, Ma, C. Zhou, J. Zhou, and H. Yang. Ofa: Unifying architectures, tasks, and modalities through simple scquence-t--sqqueee International Conference on Machine Learning, pages 2331 23318-23340. PMLR, 2022.
learning framework. In
[40] C. Wu, Lin, Cohen, T. Bui, and Maji. Phrasecut: Language- based image segmentation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 0016-102255 2020.
[41] Wu, x. Li, Ii H. Ding, Tong, and Tao. Towards robust referring image segmentation. arXiv preprint Xxxi:2200.005... 2022.
[42] X. Wu, F. Zhu, Zhao, and Li. Cora: Adapting clip open-vocabulary detection with region prompting and anchor pre-matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7031-7048 2023.
[43] Y. Wu, Z. Zhang, C. Xie, F. Zhu, and R. Zhao. Advancing referring expression segmentation beyond single image. In International Conference on Computer Vision (ICCV), 2023.
[44] B. Yan, Y. Jiang, Wu, D. Wang, P. Luo, Z. Yuan, and Lu. Universal instance perception as object discovery and retrieval. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 15325-15336, 2023.
[45] L. Yao, J. Han, Y. Wen, X. Liang. D. Xu, W. Zhang, Z. Li, C. Xu, and H. Xu. Detclip: Dictionary- enriched cisuel- paralleled pre-training for open-world detection. Advances in Neural Information Processing Systems, 35:9125-91 9933,, 2022.
[46] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L.. Berg. Modeling context in referring expressions. In Computer Vision-ECCY 2016: 14th European Conference, Amsterdam, The Netherlands, October I 11-14, 2016, Proceedings, Part 14, pages 69-85. Springer, 2016.
[47] Y. Zang, W. Li, K. Zhou, C. Huang, and C. C. Loy. Opea-vocabulary detr with conditional matching. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pages (06-122 Springer, 2022.
[48] A. Zarcian, K. D. Rosa, D. H. Hu, and S.-F. Chang. Open-vocabulary object detection using captions, In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14393- 14402, 2021.
[49] H. Zhang, P. Zhang, X. Hu, Y.-C. Chen, L. Li, X. Dai, L. Wang, L. Yuan, J.-N. Hwang, and J. Gao. Glipv2: Unifying localization and vision- language understanding. Advances Neural Information Processing Systems, 35:36067-36080, 2022.
[50] Y. Zhong, J. Yang, P. Zhang, C. Li, N. Codella, L. H. Li, L. Zhou, X. Dai, L. Yuan, Y. Li, et al. Regionclip: Region-basee language-image pretraining. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793-16803, 2022.
[51] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and Misra. Detecting twenty-thossan classes using image-level supervision. In Computer Vision ECC-n-ECCV22 2022: 17th European Conference, Tel Aviv, Israel, October 23-27 27, 2022, Proceedings, Part IX, pages 350-368, Springer, 2022.
[52] Y. 7hou, Ji, G. Luo, X. Sun, Su, X. Ding, C.-W. Lin, and Q. Tian. real-time global inference network for onc-stage referring expression comprehension. IEFF Transactions on Neural Networks and Learning Systems, 2021.
[53] C. Zhu, Y. Zhou, Y. Shen, G. Luo, X. Pan, M. Lin, C. Chen, L. Cao, X. Sun, and R. Ji. Seqtr: A simple yet universal network for visual grounding. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviy Israel, October 23-27, 2022, Proceedings, Part XXXV, pages 598 615. Springer, 2022.
13


Reviews:
[{"review_id": "HDn9ab3fNo", "reviewer": "Reviewer_Q5JG", "summary": "The paper proposes a task called described object detection which involves detecting objects through free form text queries, encompassing referring expression comprehension as well as open vocabulary detection.", "questions": "## Suggestions \n* Line 103: \"Currently, OFA holds the SOTA among REC methods.\" I believe this comment is a quite outdated and can be updated (joint OVD & REC methods such as FIBER [1] outperform it). \n* For the rebuttal, the authors could explain the difference between the proposed task and phrase detection.\n\n[1]  Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone. Dou et al, 2022", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "## Clarity\nThe paper is written quite clearly.\n\n## Originality, Quality and Significance: Please see weaknesses", "weaknesses": "# Major issues: \n\n## Quality [Annotating using CLIP is insufficient]\n* DOD does not provide manually annotated explicit negative certificates for the images that are deemed as a negative for a given text query. The negatives are extracted using a CLIP matching score for each image with all the possible text queries. It has been demonstrated in several works that CLIP has no fine-grained understanding of the image [1,2,3], is largely incapable\nof performing spatial reasoning off-the-shelf, obtaining chance performance even on simple synthetic images [1] and also behaves as a bag of words when understanding the text [2]. This implies that any complex query that is more than just a category name, cannot be appropriately distinguished as being relevant or not for a given image, and using this as a step in the annotation pipeline is sure to introduce errors. Using any sort of image-text model during the annotation process inherits the biases of the underlying model and in my opinion is not a viable approach for constructing an evaluation benchmark. For a benchmark that is characterized as a detection dataset, having accurate negatives is paramount, and the authors have not demonstrated that this criteria can be met using a CLIP matching in the annotation process. \n* Further, testing the ability of models to distinguish the presence or absence of a textual query, and localize it, would be truly tested in the cases of having unlikely phrases or combinations of objects and attributes or unlikely relations. Using CLIP in the annotation process would completely fail in these cases, as it has been shown that CLIP has a strong Concept Association Bias [3], frequently giving the highest matching score to the most likely completion, without paying attention to the image. This would further exacerbate the difficulty in accurately evaluating models on examples that might be especially hard and interesting (especially relevant to the more challenging \"absence\" type of queries).\n\n## Originality, Significance [Proposed task is equivalent to existing benchmarks]\n* As far as I can tell, the proposed task does not differ from the Phrase Detection task proposed by [5] which addresses the problem of both identifying whether the phrase is relevant to an image and also localizing the phrase, across a whole dataset. Departing from referring expression comprehension, they allow prediction of multiple boxes per phrase, and different from object detection, they evaluate text prompts that are longer than simple category names. Missing this line of literature completely is quite a red flag given that it has been around for quite some years now. \n* Another benchmark, \"COPS-Ref\" has also been proposed [6] that focuses on referring expressions with varying degrees of complexity and in which the localization must be done across multiple images, also containing distractors. This work also does not acknowledge or differentiate from COPS-Ref. \n\n\n[1] ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension. Sanjay Subramania et al, 2022\n\n[2] When and why vision-language models behave like bags-of-words, and what to do about it? Mert Yuksekgonul et al, 2023. \n\n[3] When are Lemons Purple? The Concept Association Bias of CLIP. Yutaro Yamada et al, 2022\n\n[4] Revisiting Image-Language Networks for Open-ended Phrase Detection. Plummer et al, 2020\n\n[5] Cops-Ref: A new Dataset and Task on Compositional Referring Expression Comprehension. Zhenfang Chen, 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. **Due to the page limit, please refer to the *[general response (author rebuttal)](https://openreview.net/forum?id=0hwq2vOHT4&noteId=EtVOyLQxeQ)* for the description of annotation process of D3 dataset.**\n\n### 1. Annotation using CLIP is insufficient. DOD does not provide manually annotated explicit negative certificates for the images deemed as negative for a text query.\n\nThanks for the question.\n\nWe want to clarify that we do provide **manually annotated negative certificates**. We apologize for not describing the annotation process detailedly in the paper, and have added a diagram to illustrate this process, together with detailed explanation, in the ***[general response (author rebuttal)](https://openreview.net/forum?id=0hwq2vOHT4&noteId=EtVOyLQxeQ)*** above. We hope the reviewer will look into this.\n\nActually, we provide **negative certificates for all categories except positive categories** on an image. We do not take the federated annotation manner in large-scale, many-classes detection dataset, which only labels partial negative categories for an image due to large annotation cost of negative classes.\n\nIn our annotation process, we composite the following behaviors to ensure the negative labels are accurate and not missed:\n\n1. For the **data source**, the images are divided into different scenarios (groups), and the refs from different group are manually designed to have a small overlapping with each other (i.e., the refs of one group are not likely (but possible) to appear in images from another group).\n2. For image A1 from group A, we select (1) all the refs from group A, which are likely positive, and (2) partial refs from groups except A, proposed by by top-(n) according to the matching score between the image and refs by **CLIP**. To avoid this operation to filtering out some positive refs that should be kept, we use a rather large value of n (initially 40).\n3. The **annotators** select 5% images to check that the selected candidate refs covers all positive refs and the filtered out refs are negative. If there is a positive refs filtered out, the value of n is increased to cover that ref, and go back to step 2. After this check, the selected refs includes both positive and negative and the refs filtered out are negative only.\n4. The **annotators** annotate each image by selecting positive refs from candidate refs and adding boxes. They check that other candidate refs are negative refs.\n5. The **annotators** check all the images for 2 rounds, the first on all images and the second on 5% images.\n\nWith the 5 conditions above, we make sure that for each image, the refs not labeled as positive will be given a manual negative certificate. Such exhaustive and complete annotation is possible by limiting the scale to be 10000+ only, as an evaluation benchmark, and utilizing CLIP. But we do not rely on CLIP for deciding a category is positive or negative for an image.\n\n### 2. Using CLIP in the annotation process would fail in cases of having unlikely phrases, attributes or relations.\n\nThanks for the insightful question. As shown in the annotation process (diagram and text) and the answer #1, in the annotation process of D3, CLIP merely provides some initial candidate refs. Since\n\n(1) refs likely to be positive are bound with the image's group and always kept as candidates,\n(2) the selection percentage of CLIP is large (>10%) and adjusted based on manual check,\n(3) the refs not selected by CLIP is manually checked by annotators to be negative,\n(4) the annotators decide a ref is positive or negative,\n(5) the final annotations are checked by annotators twice,\n\nwe argue that the proposed D3 dataset offers manually labeled, accurate negative and positive labels with explicit negative and positive certificates, and CLIP only serves as a tool for accelerating the annotation process without deciding the positive/negative or harming the annotation accuracy.\n\n### 3. Difference with Phrase Detection task and existing benchmark COPS-Ref.\n\nApologies for leaving out these two relevant works. The main difference between DOD and Phrase Detection [4] is that Phrase Detection lacks explicit negative certificates. Negative instances are not labeled, so Phrase Detection is not a detection task. In DOD, we ensure that positive instances are annotated exhaustively and all the other references are reliably negative labels. Additionally, Phrase Detection focuses solely on the form of phrases, while DOD encompasses OVD and REC, allowing expressions to be words, phrases, or even sentences. \n\nCops-Ref [5] focuses on assessing the grounding capability of the REC method in difficult negative regions with related/distracting targets, ensuring explicit negative certificates for a small set of images in their benchmark. Thus, achieving the 'explicit negative certificates across a whole dataset' attribute, like in detection tasks, is only feasible in DOD.\n\nWe will include this discussion in the manuscript. Thanks for your valuable suggestions.\n\n### 4. Update REC SOTAs like FIBER. OFA is not SOTA.\n\nThanks for the reminder. In this work, we divide existing methods into REC, OVD and bi-functional methods. As FIBER handles both detection and REC, we believe it is more suitable to be classified as bi-functional methods rather than REC methods. Therefore, we think the expression \"OFA holds the SOTA among REC methods\" is valid. We also make comparison to joint OVD & REC methods like UNINEXT (outperforming FIBER on most metrics of REC) and Grounding-DINO (comparable to FIBER) in the paper.\n\nWe will add methods like FIBER in bi-functional methods and MDETR in REC methods in the related work of the manuscript. Thanks for your suggestions."}, {"review_id": "cPohw3QX9S", "reviewer": "Reviewer_SdJt", "summary": "Brief Summary: The paper presents a new task Described Object Detection which extends open vocabulary object detection (OVD) to use phrases. This, in turn, extends referring expression (REC) to include objects not seen in the training data. To this end, the authors introduce a new dataset called D3 building on existing GRD dataset [44]. \n\nThe key idea behind creating D3 are to have complete annotation (i.e. each referring expression has a bounding box in each image if present), have natural language (extends OVD), include absence expression (i.e. objects NOT having a particular feature / attribute such as blackboard with no signs), and one expression referring to more than one instance. \n\nThe authors experiment on the provided D3 dataset and provide a detailed benchmark with multiple baselines ranging from REC methods like OFA, OVD methods like OWL-ViT and bifunctional method like Grounding-DINO. The authors further propose a new baseline OFA-DOD which changes some pre-training schemes such as including additional localization tasks and find that it outperforms competitive baselines.\n", "questions": "See weakness section. \n\n", "limitations": "Yes, the authors have discussed some limitations but more could be included such as:\n\n1. The evaluation is heavily dependent on the choice of 412 phrases and the dataset used for GRD. \n\n", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Pros:\n\n1. Dataset contribution is always welcome. It is clear the authors have put thought into the dataset construction. In particular, absence expression are quite interesting.\n\n2. A number of baselines are considered and the takeaways are quite interesting that REC methods fail at this task due to being unable to effectively choose more than one boxes. While OVD methods  perform better, there is a very large gap compared to bifunctional methods like grounding-dino. The proposed baseline OFA-DOD makes sense and it is good to see that it outperforms other baselines. \n\n3. Visualization of the dataset as well as results in suppl. are very useful. Ablative studies on the baseline such as effect of training data (Table 5b) are interesting.", "weaknesses": "Cons:\n\n1. The authors should compare their work with zero-shot grounding [Ref1] which also extends REC to new objects. The obvious difference is that DOD can have more than one instances but a clear distinction would be helpful.\n\n2. The main idea behind DOD is to encompass both OVD and REC. The authors need to motivate this setting more. In my opinion, having the two cases separate can be much more revealing than trying to combine the two. OVD is strictly object detection (with phrases) while REC explicitly requires disambiguation between different objects. For instance, \"oversized glove on left-hand\" in suppl fig 2 (last row), is simply object detection of glove and doesn't require reasoning whether it is \"oversized\" or on \"left-hand\". To me a more natural setting is to separate the two. I would like to know about the author's motivation for the task.\n\n3. It is unclear in the text but in OFA-DOD which OFA is chosen, base or large? To have a fair comparison with Grounding-DINO it should be base but it is not clear (in main text as well as in suppl).\n\n4. In suppl. Table 2, it seems Grounding-DINO outperforms OFA-DOD by considerable margin on Average Recall. Why is this the case? For practical use, wouldn't one prefer using Grounding-DINO?\n\n\n[Ref1]: Sadhu, Arka, Kan Chen, and Ram Nevatia. \"Zero-shot grounding of objects from natural language queries.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4694-4703. 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the review for the positive feedback on the contribution and design on the dataset. as well as the visual and experimental analysis and findings.\n**Due to the page limit, please refer to the *[general response (author rebuttal)](https://openreview.net/forum?id=0hwq2vOHT4&noteId=EtVOyLQxeQ)* for the motivation of DOD task.**\nWe address the reviewer's comments below.\n\n### 1. Comparison with zero-shot grounding.\n\nThanks for your reminder. Zero-shot grounding is an intriguing study focused on locating concepts absent in the training set. However, it assumes the presence of objects described by query phrases in images (as shown in the Figure 4 of their paper), still falling under the REC task. In contrast, DOD aims to detect objects described by flexible expressions throughout the dataset. Thus, there can be zero, one or multiple objects described by the language reference in an image. The specific differences includes:\n- assuming the existence of objects (zero-shot grounding) vs. no such assumption (DOD).\n- one target only vs. multiple target.\n- short phrases vs. varied language description (from short category name, to phrases, and long descriptions).\n\nDOD and zero-shot grounding have different focus and zero-shot grounding can be regarded as a variant of REC and a subset of DOD. We will incorporate this discussion into the manuscript to avoid potential misunderstandings. Thank you for your suggestions. \n\n### 2. Motivation for the DOD task. Why not having DOD and REC separated.\n\nAs another reviewer also ask about the motivation, we put the clarification on the motivation in the ***[general response (author rebuttal)](https://openreview.net/forum?id=0hwq2vOHT4&noteId=EtVOyLQxeQ)***.\n\nAdditionally, regarding the mentioned \"oversized glove on left hand\", simplifying it to \"glove\" for object detection is used to highlight the second attribute (unrestricted description) of this dataset. For illustration purposes, only positive examples are included in the figure to avoid introducing information related to the first attribute (complete annotation). In reality, a glove might not necessarily be on the left hand, nor is it guaranteed to be oversized. Thus, the annotation of \"oversized glove on left hand\" does not align with the label \"glove,\" and there are cases where a positive example of \"glove\" is a negative example for \"oversized glove on left hand\". For instance, in supp. Fig. 1 top row, \"partially damaged car\" cannot be simplified to \"car.\" Applying a \"car\" object detector would lead to detecting numerous undamaged cars, generating an excessive number of detection results that do not achieve the intended objective.\n\n### 3. In OFA-DOD which OFA is chosen, base or large?\n\nThanks for the reminder. We build and evaluate OFA-DOD based on OFA-base. We wil add this note in the text and add the \"base\" subtext in Tab. 2 in the manuscript.\n\n### 4. In supp. Table 2, why is Grounding-DINO better than OFA-DOD on Average Recall? Wouldn't one prefer G-DINO in practical case?\n\nWe discuss the metric average recall in the supp. Line 180 - 183. Recall is a metric used by REC task, which only requires the model to locate an object known to exist, but no need to reject false positives. A model pursuing high recall can predict as many false targets as it want as long as the ground truth targets are included. Recall is not suitable for detection tasks like OVD or DOD, which requires the model to distinguish and reject negative instances.\n\nIn the evaluation of models for DOD, we use mAP to evaluate models' ability to both locate positive instances and reject negative instances. Average Recall is merely a metric for analyzing the characteristics of different models and its value does not reflect the quality or applicability of a model. Actually, we show in Sec. 5.2 of the manuscript that REC or bi-functional methods like G-DINO are difficult to reject false positive instances.\n\nThe proposed model is lower on Average Recall compared to G-DINO. This implies it is more \"conservative\" in prediction and tends to predict instances when it is rather certain. The choice of models depend on the use cases. For most detection settings where false positives are not welcomed, the proposed baseline should be more competitive in performance. When false positives are OK and the user just want to cover as many targets as possible, G-DINO is a nice choice, also considering its wide application and integration available in the community.\n\n### 5. More discussion on limitation.\n\nIndeed. Thanks for the suggestion. We will add more discussion as below in the \"limitation\" section of the manuscript:\n\n> As a human-curated dataset, D3 benchmark inevitably contains some bias during data collection and annotation. When designing the dataset, though we try our best to cover as many scenes as possible and make the image distribution and language diversity very broad, the evaluation is still heavily dependent on the choices of language descriptions and the distribution of images.\n"}, {"review_id": "qtj4uhJI5X", "reviewer": "Reviewer_EwWL", "summary": "This paper introduces a new Described Object Detection (DOD) task, which extends the existing Open-Vocabulary Object Detection (OVD) and Referring Expression Comprehension (REC) tasks into a more general paradigm. For this new task, the authors build a Description Detection Dataset (D3), and find the troublemakers that currently hinder current REC, OVD, and bi-functional methods. They further propose a baseline that outperforms existing methods on the DOD task. ", "questions": "Please refer to the weakness part. ", "limitations": "The authors have discussed the limitations and broader impacts. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tFocusing on language-driven object detection, the authors propose a new DOD task, which they argue is more practical and presents more challenges than the current OVD and REC tasks. They also introduce a new dataset D3 for this task. \n2.\tOn the DOD task, the authors thoroughly investigate the challenges faced by current REC, OVD, and bi-functional approaches, and they put forth a baseline method with state-of-the-art performance on this task. \n", "weaknesses": "1.\tWhile the authors introduce this new DOD task, instead of constructing a new training dataset or setting, they focus on analyzing the performance of models trained on the old OVD or REC tasks for this new task. This may not fairly represent the performance of existing methods for the new task.\n2.\tAs the DOD task can be seen as a more generalized version of the OVD task, where category names are extended to language descriptions, the experimental results and conclusions for OVD methods highly depend on their training data and settings. It would be beneficial to assess OVD baselines trained following the style and setting of the DOD task. \n3.\tThe compatibility of the DOD task with the existing OVD and REC tasks is not studied. Given that the DOD task is essentially a superset of the OVD and REC tasks, it would be beneficial to evaluate the baseline OFA-DOD in comparison with other methods on the OVD and REC tasks. \n4.\tThe existing methods and the proposed baseline are evaluated and compared under a zero-shot setting. How would the performance vary when the models are fine-tuned on the D3 dataset? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1. Analysis of existing methods on DOD may be not fair because this work does not introduce new training dataset.\n\nOur approach focuses on evaluating DOD using OVD/REC-trained models and providing insights for transitioning to DOD, emphasizing differences in training tasks and formats.\n\nCurrently, the evaluation is zero-shot for existing baselines. Ideally, we would also provide a dedicated DOD training set. However, due to the substantial annotation costs, we have not included a separate training set. Instead, we offer a dedicated test set with accurate annotations as benchmark.\n\nFurthermore, existing OVD/REC data can be leveraged to train DOD models, given the similarity in task formats between OVD/REC and DOD. Taking into account the huge annotation costs for a DOD training set, creating a new training set would provide limited value, and is not necessary or cost-effective.\n\nTherefore, this work focuses on:\n\n- Comparing generalization performance: We analyze the performance of existing methods on D3, aiming to uncover the capabilities of different models across various types and tasks for DOD. The primary goal is to reveal distinctions between training tasks and formats (OVD/REC/bi-functional), rather than a direct comparison of superiority. Thus, we allow for variations in architecture and data among these models.\n- Providing insights and guidelines: We offer insights and guidelines for transitioning from existing OVD/REC to DOD, and training DOD models using available data. These insights are not limited to a specific baseline. OFA-DOD is to demonstrate that simple adjustments can enhance the effectiveness of an REC model originally unsuitable for DOD.\n\n### 2. Assess OVD baselines trained following DOD settings rather than original setting.\n\nWe modified and trained OVD methods with adjustments similar to those applied in OFA-DOD (i.e., the reconstructed data step, which we reconstructs REC to DOD format). The results are shown in the table below, and the methods trained under DOD setting are denoted as OWLViT-DOD and CORA-DOD. Notably, the performance of these models on D3 surpasses original baselines (OWLViT and CORA). This shows the proposed modification over existing methods like OFA are transferrable to other existing OVD/REC methods.\n\n| Model | FULL | PRES | ABS |\n| --- | --- | --- | --- |\n| OWLViT | 9.6 | 10.7 | 6.4 |\n| CORA | 6.2 | 6.7 | 5.0 |\n| OFA | 3.4 | 3.0 | 4.3 |\n| OWLViT-DOD | 12.1 | 12.8 | 10.1 |\n| CORA-DOD | 7.9 | 8.2 | 7.1 |\n| OFA-DOD | 21.6 | 23.7 | 15.4 |\n\n### 3. Evaluate OFA-DOD on OVD and REC in comparison with other methods (to study the compatibility of DOD with OVD and REC).\n\nWe evaluate OFA-DOD on OVD/REC datasets. The results indicate **substantial improvements over OFA** for both OVD and REC. This shows the improvements of OFA-DOD over OFA make it better for DOD/OVD/REC, and when a model is improved to be more suitable for DOD, it also exhibit corresponding performance gains on REC and OVD. This implies that the DOD task is compatible with REC and OVD.\n\nCompared with SOTAs on REC (without fine-tuning), OFA-DOD outperforms SOTA like G-DINO. Compared with SOTAs on OVD (fine-tuning on base classes), OFA-DOD is not good as CORA, but outperforms Detic on novel classes, showing good generalization ability. We argue the reason why OFA-DOD does not obtain SOTA on OVD is that the original OFA is not suitable for detection tasks, incapable of rejecting negative instances, lacking compatibility with multi-target outputs and yielding poor results. Although OFA-DOD has augmented its detection ability and improved its performance on OVD by more than 20 mAP, it is still far from perfect for OVD and DOD. This is no surprise as it is only a baseline for future research.\n\nREC results:\n\n| Benchmark | refcoco |  |  | refcoco+ |  |  | refcocog |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Split | val | testA | testB | val | testA | testB | val-u | test-u |\n| OFA-base | 61.00 | 62.24 | 58.59 | 45.02 | 48.36 | 38.70 | 46.96 | 46.96 |\n| OFA-DOD-base | 75.92 | 78.74 | 72.16 | 64.98 | 71.32 | 59.25 | 71.52 | 71.76 |\n| G-DINO-L | 73.98 | 74.88 | 59.29 | 66.81 | 69.91 | 56.09 | 71.06 | 72.07 |\n\nOVD results:\n\n| Benchmark | COCO-OVD |  |  |\n| --- | --- | --- | --- |\n| Split | novel | all | base |\n| OFA | 3.2 | 7.4 | 8.9 |\n| OFA-DOD | 28.4 | 30.1 | 30.7 |\n| Detic | 27.8 | 45.0 | 47.1 |\n| CORA_R50 | 35.1 | 35.4 | 35.5 |\n\n### 4. How would the performance vary when the models are fine-tuned on D3 rather than zero-shot eval?\n\nThis work does not propose a new training set (due to the reasons in Q1, including huge annotation costs and availability of existing detection & REC data), making fine-tuning unfeasible. However, to address the question from the reviewer, we partitioned the D3 dataset into training (80 groups, 238 presence and 80 absence refs) and testing (26 groups, 78 presence and 26 absence refs) subsets and fine-tune various baseline methods on the training subset, subsequently evaluating the performance on the testing subset.\n\nWe show the results of methods without or with (*) fine-tuning on the training subset and evaluated on the testing subset (not the complete D3). As the results shows, OVD/REC/bi-functional/DOD methods are improved by certain margins (2 to 5 mAP). Note that D3 remains as an evaluation benchmark, not providing a training set. The models are expected to be trained on datasets such as OVD/REC and then tested on the D3 dataset. The introduced training/testing split here is solely for validation purposes.\n\n| Task | Model | FULL | PRES | ABS |\n| --- | --- | --- | --- | --- |\n| OVD | OWLViT | 9.7 | 10.5 | 7.1 |\n|  | OWLViT* | 13.0 | 13.8 | 10.7 |\n|  | CORA | 5.7 | 5.8 | 5.4 |\n|  | CORA* | 8.6 | 9.1 | 7.2 |\n| REC | OFA_base | 4.0 | 3.9 | 4.1 |\n|  | OFA_base* | 8.2 | 8.1 | 8.4 |\n| bi-func | UNINEXT_huge | 20.2 | 21.7 | 15.6 |\n|  | UNINEXT_huge* | 22.3 | 22.8 | 20.9 |\n| DOD | OFA-DOD | 21.4 | 22.6 | 17.7 |\n|  | OFA-DOD* | 24.2 | 25.8 | 19.5 |\n"}, {"review_id": "MzYXxdjmDI", "reviewer": "Reviewer_YHXi", "summary": "In this paper, the authors propose the definition, dataset, evaluation metrics and benchmark results of a new task - Described Object Detection (DOD). DOD is designed to detect objects aligned with full, presence or absence langauge descriptions. The proposed evaluation metrics have three groups based on full, presence or absence descriptions. The authors also propose the Intra-scenario mAP that only detect the categories appear in the image, and Inter-scenario mAP that detecting all the categories. The authors provide the benchmarks results on the proposed task with a new OFA-DOD baseline.", "questions": "3. In the proposed DOD task, the category-level definition is vague, and let me feel confused. The authors pre-defined 422 expressions or categories and use these 422 categories for evaluating the Inter-scenario mAP. But are these categories mutually exclusive or overlapped? For example, I guess some short expressions (e.g., 'backpack') may be a parent node of other long expressions (e.g., 'backpack with yellow color'). The authors may consider this point when designing their evaluation metrics, like adopting the positive/negative labels in the OpenImage and LVIS datasets.\n- 3.1 Figure 3 (b) and (c) use the \"number of instances\", does it mean \"number of categories\"? \n\n4. The motivation for the new DOD task is not clear to me. Can the authors provide straightforward application scenarios or real-world examples that previous tasks are limited while the proposed new DOD task offers broader possibilities?\n", "limitations": "In #341-345, the authors have discussed their limitations regarding potential abuse and carbon emissions.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors analysize the limitation of previous tasks\n- The authors provide comprehensive benchmark results on the proposed new settings", "weaknesses": "First, I want to clarify two different tasks: category-level object detection and visual grounding. The category-level tasks like OVD focus on recognizing pre-defined C categories, e.g., \"cat,\" \"dog,\" etc. These categories are commonly mutually exclusive (COCO) or share a parent-child relationship (LVIS). The OVD task will divide the C categories into base and novel two disjoint groups and the models are required to train on base classes and test on novel classes to verify the generalization ability. Instead of detecting C classes, the grounding tasks like REC, aim to align each region to a text phrase. The REC datasets like RefCOCO do not pre-defined a set of categories.\nBased on my personal understanding on the OVD and REC tasks, I think this paper has the following weaknesses:\n\n1. The authors argue that DOD is a superset of OVD (#35 and the right part of Figure 1). But I disagree with this argument. The OVD task focuses on evaluating the generalization ability. However, for the DOD task, the proposed dataset evaluation metrics do not consider dividing base/novel groups. Thus, OVD is suitable to evaluate the generalization ability, but DOD is not suitable. If the authors think that the DOD task does not focus on the generalization ability, they should not argue that \"superset of OVD\".\n  - 1.1 For the proposed D3 datasets for DOD, the authors pre-defined 422 categories. What happens to the other expressions that are not pre-defined? The authors may add some novel classes in the test set that are not available during training to simulate this situation. I think the base/novel setting is one advantage of OVD over DOD. If the proposed D3 dataset has already some novel expression in the test set, the authors may consider adding evaluation results on base/novel groups.\n  - 1.2 The left part of Figure 1 is the definition of object detection rather than open-vocabulary object detection with the base/novel setting.\n  - 1.3 From my perspective DOD is focused on fine-grained language understanding (e.g., the unrestricted language description in #123-127), which is not the main focus of OVD.\n\n2. #188-122: For the REC task, the model is required to align the region with the input language expression and the datasets of the REC task. In my opinion, I don't consider \"complete annotation\" to be a disadvantage of REC. For instance, REC datasets such as RefCOCO & RefCOCO+ consist of 141,564 language queries, which is significantly larger than the proposed D3 dataset with only 422 language queries. Can we conclude that the vocabulary size of REC datasets is larger than that of DOD? Can we argue that due to the cost of annotation, the \"complete annotation\" requirement restricts the vocabulary size?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments. **Due to the page limit, please refer to the *general response (author rebuttal)* for the motivation of DOD task.**\n\n## 1. Is DOD a superset of OVD or not?\n\nWe want to clarify that DOD is a superset of OVD: The D3 dataset is designed solely for evaluation and does not include a training set, so models are trained on OVD/REC datasets and then evaluated on D3. Since the descriptions in D3 has little overlap with existing datasets, the categories in D3 can be regarded as novel categories relative to the training categories (regarded as base) in OVD/REC datasets.\n\nThe evaluation of D3 is conducted in a zero-shot/open-vocabulary manner, which is analogous to the evaluation of novel categories in OVD. So DOD also \"focuses on the generalization ability.\" Based on the clarification in the reviewer's comments, DOD does qualify as a superset of OVD.\n\n### 1.1 Base/novel split in D3.\n\nWe can regard all classes in training datasets (OVD, REC) as \"base\" and all the classes in D3 as \"novel\".\nAs mentioned above, D3 is exclusively designed for evaluation, and its categories has little overlap with the categories in the training datasets (OVD, REC, etc.), given that the 422 reference categories in D3 are specifically designed. Consequently, D3 can be seen as a test set comprising novel categories, which corresponds to the novel split in OVD. Therefore, the results reported on D3 so far can be regarded as zero-shot generalization results on \"novel\" categories.\n\nWe add a table to clarify this:\n\n| task | training set | test set |\n| --- | --- | --- |\n| OVD (e.g. COCO OVD) | base group in OVD dataset (e.g. COCO-base) | novel group in OVD dataset (e.g. COCO-novel) |\n| REC (e.g. RefCOCO) | training split of REC dataset (e.g. RefCOCO train set) | testing split of REC dataset (e.g. RefCOCO test set) |\n| DOD | Existing OVD/REC datasets (e.g. COCO, RefCOCO, etc.) | D3 |\n\n### 1.2 Fig. 1 (a) defines object detection rather than OVD (with base/novel setting).\n\nThe illustration of Fig. 1 was simplified to facilitate presentation of different tasks. In (a), our primary intention was to illustrate that for OVD (1) only short category names are provided without lengthy descriptions, and (2) a category may or may not appear in an image. It does not  emphasize the aspect of zero-shot generalization (given that both OVD and DOD tasks inherently involve zero-shot scenarios).\nActually Fig. 1 (a) is not limited to either OD or OVD, as we did not specify whether categories like \"oven,\" \"dog,\" \"person,\" etc. in the example, exist in the training dataset or not. The figure does not focus on generalization, does not distinguish between training and testing and thus does not delve into concepts such as base/novel divisions.\n\n### 1.3 DOD focuses on fine-grained language understanding, which is not the main focus of OVD.\n\nThe DOD task includes short or long descriptions, not always fine-grained. It is the D3 dataset rather than the DOD task itself that focus more on fine-grained descriptions (but still covers different granularity).\nIn the case of the DOD task, when dealing with longer descriptions, it indeed involves fine-grained language understanding. However, when descriptions are shorter (especially just one or two words), it is more similar to category-based detection. This can be observed in Fig. 2 (b) of the paper and the examples in supp. Fig. 2. When descriptions are shorter, the task aligns with the essence of OVD, hence making DOD a superset of OVD.\n\nFor the proposed D3 dataset, its annotations do lean towards relatively longer descriptions, focusing more on fine-grained understanding. This design choice was intended to introduce higher levels of challenge.\n\n## 2. In REC and DOD, complete annotation limits the scale of references, so not having complete annotation is not a drawback of REC.\n\nIn REC datasets, forgoing \"complete annotation\" did led to more references defined compared to DOD. However, lacking such complete annotation make REC not applicable to many practical detection-related scenarios (as described in the general response), which makes the DOD task with complete annotation useful.\n\n## 3. Are categories in D3 mutually exclusive or overlapped? Consider this when designing the evaluation metrics.\n\nThese categories are not necessarily mutually exclusive; they can overlap in certain instances (hence, the classification in the DOD task is multi-label, not single-label classification), but there's no hierarchical inclusion relationship.\nWhen designing the dataset's categories, we deliberately avoided including categories with hierarchical relationships (such as \"backpack\" and \"backpack with yellow color\") to prevent D3 from becoming too straightforward in terms of difficulty.\nIn D3, for an image, categories not positively labeled are manually verified as negative. In other words, our annotations are exhaustive, eliminating the need for partial negative labels to explicitly denote some negative categories in the federated annotation manner of datasets like OpenImage or LVIS. Therefore, naive mAP is suitable for D3 while for OpenImage and LVIS positive/negative labels needs to be considered.\n\n### 3.1 number of instances\" or \"number of categories in Figure 3 (b) and (c)?\n\nAs shown in the caption of Fig. (3) (b) and (c), they mean \"number of instances\". More specifically, Fig. 3 (b) means number of positive instances in all images for one description (i.e., one category), which shows that each description has a sufficient number of instances across the dataset. Fig. 3 (c) means number of positive instances in a positive image for one description, which shows a description can have one or multiple (usually 2 to 5) instances in an image.\n\nNote that for \"number of categories\", we show the distribution of number of descriptions (categories) on one image in supp. Fig 4 (a).\n\n## 4. Motivation of DOD and real-world application that previous tasks are limited?\nPlease see the ***general response***."}, {"review_id": "IzcN68bpjh", "reviewer": "Reviewer_wkHv", "summary": "The paper presents a new multi-modal computer vision task, called Described Object Detection, which is a superset of existing OVD and REC tasks. In particular, the DOD task seeks to create models which can detect multiple instances of something in images, from textual descriptions, which could include describing the absence or presence of something. The paper then goes on to create a new dataset for this task, based on an existing one, called $D^3$. It then shows how existing OVD, REC, Bi-Functional, and proposed a baseline method perform on the data set. ", "questions": "Please see the weaknesses section on the questions related to the annotation process and the OFA-DOD model.", "limitations": "The paper adequately addresses its limitations and potential negative societal impact.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The paper is attacking a significant problem, is of high experimental quality, and is novel. The idea of having free-text descriptors for finding things in images is extremely important to the adoption of ML for various important, real-world tasks. In fact, I was surprised to learn that given the importance of this concept, something like DOD had not been proposed before. The paper is also of high quality in that it not only identifies this shortcoming of real-world, open-set detection tasks (i.e. DOD), but also creates a dataset, tests it across a range of existing methods, and proposes a new baseline. Because the paper does propose a new formal problem definition for something that is desired in the real-world performance of ML systems, and therefore realizes how current tasks like REC and OVD fall short of what is actually needed for things like open-set, zero-shot object detection it is a novel look at the problem domain. Finally, it is also worth mentioning that the inclusion of the absence examples is a novel idea, which shows in the poor performance of models on absence instances in the data set.", "weaknesses": "The only major weakness of the paper is its clarity. This lack of clarity manifests primarily in two places. First, the description of the creation of the data set needs more detail. Specifically, how was CLIP utilized for adding complete annotations? Were full images given to CLIP with all of the possible annotations and then the most probable selected as an annotation for the image? And, if so, how did you deal with multiple annotations being present in the same images? Additionally, how were the negative image annotations created? Were these done by hand, and, if so, by how many people? Similarly, how many people were involved with creating multiple-instance bounding boxes? The annotation section might benefit from a diagram to better explain this process. \n\nSecond, the baseline method is poorly described. It's not clear from the main paper or the semantic similarity how the different proposed components come together into the OFA-DOD model. The paper would greatly benefit from a figure displaying the OFA model and the OFA-DOD model to better understand the architecture of the OFA-DOD model. As it stands now, a reader would have a hard time recreating the OFA-DOD model from the given text.\n\nFinally, there are some minor typo issues that need to be addressed. For example, line 130 page 4 repeats “their” twice, and line 212 on page 8 has the wrong subject-verb agreement (i.e. “fails” instead of “fail”). The paper could use another proofread for typos and grammar.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedbacks.\n**Due to the page limit, please refer to the *[general response (author rebuttal)](https://openreview.net/forum?id=0hwq2vOHT4&noteId=EtVOyLQxeQ)* and the PDF file there for the description and diagram of annotation process of D3 dataset, and the figures showing the model structures of OFA and OFA-DOD.**\nWe address the comments below.\n\n### 1. Clarity on the creation and annotation process of the proposed dataset.\n\nThanks for the helpful suggestion.\n\nWe have added a diagram to show the annotation process, along with text to describe the details of each step, in the ***[general response (author rebuttal)](https://openreview.net/forum?id=0hwq2vOHT4&noteId=EtVOyLQxeQ)*** (diagram in the PDF). We sincerely hope the reviewers will look into this, and believe this diagram and the text will be able to answer the previous questions regarding the clarity on annotation process.\n\nAdditionally, we answer the specific questions related to the annotation process as below:\n\n- [Simple explanation of the annotation process] The image are divided into different groups and the description in different group are unlikely to appear in images from other group. For each image, the refs in its group are used. Refs from other groups may also appear, but with a smaller probability, so we use CLIP for selecting a large number of candidates from these refs from other groups. We manually check by statistics that such CLIP filtering usually do not miss positive refs. Then annotators select the positive refs from these refs (rather than all refs in the dataset) and add boxes for each image.\n- [How to use CLIP] CLIP is used merely to filter the references from other groups and decide some negative refs (not all). The refs kept are candidate refs.\n- [How to decide positive and negative for an image, and how to deal with multiple annotations in a image] The annotators select the positive refs (one or more) from candidate refs. Candidate refs can be both the refs from the image's group and the refs filtered and kept by CLIP. They add boxes manually.\n- [How were the negative image annotation created] For each image, the negative refs are (1) those filtered out by CLIP, which the annotators will check and make sure no positive exists, or (2) the candidate refs decided as not positive by the annotators. So, all refs not labeled positive are negative, and they all have manual negative certificates from the annotators.\n- [Human annotation cost] The number of person involved in the annotation process: \n   - Data source and step 1: the authors (3 people, 3 days)\n   - Step 2: automatic, by programs (with CLIP)\n   - Step 3: trained annotators (5 people, 1 week)\n   - Step 4: trained annotators (15 people, 3 weeks)\n   - Step 5: trained annotators (15 people, 2 weeks)\n\nAs the annotation process is rather complicated, we recommend to look into the annotation process description in the general response first.\n\n### 2. Clarity on the proposed OFA-DOD baseline and a figure of model structure.\n\nThanks for the question. We did not elaborate the details of OFA-DOD due to limited pages of the manuscript and its less important contribution compared with other parts, so we provided a more detailed description on the components differentiating OFA and OFA-DOD in the supplementary material, including the model structure, training setting and inference strategies.\n\nIn this response, we provides figures showing the differences between OFA (Fig.2) and OFA-DOD (Fig. 3) in the author rebuttal PDF file. As shown in the figures, \n\n- The first modification, granularity decomposition, corresponds to replacing a shared decoder to two parallel decoders, one for global tasks and one for local tasks. This is likely to alleviate the conflicts between tasks of different granularity and make the model more suitable for localization tasks.\n- The second modification, reconstructed data, refers to the reconstructed OVD & REC data for the local decoder. After the reconstruction, for both task, the input can be one or multiple references (or object category names) and they can corresponds to zero, one or multiple targets.\n- The third modification, task decomposition, is depicted by adding a binary classification in the global branch. This task determines if a bounding box and a description is matched, and is used as a second step to reject negative instances in inference.\n\n### 3. Some minor typo issues. \n\nThanks for the suggestions. We just fixed the typos you mentioned together with a few others we found and proofread the whole paper several times. Thank you again for the detailed suggestion."}]

Author rebuttal:
We thank the reviewers (R1: Q5jG, R2: SdJt, R3: EwWL, R4: YHXi, R5: wkHv) for their positive feedbacks, such as the contribution of the dataset (R2, R5), the significance of the target problem (R3, R5), the design of the dataset is throughly considered (R2), the absence description characteristics is interesting (R2, R5), the findings in the comparison of different tasks is informative (R2, R4, R5), the experimental and visual analysis is interesting and comprehensive (R2, R3, R4, R5), the proposed baseline is effective (R2, R3), the writing is clear and clarified (R1).

In the general response we answer some questions asked by more than 1 reviewer. We address each reviewer's question in the individual response for them. We will revise the paper accordingly.

## Annotation process of the proposed $D^3$ dataset.

A diagram illustrating the annotation process of the proposed dataset is in the PDF file. Here we describe the steps for annotation as below:

Data source: 106 groups from GRD with about 100 images and 3 ~ 4 designed refs for each group. Each group belongs to a different scenario and the overlapping between refs from different groups are small (i.e, a ref for one group are not likely (but possible) to appear in the image from another group). Now we have 10000+ images and 300+ refs.

1. [Manual] Adding absence refs: design 1 ~ 2 absence refs based on the images for each group and add them to the corresponding groups. Now we have 400+ refs.
2. [Automatic] Selecting possible positive refs: for each image, select **all the refs **(4 ~ 6) from the group it belongs to, and also the other 105 groups (top-n refs out of 400+ refs, by CLIP similarity between the image and each description). Now for each image, we have n+4 ~ n+6 candidate refs and all the other refs are filtered out. n is set as 40 initially.
3. [Manual] Verification: randomly choose 5 groups of images, and check if there are any positive refs that should not be filtered out. If so, increase n to cover that ref and go back to step 2.
4. [Manual] Manual annotation: annotation by trained annotators on all images. The annotation of boxes (and instance masks) are instance-level, dataset-wise complete, and includes absence refs.
5. [Manual] Quality check: this includes 3 small steps: 
   1. Discarding some images (ambiguous, etc., unsuitable for annotation) or categories from the dataset. About 8% samples are discarded.
   2. Quality check on 100% samples. For each group, if image with error is more than 2%, it is returned for re-annotation. Otherwise the errors are fixed and this group passes this step.
   3. Final check on 5% samples. For each group, if there are image with error, it is returned, otherwise it is accepted.

## Motivation of the proposed DOD task.

Both the OVD and REC tasks have their respective limitations.

- OVD can only perform detection based on categories, where the detection targets are limited to "certain object classes" rather than "objects with specific attributes/relationships." This approach lacks an understanding of contextual information within images and cannot leverage language to precisely control detection targets and requirements. This inflexibility prevents it from meeting specific application demands.
- REC, while capable of comprehending longer object descriptions with attributes or relationships, assumes the existence of such objects in the image. In cases where the described object doesn't exist, REC lacks the ability to reject or filter, leading to false positive errors. This issue poses a significant problem for practical applications and limits its direct usability.

Consider a practical scenario, such as detecting "individuals without helmets" in a construction site using camera data. An OVD method can detect objects like "helmets" and "people" and generalize, but it can't determine the relationship between people and helmets, rendering it unsuitable for direct application. On the other hand, the REC method produces localization results in any image, but often generates false positives, making it impractical.
The current solutions involve breaking down the process, first detecting "people" and "helmets," then training a separate model to determine the relationship between them, or determining presence first, followed by the REC method for localization. This approach requires multiple specialized models, tailored for each scenario, which is far from practical and is inefficient in terms of development.

Hence, there is a significant demand for detection based on language descriptions – a model with strong generalization capabilities, capable of determining whether the described object exists in the image and localizing it based on arbitrary language descriptions. This is where our proposed DOD task comes in.
The introduced DOD task has various practical applications, including:

- Urban security, like detecting "individuals without helmets" in construction sites, "dog outside without leash" in communities, "clothes hung outdoors" on a street, "overloaded vehicles" and "fallen trees on roadsides" on the road, etc.
- Network security, where sensitive images containing bloodshed or violence need to be detected within a massive image dataset.
- (Fine-grained) photo album retrieval based on language (descriptions, keywords, etc.).
- Retrieval and filtering of web image data.
- Detection of specific events in autonomous driving, such as "pedestrians crossing the road".

These scenarios are beyond the capabilities of both OVD and REC. This is the motivation behind DOD.

## Content of the attached PDF

In this PDF file we include 3 additional figures:
- Figure 1 is the diagram of the annotation process for $D^3$.
- Figure 2 and Figure 3 show the model structures of OFA and the proposed OFA-DOD.


Meta-review: This paper extends open-vocabulary object detection to support short and long-form descriptions rather than short category names, referred to as Described Object Detection (DOD). To address this, a (evaluation-only) dataset and benchmark is proposed, which is fully annotated and includes a variety of description characteristics. Benchmarks show that methods developed for close settings (open-vocabulary detection and referring expression comprehension (REC)) do not perform well. Some simple modifications are proposed to improve results, including a dual-decoder and reconstructed data, although the task is still challenging for modern methods. 
 
  While the reviewers appreciated the setting and clear writing, a number of concerns were raised including: 1) Dataset generation, e.g. using CLIP (Q5JG, wkHv), 2) Setting design, e.g. defining categories and evaluation metrics (YHXi), 3) novelty and motivation of the setting (Q5JG, SdJt), and 4) assessment of related methods such as OVD methods (EwWL). The authors provided a rigorous rebuttal including a thorough description of the annotation process and motivation, additional experiments (OFA-DOD and fine-tuning), and more. 

  By the end of a rigorous back-and-forth, all but one reviewer suggested acceptance and were satisfied with the response, either keeping their positive score or increasing it. For the reviewer that had a more negative score, there again was a great discussion of potential limitations in terms of overlap between the base/novel classes, evaluation of both novel and base class performance and use of mAP for this, etc. While not all points were resolved, I believe that a majority of them were and on the whole, this paper presents a valuable, impactful, and scientifically interesting setting that could form the foundation of interesting lines of research. It is also well-executed, with thorough analysis and insights, which all of the reviewers agreed. 

  I therefore recommend acceptance, but heavily encourage the authors to incorporate the many additions and perspectives offered by the reviewers. This will significantly increase the impact of the paper.
Decision: Accept (poster)