SlowGuess/NeurIPS_2023_OCR huggingface dataset:
REC methods. We employ the state-of-the-art REC method, OFA [39], with two variants. OFA is based on an encoder-decoder, sequence-to-sequence framework. It is a multi-modal multi-task generalist that deals with different tasks together and is trained on various tasks, including language tasks (masked language modeling), image-to-text tasks (image captioning and Visual Question Answering (VQA)), and localization tasks (REC). Notably, although it is trained with a detection dataset, it is not evaluated on object detection and achieves poor performance if we do. Currently, it holds the SOTA performance on standard REC benchmarks like the RefCOCO series.
OVD methods. We evaluate OWL-ViT [29] with two variants and CORA [42]. They are the SOTA methods on OVD tasks, with a vision transformer as well as a language transformer. They are pretrained with image-text contrastive learning and then fine-tuned on detection dataset.
Bi-functional methods utilizing both REC and OVD data. Methods falling into this category are not many but emerging fast recently. We test two methods: Grounding-DINO [25] and UNINEXT [44], each with two variants. Both of them are based on DETR [3]. They are pretrained on multiple datasets, including detection and REC datasets, and then evaluated with different strategies for different tasks.
4.2 A proposed baseline
$\mathrm{D}^3$  is very challenging for existing works, as we will demonstrate in Sec. 5.1. We have selected one of these works for adjustment to provide a better baseline. The chosen work should (1) be capable of understanding text of various lengths; (2) excel in their original tasks; (3) have a framework with a rather simple technical design, allowing us to modify its components easily. We have chosen OFA because it (1) is a multi-modal multi-task framework with MLM (Masked Language Modeling) and image-to-text pretraining; (2) achieves SOTA on REC; (3) has a simple seq2seq framework.
However, OFA faces several problems that make it unsatisfactory for this task, as discussed in Sec. 5. First, forcing multiple tasks of different modalities into one seq2seq framework adversely affects the performance of specific tasks, especially tasks related to localization. Second, training on the grounding task results in poor ability to handle multiple instances. We evaluated the model on COCO detection, and it achieved less than  $10\mathrm{mAP}$ . Thirdly, its REC paradigm also makes it predict only one instance, making it unable to reject negative images and irrelevant descriptions.
Therefore, we have made some modifications to OFA to make it more suitable for this task. The first modification is granularity decomposition to make it more suitable for localization. We have divided the pretraining tasks of OFA into two different granularities: global tasks (related to language modeling, such as captioning, VQA, MLM, etc.) and local tasks (related to localization, such as detection and REC). We have added an additional decoder parallel to the original decoder in OFA that handles the local tasks, while the original decoder focuses on the global tasks. This alleviates conflicts between different tasks and enhances localization.
The second modification is reconstructed data for pretraining on REC, aiming to improve multi-target localization. We have reconstructed the data for REC to ensure that (1) multiple references are input for an image, and (2) a reference does not necessarily correspond to one object, but zero or multiple. This results in a unified data format for detection and REC, although the labels may be noisy since they were not initially prepared for DOD.
The third modification is task decomposition to empower the model with the ability to reject false positives. We have reformulated the training on reconstructed data into two tasks: REC (for locating a region based on a reference) and VQA (for determining if a region and a reference match each other, essentially a binary classification). The second step is responsible for rejecting false positives.
We refer to the model with all three modifications as OFA-DOD. More details on the proposed improvements can be found in the supplementary materials. It is important to note that this model is far from perfect for the complex  $\mathrm{D}^3$  benchmark. As we will show in Sec. 5.1, although it outperforms existing methods, it serves as a baseline for future tasks on  $\mathrm{D}^3$ .
Table 2: Comparison of different methods on the proposed dataset for different mAP metrics.
Task	Method	Intra-scenario	Inter-scenario
FULL	PRES	ABS	FULL	PRES	ABS
REC	OFAbase	3.4	3.0	4.3	0.1	0.1	0.1
OFAlarge	4.2	4.1	4.6	0.1	0.1	0.1
OVD	CORAR50	6.2	6.7	5.0	2.0	2.2	1.3
OWL-ViTbase	8.6	8.5	8.8	3.2	3.7	4.7
OWL-ViTlarge	9.6	10.7	6.4	2.5	2.9	2.1
Bi-functional	UNINEXTlarge	17.9	18.6	15.9	2.9	3.1	2.5
UNINEXThuge	20.0	20.6	18.1	3.3	3.9	1.6
G-DINOtiny	19.2	18.5	21.2	2.3	2.5	2.1
G-DINObase	20.7	20.1	22.5	2.7	2.4	3.5
DOD	OFA-DODbase	21.6	23.7	15.4	5.7	6.9	2.3
(a) OFA
(b) OWL-ViT
Figure 4: Distribution of TP and FP scores from different baseline methods.
(c) Grounding-DINO
(d) OFA-DOD
5 Experimental Analyses
5.1 Comparison of baselines on our metrics
We make comparisons on the baselines introduced in Sec. 4, mainly with the intra-scenario setting. Unless explicitly noted, this is the default setting, instead of the more difficult inter-scenario.
Existing SOTAs are insufficient for DOD, and bi-functional models outperform others. As demonstrated in Tab. 2, existing methods, while achieving SOTA performance on their original benchmarks, fall short in delivering strong performance on  $\mathrm{D}^3$ . Among them, recent bi-functional methods [25, 44] are notably superior to others, and currently, OVD methods outperform REC. The inferiority of REC methods is likely due to their impractical setting described in Sec. 1, which involves predicting one and only one instance for each reference. We will delve into this further.
Rejecting irrelevant references are difficult, which REC are naturally incapable of. In contrast to intra-scenario evaluation, the inter-scenario setting assesses all references in the dataset for each image. Since references from other scenarios are likely not semantically relevant to the images, this necessitates the ability to reject irrelevant references for an image. This aligns with the evaluation in standard detection tasks. From Tab. 2, it is evident that OFA, a REC method, almost completely fails in this setting. This is caused by its prediction of a region for every reference, resulting in a large number of false positives when there are numerous candidate references. This underscores the importance of empowering REC methods with the ability to reject false positives. We find that none of the verified methods achieve good performance under the inter-scenario setting, indicating that existing methods are far from being capable of DOD. This highlights the challenge of  $\mathrm{D}^3$
The proposed baseline outperforms existing methods. The proposed baseline is based on OFA, but our improvements significantly enhance its performance. It outperforms all existing methods in the intra-scenario setting and surpasses them by a wider margin in the inter-scenario setting. This may suggest that the proposed baseline has a stronger ability to reject irrelevant references. Nonetheless, the proposed method is far from perfect and can only serve as a baseline for future research.
Table 3: Evaluation regarding different number of instances in a image for each reference.
Method	No-instance FPPC (%) ↓	One-instance mAP (%) ↑	Multi-instance mAP(%)
2	3	4	4+
OFA	100.0	14.8	9.5	7.9	5.4	3.7
CORA	17.3	9.7	8.4	9.5	9.0	8.5
OWL-ViT	41.9	21.1	17.3	16.6	16.0	14.0
UNINEXT	100.0	55.7	26.2	18.6	14.4	9.0
G-DINO	100.0	63.7	28.3	19.7	15.9	10.1
OFA-DOD	35.6	56.4	19.6	12.7	10.3	7.1
Table 4: Evaluation one references with various lengths.
Method	short	middle	long	very long
OFA	4.9	5.4	3.0	2.1
OWL-ViT	20.7	9.4	6.0	5.3
UNINEXT	18.5	23.3	17.4	16.1
G-DINO	22.6	22.5	18.9	16.5
OFA-DOD	23.6	22.6	20.5	18.4
5.2 Further analysis
Absence descriptions are more difficult for most methods. As shown in Tab. 2, the performance of baseline methods on PRES (presence descriptions) is consistently superior to that on ABS (absence descriptions). This suggests that existing methods may not effectively differentiate between the presence and absence of attributes in a language description.
REC methods fail to provide good confidence scores. We visualized the score distributions from baselines for TPs and FPs, to assess their capabilities in classification and confidence estimation. As in Fig. 4, the confidence scores from OFA do not exhibit a clear distinction between TP and FP cases. This can be attributed in part to the seq2seq framework in OFA, which does not directly yield confidence scores, and in part to the grounding formulation of REC, which identifies the image region most similar to the text description without distinguishing between positive and negative.
With a task decomposition step to enhance binary classification performance, our OFA-DOD demonstrates a significant disparity between TP and FP score distributions, yielding more reliable classification results. Note that this improvement does not necessitate modifications to the model framework or training datasets; rather, it is attributed to a more appropriate task formulation.
Multi-instance detection is challenging for methods other than OVD. For each image,  $\mathrm{D}^3$  can have zero to multiple instances for a single description. To assess how current methods handle varying numbers of instances, we conducted evaluations under three different settings: no-instance, where for a reference, evaluations are limited to images without any referred instance; one-instance, for images with a single instance; and multi-instance, for images with multiple instances. As shown in Tab. 3, OVD methods outperform others when multiple instances are referred by the description, although they may not be as competitive on the entire dataset or images with few instances. Notably, OWL-ViT maintains consistent performance even as the number of instances increases, which sets it apart from other methods. In contrast, REC and current bi-functional methods struggle in multi-instance scenarios. This highlights the strength of OVD methods in multi-target detection, while REC and current bi-functional approaches are less robust in such situations.
REC and bi-functional methods lack the ability to reject negative instances. In the no-instance column of Tab. 3, we do not report mAP since there are no positive instances in GT for the corresponding reference, making AP inapplicable. Predictions on such images are FPs, so we measure the ratio of images where FPs are produced to the total number of no-instance images for a given reference, namely False Positives Per Category (FPPC). We report the average FPPC over all references. We observe that most baselines are incapable of determining whether an image contains the referred target or not, yet they still produce predictions. This behavior is expected for REC methods. Bi-functional methods, trained and inferred with the REC task formulation, also exhibit this issue. Only the OVD method and our proposed baseline can effectively reject such negative image-text pairs.
Table 5: Ablation on the proposed baseline for its improvement components and the training data.
(a) Method components.
OFA	GD	RD	TD	mAP(%)
✓	X	X	X	3.4
✓	✓	X	X	10.5
✓	✓	✓	X	17.2
✓	✓	✓	✓	21.6
(b) Training data.
REC	OD	I2T	MLM	mAP(%)
✓	✓	✓	✓	21.6
✓	X	✓	✓	16.4
✓	✓	X	✓	14.2
✓	✓	✓	X	20.3
OVD methods suffer from long descriptions greatly while others do not. We partition the references according to their lengths and then evaluate on these partitions. The results are shown in Tab. 4, where short, middle, long and very long corresponding to references with  $1 \sim 3$ ,  $4 \sim 6$ ,  $7 \sim 9$ , and more than 9 words. For short descriptions, which is close to OVD setting, OVD and bi-functional methods obtain similar performance. However, as the length of references increases, the performance of OVD methods decrease fast, while REC and bi-functional methods suffer less from this. We can see that OVD methods are sensitive to long references, as expected, while other two types do not.
More experiments and additional qualitative results are available in supplementary materials.
5.3 Ablation on the proposed baseline
Method components. In Tab. 5, we perform ablation on the proposed improvements in our baseline, step-by-step from OFA to OFA-DOD, to see how they affect the performance. Granularity decomposition (GD) makes the method more suitable for localization task. It disentangle tasks of global or local granularity by handling them with 2 separated branch. Reconstructed data (RD) uniforms REC and OD data into the same form, and prepares multi-instance samples with both short and long references. Task decomposition (TD) is proposed to help rejecting FPs. It breaks down the DOD task into a REC step followed by a VQA step. All three of them improve the performance obviously.
Training tasks. We also perform a drop-one-out ablation on the multi-modal multi-task training data, in Tab. 5b. Detection data provides samples for localization, especially multi-instance situation. It is instinctively important for learning to localize, and indeed matters for performance. I2T (image-to-text, like image captioning and visual question answering) often helps the generalization and zero-shot performance of multi-modal methods. We find that it does affect the zero-shot performance on  $\mathrm{D}^3$  greatly. MLM is theoretically important for language understanding and generalization. However, we find it actually is not. Removing the MLM task has no significant effect on the performance. We surmise that the generalization ability of OFA-DOD on  $\mathrm{D}^3$  mainly comes from I2T.
In this paper, we bring the Described Object Detection (DOD) task to the foreground. For this task, we introduce a dataset called  $\mathbf{D}^3$ , which annotates described objects without omission and features flexible language expressions, whether long or short, complex or simple. Our evaluation of SOTA methods from REC or OVD on  $\mathbf{D}^3$  reveals challenges faced by REC, OVD, and bi-functional approaches. Based on these observations, we propose a baseline that largely improves REC methods for DOD task. We believe that the dataset and findings will contribute to advancing the understanding and development of DOD methods, facilitating future research in this area.
Limitation and broader impact. This work does have some limitations. Due to the significant annotation cost brought by our complete annotation process, we are unable to propose a huge dataset with millions or billions of images. Besides, the evaluation and findings in this work may be dependent on the choice of descriptions and the image sources. This work only serves as a starting point for DOD and we hope there will be other DOD datasets with larger scales. In the broader community, compared to traditional detection algorithms, DOD models have a lower customization threshold, enabling users to specify the detection target using language. This may lead to potential abuse.
Future work. During peer-review process, some new works with potential for DOD emerges, including Shikra [4], Kosmos-2 [30] and Qwen-VL [1]. We will continue to investigate such methods for DOD and update them in this list.
Acknowledgments. This work was supported in part by the National Natural Science Foundation of China under Grant 62076183, 61936014 and 61976159, in part by the Natural Science Foundation of Shanghai under Grant 20ZR1473500, in part by the Shanghai Science and Technology Innovation Action Project under Grant 20511100700 and 22511105300, in part by the Shanghai Municipal Science and Technology Major Project under Grant 2021SHZDZX0100, and in part by Hetao Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone (HZQB-KCZYZ-2021045). The authors would also like to thank the anonymous reviewers for their careful work and valuable suggestions.