DetGPT: Detect What You Need via Reasoning
Authors: Renjie Pi
Jiahui Gao
Shizhe Diao
Rui Pan
Hanze Dong
Jipeng Zhang
Lewei Yao
Jianhua Han
Hang Xu
Lingpeng Kong
Tong Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: embodied AI
multi-modal learning
object detection

Abstract: In recent years, the field of computer vision has seen significant advancements thanks to the development of large language models (LLMs). These models have enabled more effective and sophisticated interactions between humans and machines, paving the way for novel techniques that blur the lines between human and machine intelligence. In this paper, we introduce a new paradigm for object detection that we call reasoning-based object detection. Unlike conventional object detection methods that rely on specific object names, our approach enables users to interact with the system using natural language instructions, allowing for a higher level of interactivity. Our proposed method, called DetGPT, leverages state-of-the-art multi-modal models and open-vocabulary object detectors to perform reasoning within the context of the user’s instructions and the visual scene. This enables DetGPT to automatically locate the object of interest based on the user’s expressed desires, even if the object is not explicitly mentioned. For instance, if a user expresses a desire for a cold beverage, DetGPT can analyze the image, identify a fridge, and use its knowledge of typical fridge contents to locate the beverage. This flexibility makes our system applicable across a wide range of fields, from robotics and automation to autonomous driving. Overall, our proposed paradigm and DetGPT demonstrate the potential for more sophisticated and intuitive interactions between humans and machines. We hope that our proposed paradigm and approach will provide inspiration to the community and open the door to more interactive and versatile object detection systems.

DetGPT: Detect What You Need via Reasoning
Renjie Pi Jiahui Gao"; Shizhe Diao'; ; Rui Pan', Hanze Dong', Jipeng Zhang',
Lewei Yao .Jianuaa Han Hang Xu², Lingpeng Kong", Tong Zhang'
'Tee Hong Kong University of Science and Technology
The University of Hong Kong SShanghai Jiao Tong University
sdiaaa,, rpan, hdongaj, jzhanggr, LyyaakkJuust hk, sumi wemmieer@cnne..... hkunnnneet.hh....... hk, xbjxh@live com, hanjianhua201 2egmaaallllal22@llllll.. com. lpkêcs. hku, hk, tongzhhangetoogggaaggg.... org
Abstract
Recently, vision-language models (VLMs) such as GPT4 LLAVA, and MiniGPT4 have witnessed remarkable breakthroughs, which are great at generating image descriptions and vi- sual question answering. However, it is diffi- cult to apply them to an embodied agent for completing real-world tasks, such as grasping, since they can not localize the object of interest. this paper, we introduce a new task termed reasoning-ben based object detection, which aims at localizing the objects of interest in the vi- sual scene based on any human instructs. Our proposed method, called DetGPT, leverages instruction-tuned VLMs to perform reasoning and find the object of interest, followed by open- vocabulary object detector to localize these objects. DetGPT can automatically locate the object of interest based on the user's ex- pressed desires, even if the object is not explic- itly mentioned. This ability makes our system potentially applicable across wide range of fields, from robotics to autonomous driving. To facilitate research the proposed reasoming- based object detection, curate and open- source a benchmark named RD-Bench for in- struction tuning and evaluation. Overall, our proposed task and DetGPT demonstrate the po- tential for more sophisticated and intuitive in- teractions between humans and machines.
want to have
cold beverage.
Linear
Visual Encoder
aL
LLM
refrigerator
Detector
Figure 1: Framework of DetGPT. The VLM consisted of vision encoder and LLM interprets the user instruction, reasons over the visual scene, and finds objects matching the instruction. Then, the open- vocabulary detector localizes these objects base on the VLM's output.
exhibit human-like proficiency in complex reason- ing (Wei et al., 2022; Wang al., 2022b: Zhou et al., 2022; Zhang al., 2022; Diao et al., 2023b; Shum et al., 2023). Simultaneously, breakthroughs in the image and multimodal processing, as ex- emplified by vision-language models (VLMs) like GPT4 (OpenAI, 2023), LLAVA (Liu et al., 2023a) and MiniGPT-4 (Zhu et al., 2023), have empow- ered the LLMs with ability understand image inputs. These cutting-edge innovations are highly promising for a diverse array of applications across numerous fields.
1 Introduction
In recent years, the natural language processing field has seen remarkable advancements in the development of increasingly large language mod- els (LLMs). LLMs such as GPT-3 (Brown et al., 2020), Bloom (Scao et al., 2022), PaLM (Chowd- hery et al., 2022), Megatron- Turing-5308 (Smith et al., 2022), Chinchilla (Hoffmann et al., 2022), and others have expanded the horizons of language understanding and generation. These neural net- works, with hundreds of billions of parameters,
As highlighted by recent studies (Shah al., 2023; Brohan al., 2022: Fang et al., 2020), since intelligent robot heavily relies on interactions with humans, the field of embodied AI/ robotics is set to experience a significant transformation. With the
Equul Contribution. Code is available at https:// gi thub. http:///ttiaaaaaaeeeeeeeeeeeeeee

---

emergence of human-like intelligence of LLMs and VLMs, robots will be able to interpret hu- man instructions and reason over visual scenes, enabling them to execute corresponding actions. This breakthrough will lead to the creation of intel- ligent robots that are more helpful to humans, and opens up possibilities for various fields.
if the detection model is able to interpret human instruction, employ its own knowledge identify all objects of interest via reasoning, and finally localize them.
To this end, we propose a new research task: reasoning-based object detection. In essence, hu- mans provide abstract queries via natural language. then the model discerns and reasons which objects in the image may fulfill the query, subsequently detecting them. We made preliminary explorations in this direction. Specifically, we fine-tune a VLM (e.g., MiniGPT-4 (Zhu et al., 2023)) built on LLMs (e.g., Vicuna (Chiang et al., 2023)) to perform rea- soning and predict objects of interest based on user queries (instructions) and input images. We then provide the object names to open-vocabulary de- tector for specific location prediction. To facilitate future research in the direction of reasoning- based object detection, we curate a benchmark named RD-Bench containing 20000 images and around 120000 query-answer pairs, which will be open- sourced for the research community.
However, it is important to note that while VLMs have made remarkable progress in generating high- quality image descriptions, this alone is insufficient for robots to interact with the physical world ef- fectively. To achieve this, robots must be able to accurately identify and localize objects within vi- sual scenes, which is a vital prerequisite for per- forming actions such as "moving" and "grasping" objects. This goal of "localizing objects" is closely linked to the field of object detection, which is one of the most fundamental and extensively studied research areas in computer vision. Conventional ob- ject detection systems, such as Faster-RCNN (Ren al., 2015), Retina-Net (Lin al., 2017), and YOLO (Redmon al., 2016) can only detect fixed number of object categories, which restricts their practicality. Recently, a series of open-vocabulary detection (OVD) systems have emerged as the new trend (Gu et al., 2021; Li et al., 2022; Yao et al., 2022; Liu et al., 2023b). Specifically, those models adopt the contrastive learning approach to align the object-level visual features with the textual class embeddings extracted from a pretrained text en- coder (e.g., BERT (Devlin et al., 2019)). In this way, those models are able to detect a much wider range of objects during inference.
Related Work
2.1 Large Language Models
Recent months have witnessed a transition from encoder-based models (Lu et al., 2019; Devlin et al., 2019; Liu et al., 2019; Jiang et al., 2021; Gao et al., 2022) train seen significant progress in large language models. Models like GPT- 3 (Brown et al., 2020), Bloom (Scao et al., 2022), PaLM (Chowdhery al., 2022), megatron-turing- 530b (Smith et al., 2022), and Chinchilla (Hoff-
Despite the success achieved by OVD systems, they still require humans to provide specific object names, which is neither user-friendly nor realistic. Firstly, human users tend to provide high-level in- structions, which may not explicitly contain the object of interest. Secondly, the constraints of hu- man knowledge often hinders the users to provide the object names. For example, the user may wish to identify fruits with a high vitamin K content but lack the necessary expertise to determine which fruits fulfill this requirement. Finally, the range of object categories that humans can supply is intrin- sically finite and non-exhaustive. As an illustration, when attempting to detect "objects posing hazards autonomous vehicles," humans may only be able to enumerate a limited number of scenarios, such as compromised visibility or intricate pedestrian traffic patterns. In summary, it would be desirable
mann et al., 2022), have pushed the boundary of language understanding and generation to new fron- tiers. These massive neural networks have demon- strated human-level abilities in text classification, text generation, knowledge-intensive tasks, and even complex reasoning tasks. Recently, Meta's 's LLaMA (Touvron et al., 2023) provided series of powerful open-source models that boost language model research. For example, recent Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), and LMFlow (Diao et al., 2023a) have showcased the powerful capability of instruction-tuned LLaMA.
2.2 Vision Language Models
Given the success of language models, many fol- lowing research explored vision-language interac- tion, resulting in the development of various multi- modal models. The development in a number of

---

Thinking...
want have a cold beverage.
There is no visible beverage. Where can find it?
VLM
Oh! There is refrigerator that might contain cold beverage!
Figure 2: The illustration of reasoning- based object detection task. The detection system is able to interpret human instruction, reason about the visual scene with common sense knowledge, and finally localize the objects of interest.
language model research. Inspired by BERT-like encoder models, most of the multi-modal mod- els (Lu et al., 2019; Tan and Bansal, 2019; Chen et al., 2020; Li et al., 2021; Ding et al., 2021; Li et al., 2020; Ding et al., 2023, 2022) before 2021 are encoder-only Transformers, which are good at cross-modal understanding tasks. However, the transition from encoder-only models to decoder- based models in language model research inspires the pattern shift in multi-modal learning, including encoder-decoder models like VL-T5 (Cho al., 2021). OFA (Wang et al., 2022a), Da Vinci (Diao et al., 2023c), and decoder-only models like GPT- 4 (OpenAI, 2023). Most recently, we have wit- nessed the potential of multi-modal learning due to the powerful language abilities of LLaMA. Re- cent works include LLaVA (Liu et al., 2023a) and MiniGPT4 (Zhu et al., 2023). Unlike these works, our DetGPT focuses on localizing object of inter- ests based on user instruction, allowing for greater control over objects through language.
tectors are able recognize objects that are unseen during training based on their semanties. Despite the success of open vocabulary object detectors, their ability is still limited in the sense that they can only perform prediction given specific object phrases. On the other hand, our DetGPT enables reasoning and localizing objects of interest given any high-level user instructions.
3 Problem Statement
Recent vision language models (VLMs) backed with LLMs have shown promising results in visual understanding based on the visual scene and natural language input. However, they lack the ability for fine-grained visual understanding and precise local- ization, which makes them difficult be applied real-world scenarios involving embodied A1, such as robotics ad autonomous driving. On the other
hand, object detection is a crucial task in computer vision, which enables models to analyze images fine-grained manner and predict precise object locations. Unfortunately, existing detection meth- ods either can only predict fixed number of object classes, need exact object names for detection.
2.3 Object Detection
Object detection is of the most fundamental tasks in computer vision, which aims localiz- ing objects images. Traditional object detectors have a fixed number of classification heads, which makes them only capable of predicting the classes on which they are trained (Girshick, 2015; Ren et al., 2015; Lin et al., 2017; Yao et al., 2021a; Duan et al., 2019; Yao et al., 2021b; Zhu et al., 2020; Carion et al., 2020). Recently, open- vocabulayy object detection has attracted lot of attention (Gu al., 2021; Li et al., 2022; Liu et al., 2023b; Yao et al., 2022). The main philosophy is to utilize con- trastive training between object visual features and their class embeddings. such a way, object de-
To address these limitations, we propose a new task termed reasoning- object detection. In this task, users provide abstract queries using nat- ural language, and the model analyzes both the image and user input, reasons about which objects in the image may fulfill the user's goal, and finally detects their location in the image. For example, shown in Figure 2, when user requests "I want to have cold beverage," the model first analyzes the image of kitchen and determines that there is no "cold beverage" available. Then, identifies a refrigerator in the image and, based on the common sense knowledge stored the LLM, infers that the

---

BLIP2
LLAVA
MiniGPT44
DedPT
find whaa's intersiing about the image
when's shout the image.
Aid whst's menering about Bee srage.
Red whan's mneessinn shour imrge..
dhone looking window
The image slanea view cityccap from sindow The wandow made of glass and leo a white fiure: Thene are teilifings and cass visible in lie distance, The sky cloudyy thore some treee the silowatic Thene nod cup
The imige shows bid peiched window all. looking out unto a baey city smcet The hint appices to he a pigson. which a cornnce sigls. uthed aias. hittss preaenor: interatin becarse it provides glarroo arton coveormenee
where
pao find
window
winskowsill watching innertyy The hind is located closerio the leil side of the image, whill the con on the right sice. nocspying the edge of the window. This cuptiriiing mommet caplares die intenctioon hetween the noo uninal, showenning their cariosky and renuall instrcts somonds cach other
image
will living
Figure 3: Comparison with other SOTA VLMs. Our DetGPT is able to find and localize the object of interest.
refrigerator may store a cold beverage.
query (instruction) and the corresponding answer that contains both reasoning process and object names in the image that matches the query. For each image, we generate one detailed description followed by several instruction-answer pairs.
The proposed reasoning-baad based object detection task and DetGPT open up a world of possibili- ties for human- auman-machine interactions, which has the potential to greatly improve the capabilities of general purpose robots.
We reorganize the annotations such that each image is associated with the corresponding query- answer pairs. The detailed system prompt for our cross-mode object detection task is shown in Ta- ble 9 from the Appendix. To enable better annota- tion generation, we further manually design two in- context examples for querying ChatGPT, which are shown in Table 10 and Table 11 in the Appendix. Examples of generated detailed description and query-answer pairs are shown in Table 12.
4 Multi-Modal Query-Answer Instruction Data Generation
The traditional way for labelling a dataset requires a lot of human labor. Recently, LLMs such as ChatGPT possess superior generation capability, which can be used to replace human labeling with automatically generated annotations (Schick and Schütze, 2021; Ye et al., 2022a,b; Meng et al., 2022; Gao et al., 2023; Ye al., 2023). However, such text-only LLMs are not able to interpret visual in- puts, which hinders their practicality for data gen- cration based on images.
Method
5.1 Model Design
As an initial attempt towards reasoning-based ob- ject detection, we propose two-stage approach that first leverages the VLM to interpret the image and generate relevant objects names/phrases that match the user's instructions via reasoning; then we leverage an open-vocabulary object detector to localize the relevant objects given the results from the VLM. Specifically, for the VLM, we employ a pre- tre-trained visual encoder to extract image fea- tures, followed by a cross-modal alignment func- tion to map the image features to the text domain. Then, we utilize a pre-trained LLM as the knowl- edge brain interpret both the image features and human instructions, perform reasoning, and deter- mine target objects that help fulfill the given user query. Our framework is illustrated Figure 1. Inspired by (Zhu et al., 2023), we employ the
Motivated by LLAVA (Liu et al., 2023a), we leverage the images of pre-existing datasets (Lin al., 2014; Shao et al., 2019a), and employ two types of textual annotations to bridge the gap be- tween visual and textual representations: (1) Image Captions, which depict the visual content from dif- ferent viewpoints. (2) Objects categories, which are the objects present in the image. Specifi- cally, we adopt COCO (Lin et al., 2014) and Ob- jects365 (Shao et al., 2019a) datasets for construct- ing RD-Bench. Based on the given captions and objects, we design query-answer prompts in- struct ChatGPT to generate the following: (1) more detailed description of the scene, which gives ChatGPT itself a better sense of the visual scene; (2) query-answer pairs, which consist of a user

---

visual encoder of BLIP-2 (Li et al., 2023) as the vision encoder and utilize Vicuna (Chiang et al., 2023) or Robin (Diao et al., 2023a) as the lan- guage model to interpret both visual and text fea- tures. For the open-vocabulary detector, we lever- age Grounding-DINO (Liu et al., 2023b) to lo- calize the target objects in the image. Following MiniGPT-4 (Zhu et al., 2023), we train a linear pro- jection layer from scratch for the cross-modal align- ment, which has been proven effective in bridging the gap between vision and language modalities.
MInGPT 4
DetGIT
Fwant cald batelbeeeegp...
reoos cold beverae
The Toekirheeitcooeen moder and speditus cned with cabiretyaat bidt appliseess
prace infipoone wicch cut oue old bevezaee Therstor (arteerree
diris, and tha shver reterrrrroooo iroure turralaa The counts light yno guarss weltianniee tweck Abbace andiwhir phorogragh hanging theveal. treet of abu cing The hooets moce hurbwoed
adloof
spallling
and insiing Thewetus the snnwer (lldg deiss, orligeanns plonogruss hust baodl how"
Challenge One straightforward approach to im- plement our proposed framework is combining off- the-shelf VLMs with open vocabulary object detec- tors without further training. However, we observe that even though with carefully-chosen prompting, VLMs are able output objects in a specific pat- tern, they tend to output redundant objects that are either not shown in the image, or unrelated to the user's instruction (shown in Figure 4).
Figure 4: MiniGPT4 vs DetGPT with appended user prompt. The reasoning process and the final sentences containing target objects are both shown for clarity. MiniGPT4 generates redundant objects, while DetGPT can accurately recognize the object of interest.
comprehend fine-grained image features and derive the object of interest given the user's instruction. Instruction tuning empowers the model to iden- tify the objects of interest in the image. However, we find that the output format of the model often varies, which poses difficulty in extracting the rele- vant object names/phrases. Therefore, we design user prompt that is helps the model output the objects strictly in given format (shown in Table 1), which makes our model more stable. The final input sequence used to train the model is "###Hu- man: (Img) > ImageHere) > ( Img) { TeettHere > User_Prompt] )" Blue color represents the input image and Red color represents user instruction.
5.2 Training and Inference
Step 1. Image-Text Pretraining. We follow (Zhu et al., 2023) and leverage a combined dataset of SBU, LAION and Conceptual Caption to con- duct image-text pretraining. We minimize the lan- guage modeling loss:
C=
Σ
logp yt (( r"]
(1)
where F represents the multi-modal model. I' rep- resents the 2th image, and vf denotes the t token of the jth answer that belongs to the 2th image. L is the length of the answer. During the process, only the projection linear layer is trainable. while all other components are kept frozen. Since the vi- sual feature from BLIP-2 (Li et al., 2023) is already well aligned with textual features, tuning just the linear layer already suffices for aligning with a new language model such as Vicuna.
User Prompt
Answer with several sentences. End the an- swer by listing out target objects to my ques- tion strictly as follows: (Therefore the answer is: [object_names]).
User Prompt. We found that prompting is necessary for listing names of objects of interest in a consistent format, which makes DetGPT more stable.
Table I:
Step 2. Instruction Tuning. After the first step, although the model is able to generate detailed de- scriptions based on the image, it is not able to well interpret human instruction and derive objects that fulfill the user's goal. Therefore, we leverage a sub- set of our curated RD-Bench to fine-tune our model. Similar to the first step, the language modeling loss is optimized, and the linear projection layer made tunable. We demonstrate that the instruction tuning phase is a crucial step that empowers the VLM to
Inference. During inference, we first provide the model with a system prompt (show in Ap- pendix), which we find to be helpful in stabliz- ing the model's output. Then, we append the user prompt after the user's query. After obtaining the generated answer from the VLM, we extract the object names/phrases from it by matching the spe- cific output format, .ic. the object names following "Therefore the answer is: " Finally, we send the

---

names/phrases and the image the object detector for localization.
of false negatives (missed detections). Then, Av- erage Precision (AP) and Mcan Average Precision (mAP) are defined as:
6 Demonstration
MAP 1 N
We present the visualization results of DetGPT in Figure 5 and evaluate its capabilities. Interestingly. DetGPT exhibits the following appealing features: I) is proficient in common-sens reasoning based on the user's abstract query and the image; 2) can utilize the rich knowledge stored in LLMs that are beyond human common sense; 3) thanks to the abundant knowledge stored in the VLM, DetGPT generalizes to a broad range of objects that do not appear during the instruction tuning.
AP Σ P(r)) ((r))AR())
AP,
FER
where represents the set of recall values, AP can be interpreted as the area under the precision-recall curve. is the total number of classes, and AP; represents the average precision for class
Rather than deriving the mAP for all the objects in the image, we calculate only the objects that ful- fill the user's queries. Therefore, detecting objects that are irrelevant the user query will be counted as FP and decrease the final evaluation metric.
7 Experiments
There exist two major issues during evaluation: (1) the language model's output can not be guaran- teed to exactly match the object names in the bench- mark, even though they share the same meaning; (2) there may exist hierarchy among the categories, e.g., a stuffed animal is also toy. To address the above problems, we first leverage FastText (Joulin et al., 2016) calculate the similarities between the LLM- -LM--reddcted objects and all the class names in the benchmark (COCO and Object365 have 80 and 365 classes, respectively). Then, we take the top 1, top 5 and top 10 class names for each LLM- predicted object, and check if the ground truth class of the object is included. This approach makes our evaluation more robust for reasoning- -bsed object detection, since the words that share similar mean- ings and those that possess hierarchies tend to have higher similarities in their word embeddings.
We conduct first stage training paired image-text data to achieve vision-language alignment. After- wards, we conduct instruction tuning and evalua- tion on our curated RD-Bench. Specifically, randomly sample subset of 5000 images that are originally from COCO dataset and the correspond- ing query-answer pairs, which accounts for around 30000 query-answer pairs for instruction tuning. For evaluation, we sample (1) 1000 images that are originally from COCO dataset to evaluate Det- GPT's in domain (ID) performance, and (2) 1000 images from Object365 dataset, which are not seen by the model during training, to test its out of do- main (OOD) performance.
Training Details. For first stage training, the learning rate is set to 10-4, the batch size is set to 128, and the model is trained for 40000 steps. For instruction tuning, the learning rate is 3.5x 10-5, the batch size is set 32, and the model is trained for 40000 steps. We use adamW as the optimizer, with cosine learning rate scheduler. We use 8 A40 GPUs to conduct all experiments.
Main Results. As the first attempt in our pro- posed reasoning- object detection task, we conduct exhaustive experiments on different vari- ants of DetGPT using our curated RD-Bench. Specifically, study the effect of instruction tun- ing, system prompt and user prompt. As shown in Table 2, instruction tuning contributes the most performance gain, and prompts are able to further enhance the model's performance.
Evaluation. We conduct evaluation using the conventional metric for object detection, i.e., mean average precision (mAP), which quantifies how well the predicted bounding boxes overlap with the ground truth ones. Specifically, the precision and recall are defined as follows:
Varying the Language Model. We conduct ex- periments on different language model families with various model sizes. Specifically, Flan-T5 (Chung al., 2022), Vicuna (Chiang al., 2023) and Robin (Diao et al., 2023a). As demon- strated Table 3, Flan-T5 model family achieves inferior performance compared with Vicuna and Robin model families. This result verifies that the
adopt
TP
TP R- TP+FN FN
P TP+FF
where TP represents true positives (correctly de- tected objects), FP represents false positives (incor- rectly detected objects), FN represents the number

---

Find food that can relieve high blood pressure.
Find polential danger for humnans.
the image, there various fruits and vegetables present on the produce stand, including bananas, apples, and carrots. These fruits and vegetables are known to be high polassium, which can help lower blood pressure.
In the there is manhole cover on the ground, which can pose pocential danger for they accidentally step fall into it.
Why boy sad?
The boy in the image oppears to be because toy car has been damaged or broken. he was playing with the toy car before was damaged. and now is upset because cannot play with anymore.
likelyy that
Find items appropriate for a romantic dinner.
cold beverage.
There several items that appropriate for romantic dinner. including glasses, candles. flowers, and nice clothing. These items can create a romantic stmosphere
In the given image, there is refrigerator which be store cold beverages.
dinner special.
How enter the desert city "Gerudo Towa" where only allowed and
prohibited?
The depicts video gume "The Legend of Zelda: Breath Wild." this game, there city called Genado Town. where only women are allowed and men are probibited.
from the
find what's interesting about image.
this city for Link?
Find an item inappropriate for children.
The image shows bird perched
game "The Legend of Zelda: Breath of the Wild," there is a city called Town,
window sill, looking out onto busy city The bird appears be pigeon, which sight in urban areas. The bird's presence image interesting bocause it provides glimpse into life a wild animal living in environment.
only allowed and probibited. To enter this city, you way disguise yourself woman. One way do this to wear Genudo outfit, found locations throughout the
common
image that may insppropriate for children, such as cigarettes. Cigarettes can harmful children's 's health and lead addiction.
garette FUST
Figure 5: Demonstration of the reasoning process and generated bounding boxes of our DetGPT. Due to space limitation, we do not show the system prompt, the user prompt, or the final sentence "<Therefore the answer is: Sbbject_names>>> " for outputting the object names.

---

COCO Domain)
Objecr365 (Out Domain)
TUNE
SYS USER
MAP(1)
MAP(5)
MAP(10)
AvG
MAP(1)
MAP(5)
MAP(I0)
AVG
55.26
59.22
64.50
59.66
22.51
27.95
30.03
26.83
26.24
30.30
33,08
29.87
9.65
10.91
11.38
10.65
41.16
42.20
49.88
44.41
17.95
20.35
22.21
20.17
V
54.13
55.36
61.39
56.96
22.78
24.97
27.64
25.13
60.01
61.69
65.20
62.30
23.62
28.27
31.50
27.80
33.70
40.32
50.80
41.60
15.16
18.05
22.16
18.46
60.59
62.04
65.98
62.89
23.94
28.55
32.01
28.17
Table 2: Test Results on RD-Bench. Instruction tuning and prompting enable DetGPT to achieve promising performances on both domain and out of domain tasks.
MODEL
SIZE
MАP())
MAP(5)
MAP(10)
AVG
DATA
REASON
MAP(I)
MAP(5)
MAP(10)
AVG
2.7B
10.27
12.30
12.59
11.72
COCO
49.04
55.04
57.28
53.78
Han-1S
6.7B
12.53
13.94
15.02
13.83
(ID)
60.01
61.69
65,20
62,30
7B
59.45
60.19
63.68
61.11
Object365
17.17
20.79
26.75
21.57
Vicuna
13B
60.01
61.69
65.20
62.30
(OOD)
23.63
28.27
31,50
27.80
7B
56.32
60.21
62.73
59.75
Robia
13B
61.03
61.29
64.36
62.23
Table 5: Riffect of adding the reasoning process.
Table 3: Performance with different language models.
CoCo ID)
Sieer2d5 1000
MODEL PROMPT
MAP(I)
MAP(5)
MAP(10)
AVG
X
52.37
55.69
58.43
55.50
Vicuna
/ 60.59
62.04
65.98
53.19
55.31
59.06
55.83
Robin
61.03
Table 4: Adding prompts during instruction tuning.
08 SeTt www.actco.....
quality of the language model crucial for the promising performance DetGPT.
Figure 6: Average mAP for various sizes of instruction tuning datasets. Only a small number of samples are needed for reaching a promising performance.
Instruction Tuning with Prompting. As shown in Table 4, we observe that instruction tuning achieves better results if prompts are augmented to the queries during training. This implies that it is desirable to keep the input format consistent during instruction tuning and inference.
ples for instruction tuming. This verifies that the knowledge stored in the base model the key DetGPT's strong performance, and only small number of samples needed to empower the model to follow human instructions and output the objects of interest in a standard format.
The Impact of Reasoning. Before outputting the objects of interest, our DetGPT first performs reasoning by describing the image content, then using commonsense knowledge to decide which objects help fulfill the user's query. In Table we analyze the impact of such a reasoning process on the accuracy of detection. Specifically, if we train the model to directly output objects of interest without reasoning, significant performance drop be observed. This verifies that reasoning is not only desirable feature of DetGPT, but also key factor that helps it accurately derive the object of interest based on human instruction.
Conclusion
We propose new task termed reasoning- object detection, in which the model needs in- terpret high-level human instructions, reason over the visual scene, and finally localize the objects of interest. To facilitate future research in this task, we curate RD-Bench, a dataset that can be used for training and evaluation. Then, we design two- stage detection pipeline, named DetGPT, which demonstrates strong ability in open-ended tasks and achieves promising performance our pro- posed benchmark. We hope our work will pave the way for a more interactive and user-friendly object detection system, which will inspire later works on embodied AI, autonomous driving, and robotics.
The Impact of Instruction Tuning Size. From Figure 6, we observe that promising performance can already be achieved with around 10000 sam-

---

Find the eiderly person.
Find the plane.
The image depicts elderly person wearing face mask while walking down the street with a cane. A young man helping the elderly person.
In the image, there is pile of toy planes
plenga
Figure 7: Demonstration of failure cases. Top: even though the multi-modal is able to understand the visual scene and find the elderly lady, the object detector localizes both the young and the elderly person, and label them both as elderly person. This may be due to the detector is not able to distinguish "young man" from "elderly person". Bottom: there only one toy plane in the image, but the multi-modal model recognizes "a pile of toy planes". This may be caused by the multi-modal model's lack of fine-graines visual recognition capability.
9 Limitation
Furthermore, the introduction of RD-Bench as curated and open-source benchmark for instruc- tion tuning and evaluation can facilitate further re- search and development in this area, potentially leading to more advanced and versatile applica- tions of reasoning baased object detection.
As the first attempt towards a reasoning-basee ob- ject detection system, despite the promising results, DetGPT still has some limitations (shown in Fig- ure 7). Due to the two-stage nature of DetGPT, the weaknesses of both open-vocabulary detector and multi-modal models become the bottleneck. For ex- ample, observe that in some cases, though the multi-modal model is able find the relevant objects from the image, the open-vocabulary de- tector is not able to localize them, which may be because the training data of the object detector does not encompass such visual concepts. In some other cases, the multi-moda model is not able to find all relevant objects in the image, possibly due to the lack of fine-grained visual recognition ability. The above limitations promote new research in this direction and demand more advanced solutions.
Overall, the proposed task and method demon- strate step towards more sophisticated and intu- itive interactions between humans and machines. We do not foresee any ethical concern regarding our paper.
References
Anthony Brohan, Noah Brown, Justice Carbajal, Yev- gen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jas- mine Hsu, et al. 2022. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817.
10 Ethic Statement and Broader Impact
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901.
The proposed task of reasoning-based object de- tection and the proposed method, DetGPT, have the potential for significant broader impact across variety of fields. By enabling an embodied agent automatically locate objects of interest based on human instructions, DetGPT has the potential to improve the efficiency and effectiveness of tasks such as grasping in robotics and object recognition in autonomous driving. This could lead to safer and more reliable autonomous systems in these fields.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23- 28, 2020. Proceedings, Part 16, pages 213-229. Springer.

---

Yen-Chun Chen, Linjie Li. Licheng Yu, Ahmed El Kholy. Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: Universal image-text representation learning. In European Conference on Computer Vision (ECCV).
Xinpeng Ding, Nannan Wang. Shiwei Zhang, De Cheng, Xiaomeng Li, Ziyuan Huang, Mingqian Tang. and Xinbo Gao. 2021. Support-set based cross- supervision for video grounding. In Proceedings ohe the IEEE/CVF International Conference on Com- puter Vision, pages 1573-11582.
Wei-L.in Chiang, Zhuohan Li, Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Xinpeng Ding, Nannan Wang, Shiwei Zhang, Ziyuan Huang, Xiaomeng Li, Mingqian Tang, Tongliang Liu, and Xinho Gao. 2022. Exploring language hierarchy for video grounding. IEEE Transactions on Image Processing, 31:4693
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021. Unifying vision-and and-language tasks via text genera- tion. In Proceedings of the 38th International Con- ference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 1931-1942. PMLR.
Kaiwen Duan, Song Bai, Lingxi Xie. Honggang Qi Qingming Huang, and Qi Tian. 2019. Centernet: Keypoint triplets for object detection. In Proceed- ings of the IEEE/CVF international conference on computer vision, pages 6569-6578.
Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cowu Lu. 2020. Graspnet- lbillio:: A large-scale bench- mark for general object grasping. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11444-11453.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Jiahui Gao, Renjie Pi, LIN Yong, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang. Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. 2023. Self belf-guided noise-free data generation for efficient zero-shot learning. In The Fleventh International Conference on Learning Representations.
Hyung Won Chung, Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finen finetuned language models. arXiv preprint arXiv:2210 10. 11466.
Jiahui Gao, Hang Xu, Han Shi, Xiaozhe Ren, Philip L. H. Yu, Xiaodan Liang, Xin Jiang, and Zhenguo Li, 2022. Autobert-zero: Evolving bert backbone from scratch.
Jacob Devlin, Ming- Heig-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4188 Minneapolis, Minnesota. Association for Computational Linguistics.
Ross Girshick. 2015. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1448-1448.
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. 2021. Open-vocabularg object detection via vision and language knowledge distillation. arXiv preprint arXiv:2 134.23100
Shizhe Diao, Rui Pan, Hanze Dong, KaShun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. 2023a. Lmflow: An extensible toolkit for finetuning and inference of large foundation models. https:// optimalscale. gi ghub. io/LMFlow/.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Train- ing compute-optinal large language models. arXiv preprint arXiv:2203. 15556.
Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. 2023b. Active prompting with chain-of- thought for large language models. arXiv preprint arXiv:2302.1 12246.
Zihang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. 2021. Convbert: Im- proving bert with span- dynamic convolution.
Shizhe Diao, Wangchunshu Zhou, Xinsong Zhang, and Jawei Wang. 2023c. Write and paint: Generative vision-language models are unified modal learners. In The Eleventh International Conference Learn- ing Representations.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.011 01759.
Xinpeng Ding, Jianhua Han, Hang Xu, Wei Zhang. and Xiaomeng Li. 2023. Hilm-d: Towards high- resolution understanding in multimodal large lan- guage models for autonomous driving. arXiv preprint 2309:23000.0111
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models.

---

Junnan Li, Ramprasaath R Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. In Conference on Neural Information Processing Sys- tems (NewrPPS).
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779-788.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong. Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. 2022. Grounded language-inan Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 10965- 10975.
pre-training. In
Teven Le Scao, Angela Fan, Christopher Akiki, El- Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagne, Alexandra Sasha Luccioni. François Yvn., Matthias Galle, al. 2022. Bloom: 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211005/0
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020. Oscar: Object- semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision (ECCV).
Timo Schick and Hinrich Schüitze 2021. Generating datasets with pretrained language models. In Empiri- cal Methods in Natural Language Processing.
Dhruv Shah, Blažej Osiński, Sergey Levine, et al. 2023. Lm-nav: Robotic navigation with large pre-traines models of language, vision, and action. In Confer- ence Robot Learning, pages 492-504. PMLR.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng- Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. 2019a. Objects365: A large-scale. high-quality dataset for object detection. In Proceedings of the IEFF/CVE international conference on computer vi- sion, pages 8430-8439.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer.
KaShun Shum, Shizhe Diao, and Tong Zhang. 2023. Automatic prompt augmentation and selection with chain-of-thought from labeled data. arXiv preprint arXiv:2302.12822.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a. Visual instruction tuning.
Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li. Jianwei Yang. Hang Su, Jun Zhu, and Lei Zhang. 2023b. Grounding dino: Marrying dino with grounded pre-training for open- set object detection.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley. Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using deep- speed and megatron to train megatron-turine turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from trans- formers. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 5100-5111, Hong Kong, China. Association for Com- putational Linguistics.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision-and lad-lage tasks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pages 13-23.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An followwin Ilama model. https:// gi thub. wtt.:/oototttteeeee lab/ cttp://tttsulla//attaarraaaaaaaa ataaaa..aaaaaaaaaa...
Yu Meng, Jiaxin Huang. Yu Zhang, and Jiawei Han. 2022. Generating training data with language models: Towards zero-shot language understanding. arXiv preprint arXiv:2202.04588 04538..
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro,
OpenAI. 2023. Gpt-4 technical report.

---

Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
Least-to-mos prompting enables complex reason- ing in large language models. arXiv preprint arXiv:2205. 10625..
Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma. Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022a. Unifying architectures, tasks, and modalities through simple sequence- to-sequence learning framework. ArXiv preprint. abs/2202.03052.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2020. Deformable detr: De- formable transformers for end-to-end object detec- tion. arXiv preprint arXiv:2010.04199.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022b. Self-coonsitency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11177
A Data Generation Using ChatGPT
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 19933
We built our RD-Bench by utilizing images from established datasets like COCO and Object365, as well as the powerful LLM ChatGPT. However, as ChatGPT is designed to handle textual inputs ex- clusively, we employed caption and bounding box annotations connect the visual and textual modal- ities. In the case of Object365,, where caption anno- tations were available, employed LLa VA generate them with the prompt "Describe the image in detail". To showcase the data generation process, we provide both system prompts and contextual examples that were presented to ChatGPT.
Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang. Zhenguo Li, Chunjing Xu, and Hang Xu. 2022. Detelip: Dictionary-ent enricche visual- concept paralleled pre- pre-trainn training for open- world detec- tion. arXiv preprint
Lewei Yao, Renjie Pi, Hang Xu, Wei Zhang, Zhenguo Li, and Tong Zhang. 2021a. G-detkd: towards gen- eral distillation framework for object detectors via contrastive and semantic-guide feature imitation. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pages 3591-3600.
Lewei Yao, Renjie Pi, Hang Xu, Wei Zhang, Zhenguo Li, and Tong Zhang. 2021b. Joint detnads upgrade your detector with nas, pruning and dynamic distil- lation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10175-10184.
System Prompt Table displays the system prompt we utilized. In this prompt, we set the role for ChatGPT as a visual assistant to gener- ate query-answer pairs based on the given image. To ensure a comprehensive and diverse range of instructions, we delineated four specific types of queries that needed to be included during data gen- eration. These types include: 1) goal-oriented queries, which associate relevant objects with high- level user queries; 2) detection of all visible objects in the image, similar to conventional object detec- tion tasks; 3) detection of specific objects based on their categories; and 4) attribute-related queries, aimed at localizing objects with specific attributes such as color and shape.
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng. Zhiyong Wu, Tao Yu, and Lingpeng Kong, 2022a. Zerogen: Efficient zero-shot learning via dataset generation. In Empirical Methods in Natural Language Processing.
Jiacheng Ye. Jiahui Gao. Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong- 2022b. ProGen: Pro- gressive zero-shot dataset generation via in-context feedback. In Findings of the Association for Compu- tational Linguistics: EMNLP 2022.
Jiacheng Ye, Chengzu Li, Lingpeng Kong, and Tao Yu. 2023. Generating data for symbolic language with large language models.
In- Context Examples To enable better genera- tion quality, we append two manually written ex- amples after the system prompt as in-context ex- amples, which are shown in Table 10 and Table 11. Specifically, we we first list the captions and the objects, which are the inputs ChatGPT. Then we provide a detailed description and the query- answer pairs, which should be the outputs from ChatGPT.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompt- ing in large language models. arXiv preprint arXiv:2210.03493.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoe Le, and Ed Chi. 2022.

---

Generated Data from ChatGPT In Table 12, we showcase the imputs and outputs generated by ChatGPT using our proposed data generation pipeline. We have also included the corresponding images for clarity. is evident from the table that the query-answer pairs generated by ChatGPT are closely aligned with the objects present in the im- ages. This proves the practicality and effectiveness of our data generation pipeline.
all objects as the "foreground category" and calculate the AP in a class -lass-agnostic manner.
Comparison with ViperGPT We compare our DetGPT with the tool learning-based approach ViperGPT. It is worth noting that ViperGPT lacks support for the "localization" functionality and only generates textual answers. To address this limi- tation, we extracted the nouns from ViperGPT's answers and subsequently utilized GroundingDino for evaluation purposes. The resulting evaluation scores are presented in the Table below (Due to the requirement of an API key from ChatGPT, we limited our evaluation to subset of 100 samples to optimize cost efficiency)
System Prompts to DetGPT
We observe that adding system prompt before the imputs DetGPT helps stablizing the outputs. Specifically, we show the prompt in Table 13.
C Additional Experimets
Evaluation on OK-VQA We conduct an experi- ment on the mentioned OK- VK-VQA benchmark and compare the performance with other SOTA meth- ods in Table 7. The results show that our curated RD-Bench dataset indeed also helps boost the per- formance open-ended VQA tasks.
We conduct additional experiments to validate the superior performance our model on the reasoning- baased object detection task.
3)
GroundingDino DetGPT
10.4
34.89
66.34
68.53
70.26
67.22
Table 6: Performance comparison with GroundingDino. We demonstrate that our DetGPT is superior at reasoning-baad based object detection task.
BLiN-
UK VQA
30.6
36.4
63.9
583
Table
Performance comparison OK-VQA dataset.
TASK
TASK 2
TASK
TASK 4
ViperGPP
10.37
13.43
23.64
16.21
DeiGPT
58.96
59.02
61.63
60.32
Table 8: Performance comparison with ViperGPT.
Comparison with GroundingDino As shown in Table 6, we conduct experiments where di- rectly use the instructions as input prompts to open- vocabulary detectors, and we pick the bounding boxes by matching the object features with text embeddings of the instructions. In our proposed JOD-Bench, there are many high-level instructions that do not explicitly contain the object names, such "find all visible objects in the image". and "What should eat gain muscle?". In such cases, the obtained bounding boxes may not be associated with an object name/category. Therefore, we turn to more relaxed criterion for evaluation, where

---

System Prompt for Data Generation
You are an visual assistant, and you are seeing a single image. What you see are few sentences, which describe the same image you are looking at. Also, some ohjects that appear in the image are given to you. are responsible for the following: first, you need to describe the image contents with necessary but not redundant details. then, you need to design queries that can be completed with objects that are shown in the image.
For the query-answer generation, you should generate the following types of queries: 1. goal-oriented queries, where the user inputs something he/she wishes to achieve, and you are responsible to find the objects in the image that helps he/she to do so.
2. detect all objects in the image, where the user wishes to find all the salient objects contained in the image, this case, you need to find all of them.
3. detect some specific images, where the user wishes to find only subset of images, such as animals, foods, or electronics, .etc
4. find objects with certain attribute, such as color or shape.
You should generate as many queries as possible, but the related objects must be contained in the five sentences or the given objects. Also, the queries for an image should be diverse, spanning across all types of queries mentioned above.
Note that should design attribute-related queries (such as color or shape), only when you certain about it. Do generate such queries if the captions provided you does not contain such information.
You must response any queries answer in the following way: Query: (QUERY) Answer: (ANSWER) Therefore the answer is: (TARGET_ )BJETT
The objects (TARGET OBJETCTS) must be shown in the image, and be used solve the query in (QUERY).
When each query, you must describe all the object (you may refer to the complete object list), (2) based sense, correct object(s) "Therefore the answer is: (TARGET OBJETCTS)""
the question, list target objects the following manner:
Table 9: System Prompt provided to ChatGPT to generate cross-modal instruction data for reasoning object detection task.

---

In Context Example 1
Captions:
Three computer screens sitting desk with headphones attached. A desk that has few computer screens it. desk with computer, laptop, headphones and telephone office desk with laptop, headphones, keyboard and dual monitors.
home computer and keyboard desk.
Objects:
computer, monitor, keyboard, book, mouse, headphone, telephone
Description:
The image depicts desk with multiple computer sereens various accessories placed Specifically, there are three computer screens present desk, each of which has headphones attached them. In addition to sereons, there also laptop, kcyboard, and telephone present on the desk. The headphones appear he over-ear style and are connected the screens cords. The keyboard positioned front of the laptop, and screens.
telephone located to
Query and Answer:
1. I.Query: wish to watch movic. What can use? Answer: image, there are computer, monitor, keyboard, book, mouse, headphone and telephone. You can use computers and laptops to watch movies online. Therefore the answer is: (computer, monitor]
2. 2.Query: is late, and wish to read before going to bed. Answer: In the image, there are computer, monitor, keyboard book, mouse, headphone and telephone. If you wish to read, you could either read from online sources using the computer, or read from the book. Therefore the answer is: [computer, monitor, book]
3. Query:: Find all the computer, mouse, key board and cake in the image. Answer: In the image, there are computer, mouse and keyboard, but there is no presence of cake. Therefore the answer is: [computer, mouse, keybourd] Query: I need to call someone. What object can I use in the image? Answer: In the image, there are computer, monitor, keyboard, book, mouse, headphone and telephone. You can use the telephone on the desk to make phone calls. Therefore the answer is: [telephone]
5.Query: What can I use to listen to music? Answer: In the image, there are computer, monitor, keyboard, book, mouse, headphone and telephone. You can use the headphone on the desk to listen to music. Therefore the answer is: [headphonel
6.Query: Detect airplanes and computer in the image? Answer: In the image, there there are computer, monitor, keyboard, book, mouse, headphone and lelephone. Therefore, I can detect the computer in the image. However, no airplane can be found. Therefore the answer is: [computer]
7. 7.Query: Detect people and telephone in the image? Auswer: In the image. there are computer, monitor, keyboard, people can be found. Therefore the answer is: [telephone]
book. mouse, headphone and telephone. However, 8. S.Query: Is there any object that rectangular in shape. Answer: In the image,there computer, monitor, keyboard, book, mouse. headphone and telephone. The monitors and the laptop are rectangular in shape. Therefore the answer is: [monitor, laptop]
Query: Detect computer in the image? Answer: the image, there computer, monitor, keyboard, book, mouse, headphone and telephone. Therefore the answer is: [computer]
10.0 Query: Detect cake and computer in the image? Answer: In the image, there computer, monitor, keyboard, book, mouse, headphone and telephone. However, cake can be found. Therefore the answer is: [computer]
Table 10: In-context example.

---

In Context Example 2
Captions:
group people standing outside of black vehicle with various luggage. person holding tennis racket about
ball tennis court. group people playing tennis sunny day. ball clay court.
tennis player about
Two people playing tennis green with blue sky in the background.
Objects:
tennis racket, tennis ball, tennis court, net, shoes, water ahottle
Description:
The image shows tennis court, a net dividing in half. person standing one side court, holding tennis racket getting ready to serve ball. The court surface appears to be clay, and there are green trees visible the background. The person wearing tennis shoes has a water bottle next them.
Query and Answer:
I.Query: want to practice my forehand. What object can I use? Answer: In the image, there are tennis racket, tennis ball, tennis court, net, tennis shoes and water abottle. You can use the teanis racket and tennis ball to practice your forehand. Therefore the answer is: [tennis racket, tennis ball] 2. 2.Qury: What can use to hydrate myself during the game? Answer: In the image, there are tennis racket, tennis ball, tennis court, net, tennis shoes and water abottle. Using the water bottle present, you can hydrate yourself. Therefore the answer is: (water bottle]
3. Querry: Find all the objects that are used in a tennis game. Answer: In the image, there are tennis racket, tennis ball, tennis court, net, and tennis shoes, all of which are used in a tennis game. Therefore the answer is: [tennis racket, tennis hall, tennis court, net, tennis shoes] 4.Query want to play tennis with friend. Answer: In the image, there is a tennis court, tennis racket, and tennis ball present, which can use to play teanis with your friend. Therefore the answer is: [teanis court, tennis racket, tennis ball]
6. Query: Find all the objects that are round in shape. Answer: In the image, there are tennis racket, tennis ball, tennis court, net, tennis shoes and water abottle. Among them, the tennis ball is round in shape. Therefore the answer is: [tennis ball]
7.Query: Find the ice cream and the tennis racket. Answer: In the image, there tennis racket, tennis ball, tennis court, net, tennis shoes and water ahottle. However, there ice cream in the image. Therefore the answer is: [tennis racket]
Table 1I:
In-context example.

---

Generated Examples
Captions:
Some people a grass field flying kite in the sky. boy playing with kite in the park. colorful kitc flying by some people the park. and young girl holding kite green field. group people standing field fying colortul kite,
Objects:
chair, kite, backpack, person
Description:
The image shows a group of people on grassy field, flying a colorful kite sky. Some the people are standing while others are sitting on chairs. One woman and young girl are holding onto the kite while flies in the air. There are trees and other greenery visible in the background.
Query and Answer:
1. Query: want to fly kite. What object do need? Answer: In the image, there is a kite present, which you can use to fly a kite. Therefore the answer is: [kite]
Query: Find all the people the image. Answer: In the image, there are multiple people visible, standing and
sitting, while flying a kite. Therefore the unswer is: [person]
3. Query: want to sit while flying a kite. What object can I use? Answer: In the image. there are chairs present on the grassy field, which you can use to sit while flying a kite. Therefore the answer is: [chair]
4. Query: Find all the objects that are colorful, Answer: In the image, there are multiple objects visible, including the kite Hying in the sky. which is colorful. Therefore the answer is: [kite]
5. Query: Find all the objects that are used for carrying things. Answer: In the image, there is a backpack visible, which is used for carrying things. Therefore the answer is: [backpack]
Captions:
Two computers sitting on top of the desk.
Two computers large wooden computer desk. A desk with a keyboard, laptop and monitor.
A computer monitor next to keybnard, laptop and mouse.
A laptop with external keyboard, mouse, phone and photo desk.
Objects:
chair, person, cell phone, cup, laptop, mouse, keyboard
Deseription:
The image displays desk with various computer accessories There computers present the desk, which uppear have external keybourds attached to them. In addition the computers, there laptop, computer monitor, and mouse present the desk. phone visible as well, along with photo frame. The desk appears to other visible objects people the
be made of wood, and there
Query and Answer:
keyboard, person, chair, laptop, mouse, cup, cell phone, and TV. To make a phone call, use cell phone on the desk. Therefore the answer is: [cell phone] Query: Find
1. Query: How can make phone call? Answer: the image,
electronic devices the image. Answer: In image, there are keyboard, laptop, mouse, computer monitor, cell phone, and TV. of these are electronic devices. Therefore the Answer: is: (keyboard, laptop, mouse, computer monitor, cell phone, TV]
objects that can be used for typing. Answer: the image, there are keyboard, laptop. external keyboard, All these can used for typing. Therefore answer is: [keyboard, laptop, external keyhoard]
Query: Find
Query: Can you find any object that is black in color? Answer: In the image, there are keyboard, person, chair, laptop. mouse, cup. cell phone, and TV. The external keyboard and computer mouse are black color. Therefore the answer is: [external keyboard, mouse]
5. Query: Detect all the objects that are rectangular in shape. Answer: In the image. there are deyboard, laptop, mouse, computer monitor, cell phone, and T'V. Among them, the computer monitor and the laptop can be considered rectangular in shape. Therefore the answer is: [computer monitor, laptop)
Table 12:
Two examples to demonstrate the instruction-followwe data. The top section displays the prompts used to instruct text-only ChatGPT, which consists of captions and objects of the visual image. The bottom section shows the responses generated by ChatGPT.

---

System Prompt for DetGPT inference
You must strictly answer the question step by step: Step-I. describe the given image in detail. Step-2. find all the objects related Step-3. list out related objects existing in the image strictly follows: 'Therefore the answer is: [object names]
input, and concisely explain why these objects meet the requirement.
Complete all 3 steps detailed as possible. You finish the with complete sentence.
Table 13:
System Prompt provided DetGPT during inference. We found that prompting is necessary for listing names of object of interest in a consistent format, which makes DetGPT more stable.


Reviews:
[{"review_id": "1hhsYgHGXG", "reviewer": "Reviewer_5jA8", "summary": "", "questions": "A. Please provide a more detailed analysis of the dataset.\n\nB. Please discuss and provide results for the proposed method on existing benchmarks or compared to existing methods.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer 5jA8 for the review and are grateful for the time you spent with our submission. We are glad for the acknowledgment that our proposed task is interesting and can potentially be integrated into a robotic system. We wish to address your concerns by giving detailed responses to each of your comments as follows:\n\n**Q1. \"Detailed analysis of the dataset and a breakdown of the model performance\"**\n\nThank you for the great question, we agree that more analysis of dataset and sub-tasks is needed, and we provide them below. As defined in the system prompt, we have developed four specific task types. Below, we provide the definitions and examples for each task type as follows:\n\n```\nTASK 1: goal-oriented task,  to find the objects that can finish the user’s goal \nQuery: I want to increase my muscle mass.\nAnswer: Here are nuts on the table. Nuts are a good source of protein and contain all essential amino acids needed for muscle growth and repair.\n\nTASK 2: detect all objects in the image\nQuery: Find all the objects in the image.\nAnswer: In the image, there are donuts, nuts, oranges, a bowl and a knife.\n\nTASK 3. detect some specific objects \nQuery: Find all the knife in the image.\nAnswer: In the image,  a knife can be found. \n\nTASK 4. find objects with certain attribute, such as color or shape\nQuery:What is the object in the image that is yellow?\nAnswer:  In the image, there are oranges visible, which are yellow in color. \n```\n\nNotably, task 2, 3 and 4 are included to support direct queries, which are also supported by common object detectors. On the other hand, queries in task 1 contain high level user instructions, which typically aim at achieving a certain goal and do not explicitly mention the object. Among these tasks, task 1 requires common sense reasoning closely related to daily life, which is also the most challenging task. \n\n\nWe further break down the percentage of instructions for different tasks:\n|  Task 1 | Task 2  | Task 3 | Task 4 |\n|----------|----------|----------|----------|\n|   goal-orienated | all obejcts  | specific object|object w certain attribute |\n|   37%   |   15%   |   26%   |  22%   |\n\nThe instructions of the goal-oriented task has the majority number of query-answer pairs. In the following table, we further evaluate our model’s performances on each one of the four tasks separately. mAP(5) metric is reported. As can be observed, DetGPT is able to perform well across all tasks after task tuning and with appropriate prompts. Meanwhile, task 1 (goal-oriented task) shows slightly lower performance, which verifies that it is more challenging due to the need of reasoning.\n\n\n|  Tune  |   Sys  |  User  |  Task 1 | Task 2  | Task 3 | Task 4 | avg|\n|----------|----------|----------|----------|----------|----------|----------|----------|\n|   ✔️  |    |    |55.36   |   60.82   |   62.31   |  58.39   |  59.22    |\n|     |  ✔️  |    |16.70   |   29.63   |   35.54  |  39.36   |  30.30    |\n|     |     | ✔️   |34.63   |   41.17   |   43.25   |  49.75   |  42.20    |\n|   ✔️  |  ✔️  |    |54.30   |   58.69   |   59.21   |  49.24   |  55.36    |\n|   ✔️  |    |  ✔️  |59.13   |   62.96   |   66.83   |  57.84   |   61.69    |\n|      |  ✔️   | ✔️   |33.75   |  42.03   |   43.52  |  41.98  |   40.32   |\n|  ✔️   |  ✔️  | ✔️   |59.36   |   63.24   |   66.57   |  59.99   |   62.04    |\n\n\n**''What type of reasoning or commonsense knowledge is required.''** Our RD-Bench is based on common objects found in COCO images, including food, ​​vehicles, animals, kitchen tools and more. The current benchmark assesses the reasoning ability of utilizing common sense knowledge in daily life to help users achieve their goals, such as understanding tool usage and the nutritional components of various fruits. Surprisingly, we have discovered that DetGPT achieves satisfactory performance even with objects not explicitly mentioned in our instruction data. For instance, it can successfully identify cracks in walls when prompted to detect potential dangers to humans. This generalization ability is made possible by the rich knowledge contained in the LLM.\n\n**Q2: \"Evaluation on OK-VQA\"**\n\nWe agree that despite the absence of a baseline for the new task, the vision language model component of DetGPT can be evaluated on the VQA datasets. As suggested, we conduct an experiment on the mentioned OK-VQA benchmark and compare the performance with other SOTA methods in the following table. The results show that our curated RD-Bench dataset indeed also helps boost the performance on open-ended VQA tasks.\n\n|             |  Flamingo80B |  BLIP-2 OPT6.7B  | BLIP-2 FlanT5XXL | DetGPT |\n|----------|----------|----------|----------|----------|\n|OK-VQA |  50.6   |   36.4  |   45.9   |  58.5   |\n\nWe wish to note that our proposed reasoning based object detection task not only requires identifying objects of interest, but also requires accurately localizing them. We believe our curated RD-Bench will serve as a helpful tool for both training and evaluation of models that aim at addressing this task. \n\n**Q3: \"Comparison with ViperGPT\"**\n\nThanks for the suggestion. As suggested, we conducted additional experiments to compare our model with ViperGPT. It is worth noting that ViperGPT lacks support for the \"localization\" functionality and only generates textual answers. To address this limitation, we extracted the nouns from ViperGPT's answers and subsequently utilized GroundingDino for evaluation purposes. The resulting evaluation scores are presented in the table below (Due to the requirement of an API key from ChatGPT, we limited our evaluation to a subset of 100 samples to optimize cost efficiency. ):\n\n\n|             |  Task 1 | Task 2  | Task 3 | Task 4 |\n|----------|----------|----------|----------|----------|\n|             |   goal-orienated | all obejcts  | specific object|object w certain attribute |\n|      ViperGPT     |   10.37   |  13.43 | 23.64    |  16.21   |\n|      DetGPT      |   58.96   |   59.02  |   61.63  |  60.32   |\n\nWe find that **ViperGPT does not follow the logic of “exhaustively detect all objects in an image, then leverage LLMs to determine the objects of interest”**; instead, it first searches for objects words (nouns) that exist in the instruction, then locate the object patches in the image.  We list a few examples of generated code and the final textual result, which shows the drawbacks of ViperGPT:\n- For some abstract instructions, ViperGPT cannot correctly understand and find the object of interest, instead, it uses the abstract word “danger” for detection. (example 1)\n- ViperGPT cannot combine the visual information for correct reasoning, it often returns the objects that are irrelevant to this image. (example 2)\n \nExample 1 from ViperGPT:\n- Input image: an image of a building with cracks on the wall; \n- Input text: From the image, find the potential danger to residents.\n- Generated code:\n\n```\ndef execute_command(image):\n    image_patch = ImagePatch(image)\n    danger_patches = image_patch.find(\"danger\")\n    danger_patches.sort(key=lambda x: x.compute_depth())\n    return danger_patches[0]\nFinal result: None\n```\n \nExample 2 from ViperGPT:\n\n- Input image: an image of a kitchen with a refrigerator; \n- Input text:  Where can I find a cold beer from the image?\n- Generated code:\n\n```\ndef execute_command(image):\n    image_patch = ImagePatch(image)\n    beer_patches = image_patch.find(\"beer\")\n    cold_beer_patches = [patch for patch in beer_patches if patch.verify_property(\"beer”, “cold”)]\n    if cold_beer_patches:\n        return cold_beer_patches[0].simple_query(\"Where can I find it?\")\n    else:\n        return image_patch.llm_query(\"Where can I find a cold beer?\")\nFinal result: Most convenience stores, grocery stores, and liquor stores will sell cold beer. Additionally, many bars and restaurants will serve cold beer.\n```\n\nWe wish to clarify the following functions used in ViperGPT: \n- image_patch.find(object_name): detects the objects given their names; \n- image_patch.sort(key=lambda x: x.compute_depth()): sort the patches by their depths; \n- image_patch.verify_property(object_name, attribute) : determines whether the object satisfy the given property.\n\n\nWe wish to note that **even if the logic follows “exhaustively detect all objects in an image, then leverage LLMs to determine the objects of interest”, it is challenging to accurately identify the target objects.** The inherent limitation with this paradigm is that the open-vocabulary object detectors require the object names as inputs for bounding box prediction. \nAs an example, if the input image is an old building, the user instruction is \"find the potential danger to residents\", and the target object is “cracks on the wall”, it is questionable whether  “cracks on the wall” should be identified as a salient object in the first place. In order to truly extensively identify all objects, there must be a giant list containing all possible objects that may appear, and the detector needs to make predictions for each of them during inference, which is neither realistic nor practical. On the other hand, our DetGPT is able to flexibly detect the target objects via reasoning based on user instruction.\n\n**Q4: \"Theoretical contribution compared with MiniGPT4\"**\n\nThanks for your feedback. We would like to first emphasize that our primary theoretical contribution is to verify that, through the 2nd-stage instruction tuning, VLMs can accurately identify objects based on high-level user instructions.  In contrast, MiniGPT4 struggles to find objects of interest and tends to output redundant objects (shown in Figure 4).  After being combined with an open-vocabulary detector, DetGPT is able to accurately localize objects of interest given high-level user instructions.\n\nFurthermore, the objective of our paper is to introduce the novel task of reasoning-based object detection. This task represents a significant departure from previous research, and we sincerely appreciate your recognition of its interesting nature and potential in application. To facilitate future research in this direction, we curate a benchmark to help both training and evaluation.\n\n**Q5: Reproducibility**\n\nThanks for the feedback. We have included the prompts, and implementation details in our paper, and uploaded code as supplementary materials to this submission. We assure we will make the trained model, training/evaluation dataset, and codes publicly available upon acceptation.  As acknowledged by reviewer MiAP, our prompt details in the appendix help with reproducibility, and the RD-Bench dataset is a resource contribution to the community. With the released codes and datasets, it is easy to reproduce our results.\n\nOverall, many thanks for your insightful points and suggestions.  We have added the experiments accordingly. We hope our answers have addressed your concerns. If you have any further questions, we are happy to address them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The task of localizing objects in images based on abstract and high-level language descriptions is interesting. \n2. The proposed method can potentially be integrated into a robotic system, where relevant objects are detected based on user instructions or high-level task goals. \n3. The authors provide many qualitative examples that demonstrate the unique ability of the proposed method.", "reasons_to_reject": "1. Since the dataset is automatically generated, we do not know what type of reasoning or commonsense knowledge is required. The authors should provide a more detailed analysis of the dataset and a breakdown of the model performance based on different types of questions or different difficulties of the questions. \n2. It’s hard to judge the significance of the results because the evaluation does not involve any known methods or existing benchmark datasets. Since this paper introduces a new task, it’s possible that no baselines are directly applicable. However, the authors could compare individual components of the system to other existing methods. For example, for predicting object names based on images, the authors could compare to well-performing methods on standard benchmarks such as OK-VQA.\n3. The authors should provide more evidence on the advantage of the trained VLM over existing LLMs to can generate programs to compose vision-and-language models, such as ViperGPT and VISPROG. I imagine that these methods can first exhaustively detect all objects in an image and leverage LLMs to determine the object that is relevant to the user instruction. \n4. There is limited theoretical contribution since the network design and training procedure of the VLM model mainly follows previous work miniGPT4.", "questions_to_authors": "A. Please provide a more detailed analysis of the dataset.\n\nB. Please discuss and provide results for the proposed method on existing benchmarks or compared to existing methods.", "paper_topic_and_main_contributions": "This paper introduces a new task called reasoning-based object detection, which aims to localize objects in images given language instructions, especially high-level instructions. A framework DetGPT is introduced to solve this task by breaking it down to training a VLM to generate relevant object names based on image inputs and using a pre-trained open-vocabulary object detector to localize the objects based on the predicted object names. A dataset is introduced to train the model and evaluate its performance. To create the dataset, ChatGPT is used to generate additional textual descriptions and question-answering pairs for images in COCO and Object365.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yHKFb15e8Q", "reviewer": "Reviewer_MiAP", "summary": "", "questions": "Summarizing questions from above:  \n- A: How does a majority-class baseline perform?  \n- B: Does training on indirect queries reduce performance on direct queries?  \n- C: Did the authors experiment with chain-of-thought prompting?  \n- D: Suggestion – Did the authors experiment with multi-step chain-of-thought (Wei et al. 2022) reasoning? Allowing multiple steps for LLM reasoning could help boost performance.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer MiAP for the positive feedback as well as the valuable suggestions. We are also glad for the acknowledgment that the problem we are working on is an interesting and challenging research problem for the vision-language community. Below we would like to give detailed responses to each of your comments.\n\n**Q1.  ''How does a majority-class baseline perform? ''**\n\nThanks for the great advice. We believe by “majority-class”, you are referring to the classification datasets where images may contain only one salient object (e.g.,ImageNet). However, we wish to note that the images in COCO and Object365 datasets usually contain many ground truth objects, and the instructions in RD-Bench may be associated with any of those objects. For instance, the fridge example in Figure 5 contains a fridge, an oven, a range hood, a cabinet, and a sink. If the instruction “I want to have a beverage” is given, any predicted objects other than the fridge will be considered false positives and decrease the AP score. \n\nAs per your suggestion, we conduct the 2 following experiments on the COCO partition of RD-Bench:\n\n(1) We pick the major category for each image in the COCO dataset and input it to GroundingDino with a generic prompt. Specifically, we utilize CLIP to calculate the similarity score between each image and all 80 category names and pick the category with the largest score as the majority class. The results are as follows:\n|             |  Task 1 | Task 2  | Task 3 | Task 4 |\n|----------|----------|----------|----------|----------|\n|             |   goal-orienated | all obejcts  | specific object|object w certain attribute |\n|      GroundingDino     |   13.29   |  12.57 | 23.03    |  13.48   |\n|      DetGPT      | 59.36   |   63.24   |   66.57   |  59.99    |\n\nWe present the task definition and an example for each task below.\n```\nTASK 1: goal-orientated task,  to find the objects that can finish the user’s goal\nQuery: I want to increase my muscle mass.\nAnswer: Here are donuts and nuts on the table. Nuts are a good source of protein and contain all essential amino acids needed for muscle growth and repair.\n\n\nTASK 2: detect all objects in the image\nQuery: Find all the objects in the image.\nAnswer: In the image, there are donuts, nuts, oranges, a bowl and a knife.\n\n\nTASK 3. detect some specific objects \nQuery: Find all the knife in the image.\nAnswer: In the image,  a knife can be found. \n\n\nTASK 4. find objects with certain attribute, such as color or shape\nQuery: What is the object in the image that is yellow?\nAnswer:  In the image, there are oranges visible. Oranges are yellow. \n```\n\n\n\n(2) We conduct experiments where we directly use the instructions as input prompts to open-vocabulary detectors, and we pick the bounding boxes by matching the object features with text embeddings of the instructions. In our proposed IOD-Bench, there are many high-level instructions that do not explicitly contain the object names, such as “find all visible objects in the image”, and “What should I eat to gain muscle?”. In such cases, the obtained bounding boxes may not be associated with an object name/category. Therefore, we turn to a more relaxed criterion for evaluation, where we treat all objects as the “foreground category” and calculate the AP in a class-agnostic manner. \n\n\n|             |  Task 1 | Task 2  | Task 3 | Task 4 |\n|----------|----------|----------|----------|----------|\n|             |   goal-orienated | all obejcts  | specific object|object w certain attribute |\n|      Grounding Dino       |   10.04   |   29.79|  34.89    |  25.24   |\n|      DetGPT      |   66.34   |   68.53   |   70.26   |  67.22   |\n\n\nWe believe that the above experiments verify the challenges of our proposed reasoning-based object detection and the effectiveness of DetGPT. We will add those experiments to the updated version of our paper.\n\n**Q2. ''Does training on RDBench reduce performance on normal object-detection that does not involve indirect queries?''**\n\nThanks for the comment. We wish to clarify that when constructing the dataset, we do not only assume indirect queries, but also direct queries that are already supported by conventional object detection. As shown in “System Prompt for Data Generation”, we incorporate four types of tasks, namely goal-oriented tasks, detecting all visible objects, detect specific objects, and detect object with certain attributes (illustrated below). We believe you are referring to the  “detect specific objects”  task, which is also supported by regular open-vocabulary object detectors. To verify the effectiveness of DetGPT on the mentioned task, we compare it with GroundingDino and on the task and report the AP(5) . For the evaluation of open-vocabulary detectors, the object categories are provided with generic text prompts. We can see that the result is on par with directly using GroundingDino. \n\n\n|             |  DetGPT | Grounding Dino  |\n|----------|-----------|----------|\n|    direct query   (generic text prompts) |67.08   |   67.32|\n\n**Q3. \"Chain-of-thought prompting.  Did the authors experiment with multi-step chain-of-thought (Wei et al. 2022) reasoning? Allowing multiple steps for LLM reasoning could help boost performance. \"**\n\nWe appreciate the suggestion regarding the incorporation of multi-step chain-of-thought prompting. It's worth noting that we have carefully considered and implemented the multi-step chain-of-thought prompting in the DetGPT inference system, as outlined in Table 10. The results in Table 2 validate the suggestion that by incorporating cot prompting, we can achieve even greater enhancements in the performance of detgpt. To provide a clear illustration, we also include the results and the COT prompt in additional tables below:\n\n\nTest results on COCO (In Domain) :\n| COT Sys Prompt | Tune| User | mAP(1) | mAP(5) | mAP(10) | Avg |\n|------|-----|------|--------|--------|---------|-----|\n|          |  ✔️    | ✔️   | 60.01  | 61.69  | 65.20   | 62.30  |\n|  ✔️   | ✔️   | ✔️   | **60.59**  | **62.04**  | **65.98**   | **62.89**  |\n\n\n\n\nTest results on Object 365 (Out of Domain): \n| COT Sys Prompt | Tune | User | mAP(1) | mAP(5) | mAP(10) | Avg |\n|------|-----|------|--------|--------|---------|-----|\n|         |     ✔️| ✔️   | 23.62  | 28.27  | 31.50   | 27.80  |\n|  ✔️   | ✔️   | ✔️   | **23.94**  | **28.55**  | **32.01**   | **28.17**  |\n\n\nOur COT inference system prompt is:\n```\nYou must strictly answer the question step by step: \nStep-1. describe the given image in detail.\nStep-2. find all the objects related to user input, and concisely explain why these objects meet the requirement. \nStep-3. list out all related objects existing in the image strictly as follows: ⟨ Therefore the answer is: [object_names] ⟩.\n Complete all 3 steps as detailed as possible. You must finish the answer with a complete sentence\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ Overall, object-detection with indirect queries is an interesting research problem for the vision-language community. Most object detectors assume a predefined list of classes; even open-vocabulary detectors assume that targets are directly specified in the query. Dealing with ambiguous in-direct queries is a challenging problem for vision-language grounding. \n+ The experiments include a good set of sensitivity analyses and investigations with various SoTA models. Flan-TF, Vicuna, and Robin are studied extensively. The effectiveness of LLM reasoning, prompting, and dataset sizes are also studied.\n+ RDBench makes a resource contribution to the community. Particularly, the recently popular LLM+Robotics community might be interested in the dataset as a standard evaluation platform. \n+ Further details on the prompts used are provided in the appendix. This helps with reproducibility.", "reasons_to_reject": "- No majority-class baseline? In the qualitative examples in Figure 5, the target objects are already apparent and salient in the visual frame. Perhaps running a standard object-detector (MaskRCNN), or an open-vocabulary detector (MDETR) with generic prompts, and picking the largest/most-confident detection in the frame, would be a good baseline?\n- Does training on RDBench reduce performance on normal object-detection that does not involve indirect queries? For instance, would DetGPT struggle with a direct query like “the fridge”? This might be worth investigating because DetGPT’s formulation assumes all queries are ambiguous, but real-world interactions could involve a mix of both direct and indirect queries.   \n\nReferences:  \nWe et al. 2022 – https://arxiv.org/abs/2201.11903", "questions_to_authors": "Summarizing questions from above:  \n- A: How does a majority-class baseline perform?  \n- B: Does training on indirect queries reduce performance on direct queries?  \n- C: Did the authors experiment with chain-of-thought prompting?  \n- D: Suggestion – Did the authors experiment with multi-step chain-of-thought (Wei et al. 2022) reasoning? Allowing multiple steps for LLM reasoning could help boost performance.", "paper_topic_and_main_contributions": "This paper presents a new dataset (RDBench) and approach (DetGPT) for detecting objects with indirect queries. DetGPT resolves queries like “I want a cold beverage” by prompting an LLM and fine-tuning a VLM to find relevant objects such as a “fridge’’. Evaluations are conducted on RDBench test sets, which consist of both in-domain (COCO) and out-of-domain (Object365) images. The experiments investigate the effectiveness of different LLMs (Flan-T5, Vicuna, Robin), and the importance of LLM-based reasoning.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nSk0lyW4Sn", "reviewer": "Reviewer_Hwbg", "summary": "", "questions": "A. What is the computational cost for different types of architecture in your experiments? It seems larger models gain better results, yet the cost is not clear.\nB. Why do you choose ChatGPT instead of text-davinci-003/002 for data generation?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer Hwbg for the positive recommendation and we are grateful for the time you spent on our submission. Below we would like to give detailed responses to each of your comments.\n\n**Q1. \"A major concern is the annotation quality. An annotator requires multi-modal resources to spot the target object, and the image descriptions may miss important visual information. \"**\n\n\nThank you for pointing out this concern. We wish to clarify that for task tuning data generation,  we aim to generate query-answer pairs for the 2nd-stage instruction tuning, which helps the VLM learn to understand user instructions and detect objects of interest.  Since the query itself is generated to find the object of interest,  fine-grained visual information is not necessary for this purpose. More specifically, we provide the ChatGPT with \n- (1) list of object categories that exist in the image and\n- (2) image descriptions from different perspectives.  \n\nSince the query itself is most related to the object of interest, rather than the visual details of the image, the information we provide to ChatGPT is sufficient for generating query-answer pairs.  \n\nIn addition, we also want to note that we conduct vision-language alignment to equip the VLM with the ability to interpret detailed image features during the 1st-phase pretraining, where we use human-annotated image-caption datasets.  This can be verified as DetGPT achieves satisfactory performance even with objects that do not appear in our 2nd-stage instruction data. For instance, it can successfully identify cracks in walls when prompted to detect potential dangers to humans.  As we are unable to attach figures during rebuttal, we will include additional analysis in our final version.\n\n**Q2. ''Format issue: the content exceeds 8 pages (Figure 7)''**\n\nThanks for pointing out this issue. We apologize for the confusion.  Figure 7 belongs to the limitation section. We will attach it to the appendix in our final version. \n\n**Q3. ''What is the computational cost for different types of architecture in your experiments?''**\n\nWe compare the inference cost of DetGPT with different architectures in terms of frame per second (FPS). We wish to note that the inference cost is mainly introduced by the LLM. Our DetGPT is much faster than ViperGPT[2], which is a tool learning framework based on code generation.\n\n\n|DetGPT-FlanT5-2.7B|DetGPT-FlanT5-6.7B |DetGPT-Vicuna7b | DetGPT-Vicuna13b  | DetGPT-Robin7b  | DetGPT-Robin13b  | ViperGPT  |\n|----------|----------|----------|----------|----------|----------|----------|\n|   1.0 FPS |   0.5 FPS  |   0.4FPS |    0.2FPS  |   0.4FPS |  0.2FPS |  0.05FPS |\n\n\n**Q4. ''Why do you choose ChatGPT instead of text-davinci-003/002 for data generation?''**\n\nIn our study, we follow LLaVA [1] to use ChatGPT for data generation, which offers a reliable approach.  Besides, ChatGPT and text-davinci-003 are the advanced RLHF versions of text-davinci-002, providing more informative responses. **An advantage of using ChatGPT is its lower cost in comparison to text-davinci-003, making it a more cost-effective option for our research**.\n\n\n| model | cost | \n|----------|----------|\n| ChatGPT  | $0.0015 / 1K tokens   |\n|  text-davinci-003 | $0.0200 / 1K tokens | \n\n**Q5: Reproducibility**\n\nThanks for the feedback. We have include the prompts, implementation details in our paper, and upload code as supplementary materials to this submission. We assure we will make the trained model, training/evaluation dataset, and codes publicly available upon acceptation.  As acknowledged by MiAP, our prompt details in the appendix help with reproducibily, and the RD-Bench dataset is a resource contribution to the community. With the released codes and datasets, it is easy to reproduce our results.\n\n[1]LLaVA: Visual Instruction Tuning\n\n[2]ViperGPT: Visual Inference via Python Execution for Reasoning", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Well organized motivation and interesting setting.\n2. Comprehensive results with different language models as backbones.", "reasons_to_reject": "1. A major concern is the annotation quality. An annotator requires multi-modal resources to spot the target object, and the image descriptions may miss important visual information.\n2. Format issue: the content exceeds 8 pages (Figure 7)", "questions_to_authors": "A. What is the computational cost for different types of architecture in your experiments? It seems larger models gain better results, yet the cost is not clear.\nB. Why do you choose ChatGPT instead of text-davinci-003/002 for data generation?", "paper_topic_and_main_contributions": "The paper designs a task named reasoning-based object detection and introduces a method called DetGPT. Vision-language models (VLMs) \n are employed to interpret human instructions and reason about the visual scene, followed by an open-vocabulary object detector to localize the objects of interest.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Format issue: the content exceeds 8 pages (Figure 7)"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The new task of using reasoning for Indirect queries in the task of object detection is very interesting. The novelty of the task that the paper addresses is exciting. The authors have resolved many of the reviewers' questions with the rebuttal. I hope that these results will be properly reflected in the camera-ready version. Reviewer Hwbg makes a good point, but I don't think it is practical to cover the various data sets proposed by the reviewer. It is important that the paper is validated to be at least adequate to the claims made in the paper.

Pros:
The task is new and unique.
The paper is clearly written.
The results sound convincing.

Cons:
Many new results were presented during the rebuttal.
Decision: Accept-Main