Ins-DetCLIP: Aligning Detection Model to Follow Human-Language Instruction
Authors: Renjie Pi
Lewei Yao
Jianhua Han
Xiaodan Liang
Wei Zhang
Hang Xu
Venue: ICLR 2024 poster (2024)
Keywords: Multimodal learning
object detection
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: This paper introduces Instruction-oriented Object Detection (IOD), a new task that enhances human-computer interaction by enabling object detectors to understand user instructions and locate relevant objects. Unlike traditional open-vocabulary object detection tasks that rely on users providing a list of required category names, IOD requires models to comprehend natural-language instructions, contextual reasoning, and output the name and location of the desired categories. This poses fresh challenges for modern object detection systems. To develop an IOD system, we create a dataset called IOD-Bench, which consists of instruction-guided detections, along with specialized evaluation metrics. We leverage large-scale language models (LLMs) to generate a diverse set of instructions (8k+) based on existing public object detection datasets, covering a wide range of real-world scenarios. As an initial approach to the IOD task, we propose a model called Ins-DetCLIP. It harnesses the extensive knowledge within LLMs to empower the detector with instruction-following capabilities. Specifically, our Ins-DetCLIP employs a visual encoder (i.e., DetCLIP, an open-vocabulary detector) to extract object-level features. These features are then aligned with the input instructions using a cross-modal fusion module integrated into a pre-trained LLM. Experimental results conducted on IOD-Bench demonstrate that our model consistently outperforms baseline methods that directly combine LLMs with detection models. This research aims to pave the way for a more adaptable and versatile interaction paradigm in modern object detection systems, making a significant contribution to the field.

Published as a conference paper at ICLR 2024
INS- ALIGNING DETECTION MODEL TO FOL- LOW HUMAN-LANGUAGE INSTRUCTION
Renjie pi'; Lewei Yao', Jianhua Han?, Xiaodan Liang'. Wei Zhang',
Hang Xu²
Hong Kong University of Science and Technology Huawei Noah's Ark Lab 8 Sun Yat-sen Unversity and MBZUAI
ABSTRACT
This paper introduces Instruction-oriented Object Detection (IOD), new task that enhances human-compute interaction by enabling object detectors understand user instructions and locate relevant objects. Unlike traditional open- ppen-vocabulary object detection tasks that rely users providing a list of required category names, IOD requires models to comprehend natural-language instructions, contextual rea- soning, and output the name and location of the desired categories. This poses fresh challenges for modern object detection systems. To develop an IOD system, we create dataset called IOD-Bench, which consists of instruction-gun -uiioed detec- tions, along with specialized evaluation metrics. We leverage large-scale language models (LLMs) to generate a diverse set of instructions (8k+) based on existing public object detection datasets, covering wide range of real-world scenarios. As an initial approach to the IOD task, we propose model called Ins-DerCI.IP It harnesses the extensive knowledge within LLMs to empower the detector with instruction-folloiingg capabilities. Specifically, our Ins-DetCLIP employs a visual encoder (i.e., DetCLIP, an open-vocabulary detector) to extract object-level features. These features are then aligned with the input instructions using a cross-min moosll fu- sion module integrated into pre-tre treirraine LLM. Experimental results conducted on 1OD-Bench demonstrate that our model consistently outperforms baseline methods that directly combine LLMS with detection models. This research aims pave the way for a more adaptable and versatile interaction paradigm in modern object detection systems, making significant contribution to the field.
INTRODUCTION
Object detection serves as one of the most important tasks in the field of computer vision, with its influence spanning across numerous applications, including robotics (Bai al., 2020 Karaoguz & Jensfeli 2019 Zhang et al. 2019: Shridhar et al., (2022). autonomous driving (Liu et al. 2021a] Feng
ct al. 2021 Qian et al. 2022), and surveillance (Zhou et al.) 2019; Ullah et al. 202 I| Sanntooh et al.
2020). The recent advent of powerful multi-modal models, such as CLIP (Radford et al.) 2021), has ushered in a new era of open- vocabulary detection (OVD) algorithms (Gu et al. 2021b Zhong et al.,
2022 Li et al. 20 Yao et al. [022a Zhou et al. 2022). ) These models overcome the limitations of predefined class labels and enable recognition of a wide range of user- proviprovdded object classes. Despite the advancements, contemporary apen-vocabulary object detectors still have limitations in terms of human-computer interaction. Specifically, they rely on users providing a comprehensive list of object classes, which may not be practical in real-world scenarios. In such cases, it preferable for models to directly understand user instructions to identify and locate relevant objects. For example, a domestic robot should be able to recognize targets like "beverages, "drinks." and "cups" when instructed to "hand me a cup of drink". This observation motivates our pursuit of more flexible human-computer interaction paradigm tailored to modern object detection systems.
The advent of large-scale language models (LLMs) (Brown et al. 2020 OpenAL. 2023 Touvron
jt al. 2003)] Vicuna.] (203)) has revolutionized user interactions by enabling natural language- based communication with models. Inspired by this breakthrough, we aim to establish natural language interaction with object detectors. This paper introduces Instruction-orienter Object Detection (IOD),
"Equal Contribution.

---

Published as a conference paper at ICLR 2024
Taski
Task2
Task3
Task4
Identify all visible objects. Detect cup and wine glass.
Discover all the seafood.
Clean dishes after dinner.
Figure/ 1: Visualization of the constructed 4 types of instructions. The first task aims to identify all visible objects in the image; the second task queries specific object categories; the third task retrieves the objects that are under mutual parent class; the fourth task is to identify objects that are related to a daily life requirement.
a novel task that requires the model to understand user requirements and accurately localize relevant objects. IOD can be seen as an advanced extension of open-vocabulary object detection, where the detector must comprehend user-provided natural language instructions and infer the corresponding objects, surpassing the conventional approach which detect objects based solely on user-supplied class names. The IOD task entails more sophisticated reasoning and comprehension capabilities, introducing fresh challenges to the field of object detection.
To facilitate the proposed enaructiio-ooriented Object Detection (IOD) task, we address the lack of existing public datasets designed for this challenge. We create an instruction- -urided detection dataset termed TOD-Bench and develop corresponding evaluation metrics. Our approach involves leveraging the class names provided by the Objects365 dataset (Shao et al.] 2019) and utilizing a powerful LLM (OpenAL. 2022) generate instructions related to these classes. To ensure diverse and realistic set of instructions that cover various everyday scenarios, we design instructions with multiple purposes. These instructions can express specific goals that require the model to reason and detect relevant objects, or directly request the detection of specific categories. Through querying the LLM and applying post-processing filters, we curate approximately 8,000 instructions, allocating portion for training and the remaining for evaluation. To ensure accurate evaluations in cases where the model produces correct outputs for instruction-related classes, but those outputs do not precisely match the class names present in the Objects365 dataset, we modify the native evaluation procedure and address the potential issues that may arise.
Then, we introduce a novel model called Ins- DetCLIP that integrates state-of-ther thetttrrrt open-vocabulary detectors (OVD) with large-scale language models, specifically tailored for the Instruction Object Detection (IOD) task. The training is conducted in a two-phase manner. In the initial phase, we pre- train the OVD model following the training pipeline of DetCLIP((Ya0 et al., (20226). This pre-training enables the model to learn a visual encoder capable of extracting fine-grained visual features and accurately localizing objects, laying the foundation for robust object detection. In the subsequent phase, seamlessly integrate large-scale language model (LLM), which empowers the model with the ability to comprehend human instructions and perform instruction based object detection. The model is trained with our curated dataset to generate outputs for each object conditioned on the provided instruction. We design a simple yet effective mechanism where the model predicts the corresponding category name if the instruction matches the object, while predicting the word "other" otherwise. During inference, objects labeled as "other" are filtered out, and only the objects of interest, as guided by the instruction, are selected. To establish the effectiveness of our proposed method, we further design several baseline methods for comparison. For instance, we leverage the open-source source large-scale vision- language models, i.e., BLIP-2 (Li et al., 2023a).. MiniGPT4 (Zhu et al. 2023) or LLLAVA (Liu et al., 2023a), to identify object categories related to the instructions in images. Subsequently, we utilize OVD models such as DetCLIP (ao et al.) 2022b) to detect objects belonging to these identified categories.
To evaluate the performance of our proposed approach, we conduct comprehensive experiments by comparing it with the aforementioned baselines on our constructed dataset. Our Ins-DetCLIP demonstrates significant superiority over the baselines, e.g., obtaining the 10.0% average AP improvement on in-domain/out-domall instructions, showcasing its effectiveness and potential for IOD tasks. Furthermore, due to the exceptional generative capability, Ins DetCLIP is also able to achieve the state-of-the-art performance on dense captioning bonchmark al. 2016). This
2

---

Published as a conference paper at ICLR 2024
demonstrates the model's prowess in both dense prediction and generation tasks, further showeasing its versatility and effectiveness. We wish this work can pave the way for a more adaptable and versatile human-computer interaction paradigm in modern object detection systems.
2 RELATED WORK
Open-Voue Vocabulary Object Detection has recently provided pragmatic framework for recognizing objects of unlimited categories. Thanks to the success of vision-language pre- -reitriin contemporary
research (Zang et al.. 2022; Xie & Zheng 2021 Gu et al. [202 la Zhong et al. (2022) has put forth the idea of leveraging the wisdom from a pre- are-trained VL model (e.g.. CLIP (Radford et al. (021)) within detection context. Another promising approach utilizes broader dataset for training. For example, (Gao et al. 2021 Li al.] 2021b Fontanel et al. [022 Inkawhich et al.] [022] Yao et al.) 2022b) have leveraged casy-to-access image-text pairs to expand the training domain's ' coverage through pscudo labeling mechanism. Detclipv2( Yao et al. 2023) proposes a maximum word- region similarity between region proposals and textual words to guide the contrastive objective. 202T
XDETR (Cai et al.
2022) combines the conventional contrastive learning in VLP (Radford et al.)
Jia et al. | [022] to perform image-to-text alignment. Detic (Zhou et al. 2022), on the other hand, sets out to tackle a arge-vocabulary detection challenge by assigning classification labels directly to the maximum-sized region proposals. Based on DINO (Zhang et al,, 2022a Liu et al. (023b), OpenSeeD (Zhang et al., 2023) try to jointly learn from different segmentation and detection datasets. However, none of these methods consider detection based on human- -lnguaage instructions, c.g., "find some drinks on the desk", which is important for the future applications, like in robotics (Bai (annsfell[2019)....1 ct al. 2020 Karaoguz & Jensfelt 2019: Zhang et al. 2019 Shridhar et al. (022) and autonomous
driving (Liu et al. 2021a Feng et al. [022]] Qian et al. (0222).
Multi-Modality Large
Language Model. With the rise of text-based large language models (LLMs (Zeng et al. [022 Zhang al., [0226 Chowdhery et al. (022)), such as Flan (Chung et al., 2022). ChatGPT (OpenAL. 20022) and LLaMA (Touvron et 2023), many strategies have emerged
that fine-tune these models to align visual language by leveraging visual-text data. For example, approach to vision-language pre-training, making use of frozen pre-trained image encoders and LLMs with gated cross-attention.
Flamingo (Alayrac et al.) 2022) introduces computation-efficce
BLIP-2 (Li et al. (023b) propose the Q-former to efficiently align the visual and the text features with Flan-T5 (Chung et al. 2022) and OPT (Zhang et al. 2022b). Building on the groundwork of BLIP-2 (Li et al.] (023a), MiniGPT-4 (Zhu al. 2023) employs a more powerful LLM, named Vicuna (Vicuna, 2023). to achieve advanced multi-modal generation capabilities. InstructB (Dai al. 2023) further enables the image encoder extract different visual features given different instructions. In addition, LLaVA (Liu et al., 2023a) represents the pioneering effort to utilize the language-only GPT-4 (OpenAL. 2023) model to generate multi-modal data. However, none of these work considers predicting precise location (i.c., bounding-boo coordinates) for each concepts explicitly or implicitly mentioned in the instruction.
IOD-BENCH: AN INSTRUCTION- OBJECT DETECTION DATASET
To facilitate Instraction-oriented Object Detection (IOD) task, make the first effort establish a task-specific dataset named IOD-Bench and corresponding evaluation metrics tailored to effectively address the challenges inherent in this novel task.
GPT-assisted Instruction Detection Data Generation. The community has witnessed the flourishing development in the field of object detection, giving rise to numerous high-quality public datasets, such as COCO (Lin et al.] 2014a) and Objects365 (Shao et al. 2019). However, there still lacks datasets specifically tailored for instruction detection, partially due to the task's ambiguous definition and the high cost of manual annotations. Inspired by the remarkable achievements of recent GPT
models (Brown et al. 2020 OpenAL. 2023 2022) in natural language understanding and reasoning. propose utilizing the capabilities of large-sse -cage language models (OpenAL. 2022) to generate an cntrructioooorientte Object Detection dataset based on existing public object detection datasets, thereby circumventing the laborious manual annotation process.
3

---

Published as a conference paper at ICLR 2024
Instruction type 1
Instruction type 3
1. Identify every object present in the image.
1. Discover the kitchenware.
2. Spot and label all items in the scene.
Gather all the stationery objects.
3. Locate and classify all objects visible in the picture.
3. Identify all the vehicles for tramsportation.
4. Recognize name every item captured in the image.
4. Discover all the electronics.
5. Analyze the scene and identify each object within it.
Find all the sports accessories.
6.
Instruction type 2
Instruction type 4
Identify {obj3 present the image.
1. Prepare healthy salad for lunch.
2. Assess the image to reveal the fobj] that may be depicted.
Pack the necessary items for day trip.
3. Analyze the image to find any (obj] that may be present.
Create still-life drawing.
4. In the image, identify [obj] that are present.
4. Clean the spill on the kitchen floor.
5. Look at the image and find the {obj) it contains.
5. Put out a fire quickly and efficiently.
6.
6.
Figure 2: Examples of constructed 4 types of instructions.
Specifically, we provide the category names from Objects365 to the LLM (OpenAL. 2022 and request it select several class names from the given list to construct instructions while returning the target class names involved in each instruction. Since Objects365 itself contains bounding box annotations for each class, this approach enables us obtain dataset with both instructions and corresponding bounding box annotations for the targets. To ensure diverse instructions that cater to the varied needs of everyday scenarios, we consider 4 types of instructions:
1. Detect All Categories: Instructions that requires detecting all objects within an image.
2. Detect Some Categories: Instructions that mandate the detection of user-specified objects (similar to the current open-vocabulary object detection tasks).
3. Detect Categories Belonging to a Super-category: Instructions that request the detection of all subelass targets associated with particular parent class.
4. Detect Objects that Can Achieve Certain Goals: Instructions that express a specific daily life requirement and demand the detection of related objects.
For generating the above four types of instructions, we devise corresponding prompts to query ChatGPT. Please refer to the Appendix for detailed prompts.
Model Forgetting Issue. During the generation process of the 4th type of instruction, observed that providing CharGPT with all 365 class names simultaneously presents a considerable challenge for the model to remember them all. This issue could cause the model produce class names related to the instructions, which unfortunately, are not present in Objects365. In addition, it tends to overuse certain class names, thereby leading to the generation of numerous similar instructions. To alleviate the difficulty of memorizing the object name list and increase instruction diversity, we opt for sampling 30-40 class names to form an object name batch for each query process. We design sampling strategy that makes it more likely for categories belonging to the same parent class being sampled collectively (as they are more apt to serve identical instructions). Additionally, we employ a class- bass-balanced sampling for the class names across batches, ensuring that the generated instructions cover a balanced range of categories.
Data Post-pnn pest-processig. Despite adopting the described strategy, the generated instructions still contained categories not covered by Objects365. We filter away these extraneous categories and remove the duplicated instructions. Finally, we obtained approximately 8,000 instructions in total and their corresponding related class names across the four distinct instruction types. We allocated 80% of these instructions for training, while the remaining reserved for evaluation.
Dataset Construction and Partition. To construct the dataset, for each task, we first randomly sample an object category that exists each image, then sample an instruction that is related to the category, which establishes the image-instruction pairs. Afterwards, for each image-instruction pair, we keep all the annotations of the positive objects that match the instruction and abandon the unmatched object annotations. Since only these retained annotations treated the ground truth annotations for the image-instruction pairs, the AP will be decreased if objects that are irrelevant to the instructions are predicted.
4

---

Published as a conference paper at ICLR 2024
To examine the generalization ability user instructions, adopt rigorous approach in splitting the instructions for each task into 8:2 subsets, where 80% the instructions are allocated as the in-domain set and the remaining 20% are reserved as the out-of-domain set. The training dataset is constructed using the in-domain instructions and training images from object365, while the validation set is constructed using 20,000 images from the validation split of Object365, paired with both in-domain and out-of-domain instructions.
Evaluation Metrics. During evaluation, since the LLM-geenraaee answers may not be exactly matching to the category names in the Object365 dataset, we use BERT (Devlin et al. 2018) to extract the word embeddings of both the generated words and the ground truth categories, then calculate their similarity score and map the generated answers to ground truth by selecting the category with highest similarity. If all the scores are lower than threshold (0.4), the prediction will be treated as false positive.
4 METHOD
Phase I: DetCLIP Pre-Training
Phase 2: Ins-DeiCLIP TraCCLPTTraaiinn
Pre- Trained LLM
Visual Encoder
Reg Center loss
Fezal Forward
Grounding Data
Fine-grained Cuntrastive loss
Crass Aliention
Detection Data
Self Attention
Image-Text Pairs
Eacoder
Text
Encoder
Visual $
Glases
Instruction: Find all equipments
serve for drink.
The overall framework of Ins-DetCLIP. The training conducted in 2 phases. The first phase resembles the pre-training of open- vece-oocyyl object detectors, where we adopt training data from detection, grounding and captioning tasks. In the second phase, we empower the object detector with the ability to follow human instructions by introducing large language model into the model. The model is trained on our IOD-Bench to predict object category names only for those objects that match the instruction.
Figure 3:
Based on the proposed dataset, we present a novel object detection system named Ins-DerCLIP. Our model promotes the ability to detect any objects when provided with human language instructions. This capability is made possible by the integration of two key components: 1) the powerful open- vocabulary detectors as the visual encoder to provide fine-graines visual features; 2) the LLM with vast amount of knowledge to interpret human instruction and generate corresponding output. this section, will provide a detailed overview of our proposed method. We begin by outlining the design of our model and follow up with a comprchensive explanation of our training scheme.
4.1 MODEL DESIGN
Visual Encoder.
The visual encoder is open- vocebuoca object detector pretrained using combined datasets from detection, grounding and captioning tasks following Yao et al. (2022) 2023). Different from the encoders used in previous multimodal models that extracts image-leves features (Zhu et al. 2023] Alayrac et al., 2022 Li et al., (2023a), the encoder of Ins-DetCLIP is not only able to extract fine- fine-grained features object level, but also locate the objects the image. In addition, text-aligned object visual features facilitate instruct tuning in the later phase.
Object-level Cross Modal Fusion. To enable the interpretation and the ability to follow human instructions, we resort to the pre- trained large language models (LLM). Specifically, we primarily adopt the FlanT5 (Chung et al. 2022) model family as the candidates for the language decoders due to their superior instruction-folle following ability. We fuse visual and language features by inserting cross-attention layers among the self-attention layers of the language model's 's decoder, which are randomly initialized and trained from scratch. To ensure stable training at the beginning, adopt tanh-gating mechanism following Alayrac et al. (2022), which can be formulated as follows:
hL = (Lan(((() x Alln( AAln(AAln((h) (lr.)))r) V) + Alin(hr.)
(1)
where FF, Attt and Attn are feed forward network, cross attention layer and self attention layer, respectively. by is the hidden states I. th layer, and V is the object visual feature. is learnable
5

---

Published as a conference paper at ICLR 2024
scalar initialized to O. In this way, the output of LLM remains unchanged in the beginning, and the visual features can be gradually integrated with the language model without disrupting the outputs.
4.2 TWO-PHASE TRAINING SCIEME
Phase 1: Open-vocabulary Pretraining. In the first phase, we conduct OVD pretraining following DetCLIP (Yao et al. (202bb). As show in Phase 1 of Fig. Fl we pre-train the detector by optimizing the fine-graing contrastive loss between text embeddings and object-level visual embeddings, along with the centerness loss and bounding box regression loss. We utilize datasets from detection, grounding and image-text pairs for conducting visual-textual feature alignment. After the first phase, we derive a powerful open-vocabulary object detector which is able to extract visual embeddings well aligned with textual embedding from the pre-trained CLIP (Radford al. 202T) text encoder.
Object Feature Extraction. After the first stage, the DetCLIP model is able to propose bounding boxes given a list of category names. However, in our proposed IOD task, only open-ended user instructions are provided. To address this issue, we introduce an extra classification head that distinguishes foreground objects from the background a class-agnostic manner, resembling region proposal network (RPN). The selected regional features are then treated as object features and provided to the LLM, which then predict the categories in a generative manner.
Phase 2: Object-Level Instruction Tuning. During the second phase, we leverage the training data in our IOD-Bench to perform instruction tuning. Specifically, we first freeze both the visual encoder obtained from the first stage and the pre-trained language model (Phase 2 of Fig. 3) To enable cross- modality fusion. Then, we insert randomly initialized cross attention layers within the decoder layers of the language model and train them from scratch. The image is processed through the visual encoder, which extracts object-level visual features. Simultaneously. the accompanying textual instruction is passed through the language model. Cross attention is performed between the object-level visual features and the textual features. Finally, the language modeling loss is optimized on the language model's ' outputs, which can be formulated as follows:
Linsmmet = logplvv ((t( V*]]
(2)
where ø denotes the LLM, V is the object visual feature, yi is the textual token associated with zth object at tth time step.
Target for Autoregressive Training. The main philosophy behind instruct- following behavior is to enable the detector to localize only the objects of interest. We put forth simple yet effective strategy that helps achieve this goal: In cases where the object deemed relevant to the instruction (positive), the target output for the language model is set to be the object's s category name. Conversely, if the object is deemed to be unrelated to the instruction (negative), the target output is set to "other". For every training step, randomly sample instruction for each object feature, and set the probability of sampling negative instructions to fixed constant Pneg- During inference, we can simply filter away objects that associated with the output "other".
Parallel Training Formulation. After extracting object-level visual features from the image using DetCLIP, there are ways train the model to achieve our goal. The first approach is to concatenate the object features and train the language model to sequentially predict the output for each object. However, this approach will output an extremely long sequence if the number of objects is large. Alternatively, we can allow the language model to process each object feature independently. Specifically, each object feature interacts with the corresponding instruction via the LLM, then the LLM only predicts the output for that object. In this way, the prediction of objects can be parallelized, which promotes efficiency during inference.
EXPERIMENTS
5.1
IMPLEMENTATION DETAILS
Model Architecture. For visual encoder, we adopt the same architecture as DetCLIP Yao et al.
2022a). Specifically, use ATSS architecture (Zhang et al., (020)) with the same swin-Ti [Lu et al.
2021b) backbone. For the LLM, we mainly adopt FlanTs model family (Chung et al. 2022). We
conduct experiments both F- and T-

---

Published as a conference paper at ICLR 2024
Table 1:
Performance comparison between our Ins-DeiCLIP and two-stage counterparts on the validation set of IOD-Bench. Our method outperforms the baselines by large margin.
MLLM
DETECTOR
IN DOMAIN
OUT oF DOMATN
AP
TTAKS// (TAS4))
TTASK2 (TAS4)
2023A
3.14
3.06
(105)
BLIP2-PLANTS--I ITTY MIZAA
DETCLIP
3.95 (5.37 /4.51
/2.2/2.89)
3.71 (5.21
/4.29 2.43)
MINGPTA-VICONNA
MINIGPT4 VICUNNA- 70 7A0 Aur
2023
DETCLIP
8.29 (12.3 10.1 77.51
/3005)
6.35 (10.4 70004/7.13/.22
2.57)
MINIPT4-VVCCNNE 77IEE
2077
GROUNDIGDDIO
8.22 7/ 10.3
7 29/11.0)
6.34 (10.6 7.331.666.22211
/ 2.39)
LLAVA VICUNNA-78 LIVEL
GROUNDINGDINO
(14.5/
(14.2) /6.02 3.94)
DETCLIE--PPOO OPT1.3
(22.9
DE--DEECLP--LANTE PLANTSS-ASE
DETCLIP
15.3 (24.5
11.3313/1000
13.7 (24.2) 16.0/8.62/
5.900
Ixs INs-DEYCLIP-ER PLANLIP--LAT-AAAAA
DETCLIP
16.2 (25 11.4/1117/11.0
14.4 (25.41 16.2/9.63 6.500)
Taskl
Instruction: Detect all visible objects.
Instruction: Locate all objjcts in the image
Instruction: List abjects within the imagr-
Instruction: Detect
everything in the imago.
Task2
Instruction:
caancd, blender from the scene.
Instruction: Find Following objects: person, tablet, marker.
Instraction: in the picture.
objects: bread, spoen.
Task3
Instruction: items in kitchen.
Instruction:
Instruction: Find humans in image.
Instruction: items in kitchen.
furniture.
Task4
Instruction: Eat the
Instruction: Clean
Instruction: Watch
Instruction: Citt up
your Figure 4: Visualization of bounding box results generated by Ins-DetCLIP given different instructions. Each row from top to bottom are results of one task. The text below shows the instructions associated with the given image. We observe that Ins-DeiCLIP is able respond differently according to different user instruction and detect the objects of interest.
soup spuon,
Training. Our open-vocabulary pre-training phase follows the training settings of DetCLIP, utilizing the Objects365 and Goldg datasets. We employ a cosine decay learning rate, starting from 2.8e -4 and conduct the training over 12 epochs using GPUs. For the instruction tuning phase, if otherwise specified, we set P.......... Pang to 0.1, the cross attention fusion layers are inserted into the language model's decoder every 3rd layer. The models are trained for 12 epochs. The initial learning rate is set 2.5e and is decayed by a factor of 0. the 8-th and the 11 epoch. The max token length set to 512 following (Chung et al. 2022).
5.2 BASELINE MODELS
We establish baseline methods for comparison with several powerful multimodal models, such as BLIP2 (Li et al. 2023a) and MiniGPT4 (Zhu et al., 2023) variants. To achieve Ietruction-oriemnt detection, we directly integrate those models with open- vece-vocabllary detectors (OVD). Specifically, detection is performed in a two-stage manner: we first ask the multimodal models to elicit the names of objects in the image that are relevant to the instruction, then those names prompts query
7

---

Published as a conference paper at ICLR 2024
the OVD model. For fair comparison, finetune the LLM component of the baselines using the instruction-object text pairs of the in domain partition.
5.3 INSTRUCTION DETECTION
We demonstrate the effectiveness of our model and our training strategy by comparing with the baselines on our IOD-Benee As shown in Table our Ins DesDee outperforms the counterparts by large margin. Specifically, using FlanTT5bbase model, Ins De-C-eP already outperforms MiniGPT4 baseline by 7.9 mAP averaged across all tasks. Thanks to the generalization ability of the L.MMs, our model also demonstrates promising performances on instructs that do not appear during training. For instance, the Ins-DerCIIP with Flan'T5-bass" still achieves 13.7 mAP on average for out of domain instructs, which is only slightly lower compared with results on domain instructs by 1.6 mAP.
Discussion:
The two-stage approaches in Table I [utilize multimodal LLMs like BLIP2, MINIGPT4 and LLAVA to identify objects, then employ open-vocabulary detectors for localization. However, MLLMs, trained with image-level features instead of object-level features, often struggle to recognize all related objects in the image. Additionally, these methods typically employ low-resolution images, impeding the recognition of small objects. In contrast, model not only harnesses the superior capabilities of L.LMs, but also utilizes the rich object-level visual features from DetCLIP, which provide more detailed information.
5.4 DENSE CAPTIONING RESULTS
Leveraging the exceptional generation capability of LLM,, our Ins-DeiCI.IP can go beyond predicting only the category name, and also generate detailed description for objects interest. We demon- strate the superior generation ability of our model by benchmarking it on dense captioning tasks. To ensure fair comparison, we fine-tune the regression head of Ins-DetCLIP using box annotations from model consistently outperforms other methods
dense captioning datasets. As shown in Table and establishes new state-of- mate-of-thhear results. We also visualizaition results the appendix.
Table 2: Performance on dense captioning benchmarks. Owing to the great generation ability of LLMs, Ins-DetCLIP achieves state-of-the-art performances both VG 1.2 and VG COCO datasets.
METHOD
VG VI.2 MAP(%)
METHOD
VG COCO MAP(%)
FCLN (JONNSON ET AI
2016
5.16
FCL.
4.23
JIVC YANG ET AL 2017
9.96
JIVC
7.85
COCD LI AL. 2019
9.75
COCD
7.92
CAG-NET (YN ET AL.
2019
10.51
COCG-LocSIZ
8.76
TDC SHAO ET AL.. 2022
11.90
COCG&GT
9.79
CAPDET LONG ET AL.
2023
15.44
CAPDET
13.98
GRIT WU AL.. 2022
15.50
GRIT
INS-DETCLIP CLABAALA
15.70
INS-DETCLIP-FLANBAN INS-DETCLIP FLANLARGE
14.35
INS-DETCLIP- CODDECCLIP-LLANIIII FLANLARGE
16.13
15.01
5.5
ABLATION STUDIES
Cross Attention Layer Frequency. Although it is intuitive that inserting a cross attention layer following every self attention layer achieves the best performance, also introduces higher computa- tional burden. Therefore, it is important that we strike balance between the performance and the efficiency. As shown in Fig. 5 the performance drop for decreasing the frequency of cross attention layer from adding them at every layer to every 3th layer is negligible. Therefore, we adopt this configuration for better computation efficiency.
Inference Efficiency. We compare the inference efficiency between Ins DesCLe and the 2-stage baselines in terms of frames per second (FPS) in Table [ Our model is able to achieve better performance well faster inference speed compared with baseline methods.
Size of LLM. Even though the LLM only outputs the category name, it requires strong interpretation ability of instructions to decide which objects relevant to inputs. To verify this claim, in the following 3 we show that smaller language models have weaker interpretation ability and performs worse benchmark.
8

---

Published as a conference paper at ICLR 2024
Table 3: We demonstrate the importance of the size of LLM. As shown in the following table, the per- formance of our Ins-DetCL.I increases the size of the LLMs grow. This because larger L.LMs possess more powerful reasoning and generalization ability, which is crutial for our instruction-oriented object detection task.
24.4
14:30
22:30
24.0
33.25
14.20
134
15tth 1th
TASK3
TASK4
Figure 5: Performance of Ins- Ins-DeiCLIP with different cross attention layer frequencies. Fre- quency of every 3 layer achieves good trade off between computation annd performance.
PLANTS-SMALL
60M
13.3
23.9
12.1
9.45
7.62
FLANTS-BASE
250M
15.3
24.5
15.3
11.3
10.0
LLAGE
780M
16.2
25.6
16.4
11.7
11.0
Release/freeze LLM and Detector. In Table compare the results between Ins-DetCLIP trained with frozen or released LLM. We observe that although releasing the LLM during training slightly boosts the performance degraded. This is reasonable since tuning the entire LLM makes the model more biased towards the training data, which in turn causes loss in its original knowledge. Therefore, keep LLM frozen during the second phase to promote generalization ability and training efficiency.
in domain instructions, the ability to generalize to out of domain instructs
Table 4: Experiment on releasing the LLM during instruc- tion tuning. Although in domain performance is slightly improved, out of domain performance is severely degraded.
Table 5: Inference speed comparison between Ins-DetCLIP and baselines.
MarROD
INFIRENCU SeILD (PPS)
RELEASE LLM
DOMAIN (TT /33 T4) 15.3 (24.5 15.3//15///111 1/131/111//000 11.26./11.2/11.7/111
OUT DOMAIN AP (TI T2 T4)
BLIUZ-FLANKL BLIPZ-FLANXXL MISIGPT4-VICUNNE
1.0
0.8
13.7 (24.2 16.0 5.90) 11.5 (23.8 10.8
INS-DETCLIP DDETCLIP-FLANBAE DETCLIP PLANLARGE
Ratio of Negative Instructions. In this section, we ablate on the probability of sampling negative instructions Pneg during training. We find that if the negative ratio is too high, the model will have the tendency to output "other". On the other hand, if ratio is too low, the model will output the object's category name regardless of the user instruct. We search across several values and find that 0.1 to be the most suitable ratio.
Table 6: Experiments with different ratios of negative instructions sampled during training.
NEG RATIO
DOMAIN
OUT OF DOMAIN
(TASK1
TA8K2 TASK3 TASK4)
(TASK1 TASK2 TASK3 TASK4)
0.05
14.9 (25.3/ 14,6 10.4 9.52)
13.5 15.1 8 20 5.65)
FLANTS-BASE
0.1
15.3 (24.5 15.3 11.3 10.0)
(24.2/ 16.0 8.62 5.90)
0.5
9.97 8.65) 14.0 (22.3/ 14.9
12.2 14.1 7.95 5.04)
0.05
15.6 (25.9 / 15.9 11.3/ 10.4)
14.3 11.553//1.0/9/47/666
FLANTS-LARGE
0.1
16.2 16.4//6.//111 11.0)
14.4 16.2
0.5
15.1 (22.9 15.6 12..9/15.6/11.... 10.5)
13.3 (22.3 15.23//1.7/9911//6133
CONCLUSION
In this paper, we introduce novel task of modern object detection system called Instructional Object Detection (IOD). Instead of just recognizing objects of given category lists, IOD requires the detector to understand users' instructions and finds relevant objects, bringing fresh human- computer interaction angle to the object detection system. We develop an extensive instruction- detection dataset called JOD-Bench, enriched with over 8,000 diverse instructions. Moreover, based on the proposed IOD- BoD-Bench, we further propose the Ins-DerCLIP to integrate the advantages of extensive language models within the detection framework to understand human instructions and infer corresponding objects. Experiments results demonstrate the effectiveness of the proposed method on BoD-Bench. In addition, Ins- DetCLIP also achieves SOTA performance on dense captioning tasks. We hope work can inspire further innovation in the of language-guice guided detection system.

---

Published as a conference paper at ICLR 2024
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Qiang Bai, Shaobo Li, Jing Yang, Qisong Song, Zhiang Li, and Xingxing Zhang. Object detection recognition and robot grasping based on machine learning: A survey, IEEE access, 8:181855 181879, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. NeurIPS, 33:1877-1901, 2020.
Zhaowei Cai, Gukyeong Kwon, Avinash Ravichandran, Erhan Bas, Zhuowen Tu, Rahul Bhotika, and Stefano Soatto. X-detr: A versatile architecture for instance-wise vision-language tasks. arXiv preprint arXiv:2204.05662) 2022.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Bric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction finerton--d language models. arXiv preprint arXiv:2210.1144 2022.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards models with instruction tuning, 2023.
vision-language
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv://100004805, 2018.
Di Feng. Ali Harakeh, Steven L Waslander, and Klaus Dietmayer. A review and comparative study on probabilistic object detection in autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 23(8):9961 2021.
Dario Fontanel, Matteo Tarantino, Fabio Cermelli, and Barbara Caputo. Detecting the unknown in object detection. arXiv preprint UrXiv:2208.116411 2022.
Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong. Towards open vocabulary object detection without human- bounding boxes. arXiv preprint (rXiv:2/11.09455 2021.
Xiuye Gu, Tsung- Yi Lin. Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921, 2, 2021a.
Xiuye Gu, Tsung-Y Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921, 2, 2021b.
Matthew Inkawhich, Nathan Inkawhich, Hai Li, and Yiran Chen. Self-trained proposal networks for the open world. arXiv preprint arXiv: 1050, 2022.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vasion-language representation learning with noisy text supervision. ICML, 2021.
Justin Johnson, Andrej Karpathy, and Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4565-4574, 2016.
Hakan Karaoguz and Patric Jensfelt. Object detection approach for robot grasp detection. In 2019 International Conference Robotics and Automation (ICRA), pp. 4953-4959. IEFF, 2019.
10

---

Published as a conference paper at ICLR 2024
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping trgguage-imagg pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 125997 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: 2: Bootstrapping language- pre- training with frozen image encoders and large language models. arXiv preprini arXiv:2301.12597, 2023b.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Neng-Neng Hwang, et al. Grounded language-image pre-training. arXiv preprint arXiv:2112.03883 0380357, 2021a.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Lenq-Neng Hwang, al. Grounded language-image pre-training. arXiv preprint arXiv: 12.03857, 2021b.
Xiangyang Li, Shuqiang Jiang, and Jungong Han. Learning object context for dense captioning. In Proceedings of the AAAI conference artificial intelligence, volume 33, pp. 0650-8657) 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014a.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context, In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint
2023a.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.0544 2023b.
Yuxuan Liu, Yuan Yixuan, and Ming Liu. Ground-aware monocular 3d object detection for au- tonomous driving. IEEE Robotics and Automation Letters, 6(2):919-926, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021b.
Yanxin Long, Youpeng Wen, Jianhua Han, Hang Xu, Pengzhen Ren, Wei Zhang, Shen Zhao, and Xiaodan Liang. Capdet: Unifying dense captioning and open-world detection pretraining, arXiv preprint 13t::23033004889 2023.
OpenAl. Introducing chatgpt. https///openai.combllggguuuuuuup 2022.
OpenAl. Gpt-4 technical report. CoRR, abs/2303.08774, 2023.
Rui Qian, Xin Lai, and Xirong Li. object detection for autonomous driving: survey. Puttern Recognition, 130: 108796, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8788 8763. PMLR, 2021.
Kelathodi Kumaran Santhosh, Debi Prosad Dogra, and Partha Pratim Roy. Anomaly detection in road traffic using visual surveillance: A survey. ACM Computing Surveys (CSUR), 53(6):1-26, 2020.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li. and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, pp. 8430-8439, 2019.
11

---

Published as a conference paper at ICLR 2024
Zhuang Shao, Jungong Han, Demetris Marnerides, and Kurt Debattista. Region-object relation- aware dense captioning via transformer. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pp. 894-906. PMLR, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2rXiv:2302.1397,, 2023.
Waseem Ullah, Amin Ullah, Ijaz U1 Haq, Khan Muhammad, Muhammad Sajjad, and Sung Wook Baik. Cnn features with bi-directional Istm for real-time anomaly detection in surveillance networks. Multimedia tools and applications, 80:16979-16999 16995, 2021.
Vicuna. Vicuna: An open chatbot impressing gpt-4. http:///ithub..om//nnssss//////
2023.
Jialian Wu, Jianfeng Wang, Zhengyuan Yang. Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-text transformer for object understanding, 2022.
Johnathan Xie and Shuai Zheng. Zod-yolo: Zero-shot yolo detection using vision- lssiunlanguage knowledge distillation. arXiv preprini arXiv:2 (2099/200 /2066, 2021.
Linjie Yang, Kevin Tang, Jianchao Yang, and Li-Jia Li. Dense captioning with joint inference and visual context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2202,, 2017.
Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-cnriched visual-concept paralleled pre- pre-training for open- world detection. arXiv preprint arXiv:2209.09407, 2022a.
Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detelip: Dictionary-cnriched cisncp- parallcled pre-training for open- world detection. arXiv preprint 2tX1::2209009407 2022b.
Lewei Yao, Jianhua Han, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, and Hang Xu. Detclipv2: Scalable open- vocabuacyyy object detection pre-training via word-region alignment. arXiv preprini arXiv:2304.04514 2023.
Guojun Yin, Lu Sheng, Bin Liu, Nenghai Yu, Xiaogang Wang, and Jing Shao. Context and attribute grounded dense captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6241-6250 2019.
Yuhang Zang. Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. arXiv preprint arXiv:2203. / 1876, 2022.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm- 130b: An open bilingual tre-trained model. arXiv preprint arXiv:2210.0244 10.04444, 2022.
Hanbo Zhang, Xuguang Lan, Site Bai, Xinwen Zhou, Zhiqiang Tian, and Nanning Zheng. Roi-based robotic grasp detection for object overlapping scenes. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4768 4775. IEEE, 2019.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv 2022a.
preprint arXiv:2203.0366
Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianfeng Gao, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. arXiv preprint 13.1 2023.
Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z Li. Bridging the gap between anchor- based and anchor-free free detection via adaptive training sample selection. In Proceedings of the IEEE/CV conference computer vision and pattern recognition, pp. 9759-97668 2020.
12

---

Published as a conference paper at ICLR 2024
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- pre-trained trained transformer language models. arXiv preprint arXiv:2205.0000 2022b.
Yiwu Zhong, Jianwei Yang. Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. Regionelip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16793-16803, June 2022.
Joey Tianyi Zhou, Jiawei Du, Hongyuan Zhu, Xi Peng, Yong Liu, and Rick Siow Mong Goh. Anomalynct: An anomaly detection network for video surveillance. IEEE Transactions on Information Forensics and Security, 14( 14(10))557722550 2019.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. arXiv preprint arXiv:2201.0244 2022.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
A LIMITATIONS
Although Ins-DetCLIP trained on IOD-Bench has shown promising results as the first attempt towards instruction object detection, there are still minor issues that need to be addressed. One limitation is that the category names used to construct IOD-Bence may not always align with the content of the image, resulting inappropriate instructions being assigned images. For example, an image of cup in a bathroom may be assigned the instruction "I wish to drink water." To mitigate this issue, one potential solution is use the data-gent generatienn pipeline proposed by LLAVA (Liu et al. (023a)), which leverages image captions to provide visual context to ChatGPT (OpenAL. 2022) and generate instructions that are more closely related to the image. However, this pipeline may face challenges when applied to large-scale datasets with bounding box annotations that lack caption annotations, limiting its applicability beyond COCO (Lin et al. 2014b). We will further improve our 1OD-Bench in the future work to eliminate this issue.
B MORE IMPLEMENTATION DETAILS
Addressing the Issue of Absent Candidate Categories. Traditional open-vocabulary detectors require a list of object names/phrases to derive confidence score. Therefore, without prior category knowledge, one issue that arises in ins-DetCLIP is the lack of measurement to select the bounding boxes to be feed into LLM. To address this issue, we introduce an extra classification head that distinguishes foreground objects from the background, resembling region proposal network (RPN). The selected regional features are then treated as object features and sent to the LLM for category prediction. After obtaining the category names from the language model, we calculate the class- specific scores between the predicted categories and the regional features using the text and visual encoders of DetCLIP, whose parameters are fixed during the object-level instruction tuning phase.
Additional Hyperparameters and Details. We conduct two non-maximum taa-maximum-suuppeere suppression (NMS) operations during inference, similar to 2-stage object detectors. We first apply class-agnostic NMS using the scores from foreground foreground-backgrommm backgouund classification head, where the IoU threshold is set to classification
0.6. We set the maximum number of region proposals from begground-backgroounn head to 100, which we find to make good trade-off between accuracy and inference efficiency. Then, we extract the text embeddings of the predicted object categories from the frozen text encoder. Subsequently, we perform class- wise NMS using the class- wise similarity scores calculated between the text embeddings and the object visual embeddings, with the IoU threshold set to 0.5. For the first stage, the training of DetCLIP-T with Swin-T T backbone takes 63 hours on 32 V100 GPUs. The resulting model from the pretraining stage can already serve as powerful open-vocabulary object detector. In the second stage, it takes only around 24 hours on 16 V100 GPUs for instruction tuning.
B. ADDITIONAL EXPERIMENTS
Inference Speed Comparison with Similar Model Sizes. We compare the inference speed of our Ins- Da-DetCLIPP LIP with the two-stage baseline in Table 7 We observe that even if the two-stage baseline
13

---

Published as a conference paper at ICLR 2024
leverages LLM with equal smaller size than Ins-DetCLIP, its inference speed is still slower. This is because those methods need to output the object category names sequentially, while our model is able to make predictions for different objects in parallel.
Table 7:
Inference speed comparison between Ins- mas-DetCLIP and baselines with fewer parameters. Ins-DeCCLIP still demonstrates higher efficientey.
METHOD
INFERENCE SPEED (FPS)
BLUP2-FLANBASS
1.4
INS-DETCLIP-FLANNE BABSE INS-DETCCLIP-FLANNAARGG
2.2
1.6
Impact of First-stage Pretraining, We compare the performance of Ins-DeiCLIP w/o first stage pretraining in Table In this experiment, we still utilize the weights initialize the vision backbone, and use FILIP-prtrained performance of Ins-DetCLIIP greatly drops if first stage pretraining is skipped. This verifies that pretraining is an essential step to ensure that the detecto able provide high-quality object features to the LLM.
swin- sii-ttranffomee encoder. We notice that the
Taskl:
Devise various expressions to convey the meaning of "detecting all objects in an image". here examples: 1. Identify every objects within the picture. 2. Ascertain all items present in the image. Give me 10 results.
Task2:
Express "Detect objects in the image" using various sentence structures, where is collection of object names and used placeholder. Here are some simple examples: 1. Identify the objects present in the image. 2. Perform object recognition to detect the objects in the picture. Give me 10 results.
Task3:
Group the following object names into different parent categories: class names of Objects365. Format each result as instruction-objects pair as following: find (and so on) parent category, objects: ['objectl', 'object2', ...]. An example is Look for all electronic devices: ["Moniter/TV', 'Laptop', 'Cell Phone', 'Camera', 'Computer Box', 'Tablet', 'Keyboard', 'Mouse'. 'Printer', 'Projector'. 'Telephone', 'Head Phone', RRmoote. 'Microphone' 'Speakers', "Surveillance Camera', "iir Condi- tioner', 'Fan', 'Router/modem" J. Return all the objects belonging to the parent category. give 50 results. Task4:
alllocatefiemtiiyliiicoreeeettee
Give some instructions delivering specific purpose that require the use of the following objects:(an object name list...]. To be effective, instructions should use a language that is natural and familiar, much like the way people give directions to robots. The purpose and sentence structure of the instructions should be diverse accommodate different scenarios. For each instruction, return the related objects. Format the results as following: Instruction: XXXX, Objects: ['objectl', 'object2", ...]. Only the object names provided in the list, do not use other object names. Now, give me 10 examples."
Figure 6: Detailed prompts for querying 4 types of instructions.
C MORE DETAILS FOR DATA GENERATION
The Creation of IOD-Bench To enable the object detection system to flexibly perform different tasks given natural language instructions, design <instruction><<objeet of interest pairs to help training and evaluation. Specifically, we design such pairs for each task by leveraging the generative power of ChatGPT as follows:
14

---

Published as a conference paper at ICLR 2024
Table 8: The impact of first stage pretraining. We observe drastic performance degradation without the first stage pretrainng. This verifies the importance of pretraining stage, which enables the object detector to extract high-quality object features.
MODEL
PRETRAIN
DOMAIN (TASKI/ TASK2 TASK3 TASK4)
OUT OF DOMAIN (TASKI TASK2/ TASK3 / TASK4)
MSS-DETCLIP SEHICLI--FLANTSBRAE BLANTS-BASE
8.46 (12.6/ 10.5 7.16/3.53)
(11.3/ /6.11/3.311 13.7 (24.2/ 16.0 8.62 5.90)
DO-DDTCLIP-FLANTS-BNN
Y
15.3 (24.5 15.3/ 11.3 10.0)
Table 9: Comparison with directly using open-vocabulary object detectors on observed that OVOD methods achieves poor performances, since they are not able to interpret natural language instructions.
It can be
MODEL
DOMAIN AP (TASKI / TASK22/ TASK3 / TASK4)
OUT DOMAIN
AP (TASK] / TASK2 TASK3 TASK4)
DETCLIP
2.81 (5.35 4.33 1,29 00.8))
2.97 (5.49/ 96 /1,10 /0.11)
GROUNDINGDINO
4.08 (4.49 19.33 2.22 0.21)
3.96 (..426/88899 2.41 00.99
CHETCLIP-FLANTS-BR
12.35 /1.3 10.0)
(24.2/ 16.0 8.62 5.90)
INS-DETCLIP BO-DDECCIP-FLLATS-SAEE FLANTS-LARGE
16.2
14.4 (25.4/ 16.2 9.65 /9.65 6.50)
For task let ChatGPT produce instructions that have the same meaning as "detecting all objects in an image". " In this case, all the objects that exists in the image are considered objects of interest;
For task 2. we let ChatGPT generate more templates similar to "detect <obj names> in the image", where <obj. names> is placeholder for a list of objects and are considered as objects of interest;
For task 3, we generate the instruction-oojects pairs such as "Look for all electronic de- vices: ["Moniter/TVV', 'Laptop', 'Cell Phone', 'Camera', 'Computer Box', 'Tablet', 'Key- board', 'Mouse', 'Printer', 'Projector', 'Telephone', 'Head Phone". 'Remote', 'Microphone' 'Speakers', SSurveillacce Camera', Air Conditioner', 'Fan', 'Router/modem"T" The objects mentioned in the list will be considered as objects of interest for given the instruction.
For task 4, ChatGPT needs to design high-level user instructions and select the objects of interest from the ground truth categories.
With the collected <instruction--<oheects of interest> pairs, we are able to supplement existing datasets with natural language instruction, and curate the IOD-Bench facilitate the research in instruction-orriened object detection.
Precise Definition of Task 4 In the context of our problem definition, the term "relatedness" refers to objects that are relevant or connected to the purpose described in the instruction. The related objects should ones that can be used or involved in accomplishing the specific purpose outlined in the instruction. For example, if the purpose of instruction is to assemble piece of furniture, related objects could include tools, screws, and various components needed for assembly. Similarly, if the purpose is to prepare meal, related objects could include ingredients, utensils, and cooking appliances. In our constructed dataset, the "related" objects of an instruction are comprehensive, which enables model to produce all the objects that may be associated with the instruction. Since cntructiio-orienttr detection is an intermediate step for building an intelligent system, in actual deployment, the predicted objects may be passed to another downstream module to decide what objects to be used exactly, and in which order they should be used.
Prompts for Querying Instructions. Fig. 6 provides the detailed prompts used for querying four types of instructions. These prompts are provided to ChatGPT to generate instruction annotations.
Dataset Statistics. In Table 10 we provide the detailed number instructions for proposed tasks. We generate more instructions for task 3 (692) and task 4 (7768), and less instructions for task 1 (30) and task (111), since there exists more diverse instructions for super-class-related and goal-oriented tasks.
15

---

Published as a conference paper at ICLR 2024
Table 10: Numbers instructions for proposed tasks.
Task1
Task2
Task3
Task4
Total
30
111
692
7768
8601
1OR MIS
JIVC
InstructDet
Figure 7: Qualitative visualization of JIVC and our Ins- DetCLIP on dense captioning task.
More Visualizations. Fig. provides more visualization examples for the constructed 4 types of instructions in our curated 1OD-Bench. demonstrates well correspondence between the instructions and the objects illustrated in the images.
D DEMONSTRATIONS
Instruction-oriented Object Detection. In this section, to showcase the superior instruction- following ability of our Ins- DeICLIP, we demonstrate more examples obtained with Ins-DetCLIP for each task. As shown in Fig. Fig. T0 Fig. [T and Fig. 12] our Ins-DetCLIP is able to flexibly make predictions based on user instructions and locate objects of interest in the image.
Dense Captioning We visualize the dense captioning results in Figure 7 which demonstrates that Ins- DetCLIP is able to provide detailed descriptions for each object in the image.
Scan and calegurize all viiblle objjects in the Inaage.
Iduntify all Hemms that viilble the image
Identify every object the imagr.
Identily abject within
abjects the image.
Locate classify objects visibde in the picture.
secne and recugnize abjjcts within it. visible in the picture.
objects
and clasiity
Identity every
present
the
Figure 8: Visualization of bounding box results generated by our Ins-DeiCLIP for Task 1, which detects all the visible objects the image.
16

---

Published as a conference paper at ICLR 2024
Tiskl
Racegabo
Task2
feeelly plice postin
Perfors www. resueitee
Live
in Be ptonnt
mas be praves
Task3
Task4
tradd from Nov York Loe Angdo
Figure 9: Examples of the constructed types of tasks and their corresponding instructions in IOD-Bench. The instructions align well with the corresponding object categories.
the ldeatify
kaybeard.
Tind train
EVO
Meniiffy train in the Imago
Find socee the imraoo
Detect scale the lmseoo.
Detecc ear. trucks pickup track in the imags.
Find person the imags.
Figure 10: Visualization of bounding box results generated by our Ins-DetCLIP for Task 2, which detects specific object categories in the image.
17

---

Published as a conference paper at ICLR 2024
Find the office supplies,
Discover all the vehicles for personal use
Find all the hendwears,
Ideucity the people.
electronic houschold Hroms.
Find the kitcherwares.
Mentiiy the household Hemss
Find all clectronic deviees.
toys children.
Fegetables.
electronic devices.
poople.
REGLIY
personal
the people.
all outdoor lighting objects.
Edeatity
Identify bathrooms Beens.
Figure 1I: Visualization of bounding box results generated by our Ins-DetCLIP for Task 3, which detects objects that falls under a super-category.
Pripose significant
pizza my
an efficient manner.
the train stutie.
EVO
Wash ma face the morning.
Serve wine.
Keep the mapkin the side tabile.
Watch movie TV show.
Play soccer game.
Illuminate the path while camping.
Serva a slice of Hami melon plate.
Serve appetisers during party.
Light dark room.
Teast two slices brend for breakfust.
podcast episode
Identify
drinks smacks guests,
Propare
from across
Figure 12: Visualization of bounding box results generated by our Ins-DerCLIIP for Task 4, which detects objects that are helpful for fulfilling goal.
18


Reviews:
[{"review_id": "RpDDlcjUSW", "reviewer": "Reviewer_rzsX", "summary": "The manuscript presents a new task called Instruction-oriented Object Detection (IOD), which aims for the model to accept human instruction and generate corresponding detection results. For training the model on such a task, the authors first construct a dataset termed IOD-Bench and the corresponding evaluation metrics. Based on DetCLIP, Ins-DetCLIP is presented for making the open-vocabulary object detector able to follow instructions, where an LLM is attached to the frozen visual encoder of pretrained DetCLIP.\n\nEmpirically, Ins-DetCLIP notably outperforms baseline approaches such as BLIP-2, MiniGPT-4 and LLaVA. Additionally, the model shows the capability of performing captioning for the relevant objects.", "questions": "- It would be interesting to know how much resources and GPU days are required to train such a model.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The manuscript goes beyond image-level captioning/question answering and proposes to use LLM for instructed object detection, which is novel.\n- The approach provides new insights into what a LLM could do when it is connected to an open-vocabulary object detector. \n- Compared to existing approaches such as BLIP-2, MiniGPT-4 and LLaVA, Ins-DetCLIP demonstrates an outstanding capability of performing instruction guided object detection. \n- Increasing the size of LLM could benefit all the tasks, showing good scalability.", "weaknesses": "- It is unclear how the object bounding boxes and the features are generated in the first place before object-level cross modal fusion.\n- It is not shown to what extent is the approach dependent on the performance/quality of the phase-1 training, which would be an important aspect for understanding the approach that requires two-phase training.\n- The inference speed comparison seems unfair, since the model sizes are different. It is difficult to harness whether the approach is slow or fast.\n- Citation formats are wrong.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AZHQhRf4kC", "reviewer": "Reviewer_tem2", "summary": "This paper introduces Instruction-oriented Object Detection (IOD), a novel task aimed at improving human-computer interaction by enabling object detectors to interpret user instructions for identifying specific objects. IOD necessitates the understanding of natural-language instructions and contextual reasoning to provide the name and location of the desired objects, posing new challenges to current object detection systems. To address this, the authors develop a dataset called IOD-Bench, consisting of instruction-guided detections and specialized evaluation metrics, and leverage large-scale language models (LLMs) to generate a diverse set of instructions based on existing public object detection datasets. The proposed model, Ins-DetCLIP, utilizes the knowledge within LLMs to enable instruction-following capabilities in the detector. It employs a visual encoder, DetCLIP, to extract object-level features and aligns them with the input instructions through a cross-modal fusion module integrated into a pre-trained LLM. The experimental results on IOD-Bench demonstrate that Ins-DetCLIP consistently outperforms baseline methods.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Overall, the task presented in this manuscript is captivating and aligns well with the practical applications of an intelligent detection system. The instructions have been well-crafted, bolstering my view on this matter.\n\n2. The clarity of the writing in this submission is commendable, making the content generally straightforward to grasp.\n\n3. The impressive performance of Ins-DetCLIP underscores the effectiveness of the proposed instruction tuning paradigm. Furthermore, the authors have done a good providing an extensive range of experiments to scrutinize all the design choices, which is highly valuable.", "weaknesses": "1. I would recommend that the authors take another pass at proofreading the manuscript to ensure consistent and correct formatting throughout. For instance, there is a conventional practice of inserting a space before references in both the main body text and tables.\n\n2. With respect to the main results showcased in Table 1, while they are compelling and seem to align with the authors' motivations, I note that the comparison methods, such as BLIP-2 and MiniGPT-4, do not incorporate human instructions interns of object detection. I presume that performing instruction tuning on these models could enhance them and provide insights into the generalizability of the proposed tuning method. I am curious about the feasibility of this approach.\n\n3. In terms of the dense captioning tasks, despite the impressive performance demonstrated, the comparison is made with somewhat outdated methods. There are contemporary methods, building on SAM or other models, capable of performing this task as well. A performance comparison with Ins-DetCLIP would be insightful. While Ins-DetCLIP may not outperform these methods, including such results or providing justification would help to elucidate the performance gap, potentially laying the groundwork for future enhancements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "27TRnFUq0Q", "reviewer": "Reviewer_Y4Gg", "summary": "In this paper, the authors introduce a novel task named Instruction-oriented Object Detection (IOD). In the course of developing an IOD system, an innovative dataset, IOD-Bench, is presented, along with a proposed model named Ins-DetCLIP.", "questions": "see Weaknesses", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea that the author solves real-world problems through diverse instructions is interesting.", "weaknesses": "1-I understand the authors' aspiration to address a wide range of real-world scenarios through diverse instructions. However, I find the motivations behind the four tasks to be somewhat unclear. The similarity amongst the first three tasks appears quite pronounced, and it seems that existing datasets already cater to these scenarios to a certain extent.\n\n2-In Section 5.2, the authors described integrating BLIP2 and MiniGPT4 with open-vocabulary detectors (OVD) in a two-stage sequential manner to construct baseline models. I'm curious as to why such a sequential approach was chosen for building the baseline models. Would it not be more straightforward to employ the OVD method for direct open-vocabulary detection? My concern is that a two-stage sequential manner could inherently limit the system's performance to the accuracy of both models, seeming like a suboptimal choice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dOejWOwboX", "reviewer": "Reviewer_t19r", "summary": "This paper proposes a novel detection task, called IOD, which requires the detector to detect objects related to the human instruction. To unravel the problem, the authors first created a dataset called IOD-Bench, and then designed an Ins-DetCLIP model. The idea is simple and straightforward -- by incorporating an LLM with an open-vocabulary detector. Experiments on the IOD-Bench show that Ins-DetCLIP outperforms the baseline methods.", "questions": "- As for the evaluation metrics, the authors said that they use BERT to compute the similarities between the predicted categories and the GT categories. What if the prediction is completely irrelevant to the GT categories? In this case, the similarity score distribution will likely to be random. Will this cause the metrics to be unstable?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed IOD task is novel and has practical values.\n- The method is simple and easy to understand.", "weaknesses": "- Instruction types 1, 2, and 3 seem to be variants of traditional detection or OV detection. With the help of LLM, it should be easy to convert these instructions to traditional detection or OV detection problems. Intuitively, I believe that using off-the-shelf detectors and LLM will achieve good results on these types of instructions. If not, a discussion of the reason is expected.\n- Instruction type 4 is abstract enough to meet my expectations of IOD. However, the objective of instruction type 4 is to detect related objects, which is vague. For instance, should cutleries be detected if the instruction is \"Prepare a healthy salad for lunch\"?\n- In Phase 2 of Ins-DetCLIP training, the visual encoder is frozen. This suggests that the LLM can only pick objects from the object proposals given by DetCLIP. However, DetCLIP is unaware of the instructions. As the instruction becomes more complex, DetCLIP will be unable to recall the target objects with a fixed number of proposals. \n- There have been several multimodal LLMs that are capable of detecting objects, i.e. Shikra[1], Kosmos[2]. The proposed Ins-DetCLIP should be compared with such methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)