InstructDET: Diversifying Referring Object Detection with Generalized Instructions
Authors: Ronghao Dang
Jiangyan Feng
Haodong Zhang
Chongjian GE
Lin Song
Lijun GONG
Chengju Liu
Qijun Chen
Feng Zhu
Rui Zhao
Yibing Song
Venue: ICLR 2024 poster (2024)
Keywords: Referring Object Detection; Visual Grounding; Foundation Models
Primary area: datasets and benchmarks
TL;DR: We use foundation models to produce detection instructions covering common user intention, the instructions are utilized to largely improve referring object detection for practical usage from a data-centric perspective.

Abstract: We propose InstructDET, a data-centric method for referring object detection (ROD) that localizes target objects based on user instructions. While deriving from referring expressions (REC), the instructions we leverage are greatly diversified to encompass common user intentions related to object detection. For one image, we produce tremendous instructions that refer to every single object and different combinations of multiple objects. Each instruction and its corresponding object bounding boxes (bbxs) constitute one training data pair. In order to encompass common detection expressions, we involve emerging vision-language model (VLM) and large language model (LLM) to generate instructions guided by text prompts and object bbxs, as the generalizations of foundation models are effective to produce human-like expressions (e.g., describing object property, category, and relationship). We name our constructed dataset as InDET. It contains images, bbxs and generalized instructions that are from foundation models. Our InDET is developed from existing REC datasets and object detection datasets, with the expanding potential that any image with object bbxs can be incorporated through using our InstructDET method. By using our InDET dataset, we show that a conventional ROD model surpasses existing methods on standard REC datasets and our InDET test set. Our data-centric method InstructDET, with automatic data expansion by leveraging foundation models, directs a promising field that ROD can be greatly diversified to execute common object detection instructions.

Published as a conference paper at ICLR 2024
INSTRUCTDET: DIVERSIFYING REFERRING OBJECT DETECTION WITH GENERALIZED INSTRUCTIONS
Ronghao Dang
Jiangyan Feng"*
Haodong Zhang
Chongjian GE
Lin Song Yibing Song S*
Lijun Gong²
Chengju Liu'
Qijun Chen'
Feng Zhu
Rui Zhao?
TTonjii University
SenseTTime Research The University Hong Kong
*Tencent AI Lab
Alibbba DAMO Academy
dangronghaoB.om en
Zeng jiangyandsensee Time,
yibiogsong, www.weevgll...oooommmm.........e
Viszal Grounding
Referring
Grounding
Expresiia Camprehension
robiit & puiliing withi a draninn bosand, and sbov are raany people seate crised hur
People rinnding and booking pativings
Referring Detection
Streccli pod
Buy loe ligh corner
illonii iudy Intorested pointing
paistor reku soacing port and pastnnge vmsgee
hard, secaring whitee ligh for vus with his Aunds and back rharty
incitied fach
Figure Our ROD aims to execute diversified user detection instructions compared to visual grounding. For images with object bbxs, use foundation models to produce human-like ob- ject detection instructions. By training a conventional ROD model with incorporating tremendous instructions, we largely push ROD towards practical usage from data-centric perspective.
ABSTRACT
We propose InstructDEET data-centric method for referring object detection (ROD) that localizes target objects based on user instructions. While deriving from referring expressions (REC), the instructions we leverage sified to encompass common user intentions related to object detection. For one image, produce tremendous instructions that refer to every single object and different combinations of multiple objects. Each instruction and its corresponding object bounding boxes (bbxs) constitute one training data pair. In order to encom- pass common detection expressions, we involve emerging vision-language model (VLM) and large language model (L..M) to generate instructions guided by text prompts and object bbxs, the generalizations of foundation models effee- tive to produce human-like expressions (c.g., describing object property, category, and relationship). We name constructed dataset as InDET. It contains images, bbxs and generalized instructions that are from foundation models. Our InDET is developed from existing REC datasets and object detection datasets, with the expanding potential that any image with object bbxs can be incorporated through InstructDET method. By using our InDET dataset, tional ROD model surpasses existing methods on both standard REC datasets and our InDET test set. InstructDET, our data-centric method with automatic data ex- pansion by leveraging foundation models, directs a promising field that ROD can be greatly diversified to execute common object detection instructions.
greatly diver-
show that conven-
R. Dang and J. Feng contribute equally. "Y. Song is the corresponding author. This work is done when R. Dang is intern at Sensetime. The code is available at https: //g" thub. com/ jyF engGoGo/ Inst ructilet.

---

Published as a conference paper at ICLR 2024
INTRODUCTION
Referring object detection (ROD) aims to detect target objects according to language reference that represents user intentions. ROD is closely related to visual grounding where there are phrase ground- ing (Akbari et al., 2019; Li et al., 2022a; Gao et al., 2023) and referring expression comprehen- sion (Su et al., 2020; Zhu et al., 2022). As shown in Fig. 1, phrase grounding detects all objects mentioned in one sentence, while referring expression comprehension (REC) only detects one sin- gle object that the text refers to. As such, the language reference in REC shall be discriminative and specifically relates one object without ambiguity.
Currently, visual grounding develops at an initial stage and leaves a gap for practical usage. The phrase grounding does not differentiate which object ought be detected via language description, while REC only targets for one object with single text reference. In the current REC datasets, each image contains few expressions (e.g., I phrases). These expressions are insufficient to represent user intentions. In an image where there are several objects, users may want to detect each single object by using different descriptions (e.g.. object color, shape, or location), or detect multiple ob- jects in different combinations (e.g., similar properties or relationships). These diverse expressions are not conveyed within current REC datasets, leaving the gap for existing methods to practically fulfill user intentions for visual grounding. Moreover, the manual collection of these expressions are cumbersome, and subject bias prevents an effective coverage of common user intentions when perceiving each image. Therefore, the practical user expressions are not well fulfilled when they expect to detect various objects in one image.
In this work', we aim push visual grounding toward practical usage from data-centric perspec- tive. Instead of developing REC models to generalize based on current data, we set up referring object detection (ROD) scenario to automatically diversify user expressions. Inspired by the gen- eralization of foundation models that execute common user instructions based on the image and text inputs, our InstructDET borrows their capabilities produce human-like instructions that en- compass user intentions related to object detection. The generalized instructions produced by the foundation models can be regarded as an expansion of existing user expressions in REC. We pro- duce instructions that describe single object from two pipelines. In the first pipeline (i.e., global prompt), we convert an image into an elaborate text description via LLaVA (Liu et al., 2023a). The text description, together with object bbxs coordinates, are sent to the LLaMA (Touvron et al., 2023) for instruction generation in global prompt. During generation, we manually write in-context ex- amples and leverage the in-context learning (Dong et al., 2023) ability of LLaMA to describe the content related to each object following the format of our examples.
In the second pipeline (i.e., local prompt), we send the image and text prompts into LLaVA. The objects in the image are marked with bbxs and the text prompts require LLaVA to describe the object content. We initialize LLaVA with miniGPT4 weights and find it tends to produce lengthy and global descriptions. So we perform a partial finetuning on LLaVA by using REC data to let focus on local objects. Through these two pipelines, we observe that instructions generated from global prompt pipeline focus more on the object relationship, while instructions generated from lo- cal prompt pipeline focus more on rich visual details and advanced logic reasoning. Naturally, we combine instructions from these two pipelines to formulate expressions for single referred object. During instruction generation, the uncontrolled model hallucination (Li et al., 2022b) brings incor- rect or irrelevant instructions. We propose to use visual-textual verification via CLIP (Radford et al., 2021) for effective instruction filtering.
The generalization and reasoning of foundation models (Wang et al., 2022; Zhou al., 2022) provide sufficient instructions encompassing user intentions for single object description. When describing multiple objects, we divide descriptions into two parts. The first part is to independently describe each single object followed by concatenation, and the second part is to summarize commonalities of multiple objects. The commonality summarization requires unifying similar or related objectives by a higher-level language abstraction that describes their similarities and relationships. We collect the
We do not differentiate "instruction" and "expression" in this paper, as both of them represent user inten- tions. For presentation clarity, in our InstructDET pipeline we refer expressions that are generated by foundation models, and we further refine expressions to instructions for InDET inclusion. As we only focus on ROD, we formalize our instruction by simply adding the word 'detect' heforchand.
2

---

Published as a conference paper at ICLR 2024
combinations of different objects via semantic clustering, then utilize LLM to generate commonality summarizations for each combination.
We automatically collect instructions targeting for single multiple objects in images and con- struct our InDET dataset. Sec. 4 shows an in-depth analysis of our dataset where we establish a guideline organize these instructions from 6 aspects. Compared to existing REC datasets where the instructions only reside in sub-parts of our groups, our InDET is more comprehensive to in- corporate user intentions of object detection. Fig. I shows an intuitive example of the generalized expressions produced by foundation models. By using our InDET dataset, we train a conventional ROD model and find it surpasses existing VG models on standard benchmarks and our InDET test set. Moreover, we also validate that our model has learned to effectively understand the meaning of instructions rather than only recognize key words, which because of the tremendously expressive instructions incorporated for our model training. Our InstructDET method can automatically expand training data by using in-the-wild images with object bbxs, which improves our model generaliza- tions towards practical usage. In addition, our model can already serve as the detection module of the neural-symbolic visual compositional task solution given arbitrary language instructions beyond object detection (e.g.. Visual ChatGPT (Wu al., 2023), VISPROG (Gupta & Kembhavi, 2023)).
2 RELATED WORKS
Visual Grounding. Studies visual grounding (Kamath et al., 2021; Chen al., 2021; Deng al., 2021; Su et al., 2023) can be mainly categorized as phrase grounding (Plummer et al., 2022; Kojima al., 2023) and REC (Hudson & Manning, 2018; Li & Sigal, 2021). Phrase grounding detects all objects mentioned in the text while REC localizes one object that the text referred to. In (Zhang al., 2022: Liu et al., 2023c), the objects mentioned in the text are verified to each visual object proposal one-by-one. These methods require clear and specific object referring in the text. On the other hand, methods (Zhu al., 2022; Yan et al., 2023) based on DETR (Carion et al., 2020) can accept abstract and summarized descriptions such as "red objects" and "all objects". Our ROD model follows DETR-based based design to enrich interpretation of various instructions. Note that our model is learned via InDET dataset where instructions are produced based preset object bbxs.
Referring Expression Datasets. The REC datasets are usually constructed via manual annotation on the images. A two-player game is utilized in (Kazemzadeh et al., 2014) where the text descrip- tions are concise due to limited relevant visual contents. The RefCOco, RefCOCO+ (Yu et al., 2016), and RefCOCOg (Mao et al., 2016) employ MSCOCO (Lin et al., 2014) images for man- ual expression production. The expression flexibility and diversity of these datasets are limited to encompass common detection intentions. Recent datasets (Krishna et al., 2017; Kuznetsova al., 2020; Kebe et al., 2021) focuses on data scalability rather than auto generation. Cops-Ref (Chen al., 2020) leverages scene graph as reasoning groundwork, thus forming tree structure gen- erate expressions with varying compositionality. Different from these methods based on template guided expression generation, our InstructDET relies on foundation models produce well gener- alized and human- instructions.
Data Generation via Foundation Models. The InstructGPT (Ouyang al., 2022) and GPT4 (Ope- nAI, 2023) have shown generalization and reasoning abilities for data generation. LLaVA (Liu et al., 2023a) first uses GPT4 for multi-modal data generation following instructions. Otter (Li et al., 2023a) performs multi-modality in-context instruction tuning by levering multiple images, ques- tions, and answers. Currently, these models focus on global image and language understanding, with less focus on local object analysis. Moreover, these multi-modality models, although pro- cessing modai-y data, still outputs mingle-modality text description. There is a gap for these foundation models to function in the computer vision scenarios, especially visual recognition. In comparison, our InstructDET uses foundation models to benefit ROD model training, which con- tributes directly to improve object detection performance.
3 INSTRUCTDET
Fig, 2 shows overview of our InstructDET method for data construction. Given input image with object bbxs, we use two pipelines to produce detection expressions from foundation models. The expressions are further refined to instructions and incorporated into our InDET dataset.
3

---

Published as a conference paper at ICLR 2024
VLM Prompt
LLM Prompt
Expressio
Deseription
Parts Finchasing
Traitin Image
Tonk Deseription
Sumple
Dropout
LLAVA
LLAMA
Post Processing
Oltyart bix axis
sangugge Description
Instrattoon
Local Procupe Pipcline
InDET
Global Prompt Pipeline
Figure 2: An overview of our InstructDET. We use two pipelines to produce detection expressions via foundation models. In the global prompt pipeline, we use LLaVA to describe an image via text, and combine this text with other text prompts for LLaMA input. In the local prompt pipeline, we use the same image with object bbxs and text prompts as multi modality input for LLaVA. The produced expressions are further refined instructions and incorporated into our InDET dataset.
3.1 GLOBAL PROMPT PIPELINE
The large language model (LLM) has shown surprising generalizations to well execute common user instructions. We use LLM to simulate user intentions when perceiving objects in an image. Our global prompt pipeline produces text prompt for the LLaMA model. This prompt consists of several contents including global image description, object bbx coordinates, in-context samples, and task description. Without tuning LLaMA, we obtain instructions that describe objects an image. A detailed example is shown in Sec. C for intuitive illustration of how we leverage foundation models to produce expressions. We elucidate the key steps during this process as follows:
Given an image with object bbxs, we first obtain global image description in text form. If this image already contains dense captions (e.g., from Flicker30K), we directly load these captions. Alternatively, leverage LLaVA to generate the global image description. The text prompt for LLaVA contains our language guidance to emphasize that specific interested object categories shall be mentioned. As such, LLaVA will describe each labeled object in its output. As for object bbx content, if the image is from REC dataset, we use referring expression as the object content. Otherwise, we simply use the category name.
When designing the task description prompt, we expect LLaMA to produce diverse expressions that contain different properties of single object as much as possible. We manually list the attributes from the user perspective. including the object type, color, function, motions, etc. Besides, we include the object attributes of its relationship with other objects in image, such as object interactions, object relative positions, etc. When using these aforementioned prompts for LLaMA input, we find that the output text varies significantly and might be irrelevant to the target objects. Inspired by the in- context learning ability of foundation models, we manually design in-context samples to regularize the output content and format. The output results will thus resemble our in-context examples but with our expected diversified object descriptions.
3.2 LOCAL. PROMPT PIPELINE
The global prompt pipeline produces expressions according to text prompts. Naturally, we can feed both image and text prompt to the multi-modality foundation model for object description. Given an image, we mark the object with bbx rectangle, and send this image to LLaVA, together with the text prompt that requires LLaVA to describe the object according to the bbx. Here, the bbx serves as visual highlight for LLaVA to comprehend the target object that we expect to describe.
Our LLaVA model is initialized with miniGPT4 weights. When we send these multi-modalle inputs to LL,, we observe that LL produces detailed and dense descriptions for the image, rather than expressions of the specific target object. We analyze that the lisionn alignment module in LLaVA is the Q-Former (Li et al., 2023b), which transforms one image into only 32 visual tokens without concentrating on the local objects. Mcanwhile, LLaVA itself tends to produce lengthy and dense descriptions. In order generate instructions suitable for ROD, finetune a part of LLaVA by using existing REC datasets. Specifically,, we only update a linear layer that transforms visual
IIn this paper, we use variant of LLaMA (i.e., Vicuna 13B) that has gone through instruction tuning. Besides, we use variant of VA,, which is multi-modal paradigm that maps visual features into token emheddings with further alignment to the text domain.
4

---

Published as a conference paper at ICLR 2024
tokens to the text embedding space during training. The linear layer is learned to attend local objects with concise expressions. After finetuning, we observe the LLaVA output becomes informative and closely related to the target object. Detailed examples of generating expressions in local prompt pipeline are shown in Sec. B, and the detailed analysis on how finetuning improves LA output provided in Sec. H.
3.3 EXPRESSION FILTER
In global and local pipelines, we have regularized the output of foundation model from several as- pects including text prompt specification, in-context learning, and model finetuning. In practice, still observe the model hallucination phenomena that the model sometimes generate expressions describing objects not even exist in the image. Moreover, the expressions from the local prompt pipeline sometimes describe the whole image rather than local objects. This is due to the miniGPT4 initialization of LLaVA, which utilizes dense captions for instruction tuning. The tendency to gen- erate global image description is mitigated via our model finetuning focus local object, but not completely disappeared. To further improve the expression quality, we introduce visual and lan- guage matching via CLIP (Radford al., 2021) to filter out inappropriate expressions. Fig. 3 shows an overview. It contains image visual prompting and disual-textur matching.
Visual Prompting We study visual language pretraining (VLP) (Yang et al., 2023; Shtedrit- ski et al., 2023) where visual prompting is de- veloped for images. We observe that in zero- shot REC, coupling VLP with visual prompts enables robust pairing of local image region and corresponding text description. In the pairing process, the design of visual prompting heavily influences the visual-textual matching results. Specifically, we employ the superposition of red ellipse and the target Gaussian blur reversion as visual prompts. A detailed pipeline illustrating visual prompting is in Sec. D.
Visual Prompting
Gencratel Espreades
CLIP
CLIP
Expression Drupout
Figure Expression filtering by image visual prompting and visual-textual matching via CLIP.
Hisual-Textun Matching. We use images with visual prompting that emphasizes target objects verify the corresponding text descriptions via frozen CLIP model. While local object con- tents are well aligned with target referring expressions, we observe that expressions describing the whole image are not eliminated by CLIP. We analyze that CLIP originally trained focus on the correlations between global image features and global textual semantics. The global visual-textual matching makes CLIP model to prefer global image description accordingly. To remove this ef- fect, we establish referring measurement from both local and global perspectives. For the image in Fig. 3, we compute a global score Sg and local prompt score Sj. The magnitude of referring expression be measured via our local enhancement score S. Si Sg. Our final expression evaluation score can be computed as:
S = S. + +2 (a) + ag)) - =
S1
where 0i and a2 are scalars balancing the contributions of Sg and St with an a 1. So On E [0. adjusts the final score towards local content referring or global semantics. Note that introduce S. to measure difference between local and global scores. If the expression is more related to the target object, S. becomes higher after visual prompting for object highlights. After computing Sy, set dynamic threshold to filter out expressions. This because Sy is based on CLIP's preference that a small target object with well matched expression achieves lower score than large object with mismatched expression. Therefore, we use provided expression (for images from REC) or category name (for images out of REC) to compute final score, and discard generated instructions whose Sy is lower than this score.
3.4 MULTI-OBIECTS EXPRESSION GENERATION
Our expression generation pipeline illustrated above targets for each object independently. In prac- tice, users may refer to multiple objects in one image. We study common user expressions for multi-objects, and conclude them into two aspects. The first contains splicing expressions that
5

---

Published as a conference paper at ICLR 2024
combine different single object expressions with 'and' comma. In this case, the objects mentioned in the expression are not related to each other. The second aspect contains generalization expres- sions that summarize the common properties of multiple objects (e.g.. color, category, or location) to produce an abstract and conceptual description. It resembles mining similarities between multiple objects and thus is not straightforward to conclude. Therefore, we need discover object combi- nations where similar properties may exist, and then summarize the commonalities among them to constitute the summarized expressions.
Our process to produce summarized expression shown Fig. 4. For each object, we first con- catenate all its related expressions with com-- mas. Through this concatenation, we can ob- tain this object expression from different per- spectives (i.e., different properties). Then, we use the text encoder in BERT (Devlin et al., 2018) to map this concatenated expression to semantic embedding space. As such, obtain embeddings of concatenated expressions from all objects. Then, we cluster these embeddings into indeterminate number of clusters by using DBSCAN (Ester et al., 1996) method. We use LLaMA to generate text for clusters with multiple objects. The details of using LLaMA to mine object commonalities are in Sec. E. The generated text indicates the summarized expression we aim to produce for multiple objects.
vdlov bees sust the surma onper bask lir v Aver
Tapt
Ao preen Nonua ville fund with precn is Now' slogg
porse becioa woter hove, person tlact sliis pordi parves doe
and rlrs mikng dioogge
Aonae plass plac ao Aerie beeiiinaii of'le en
Straae
Figure 4:
Mining commonalities among multi- objects via expression concatenation and text se- mantic clustering, followed by LLaMA descrip- tions on each cluster center.
Post Processing. After generating expressions for single and multiple objects, we verify and re- move the repeated expressions that pertain to the same object. Then, we utilize LLaMA to further diversify these generated expressions while preserving their original intents, i.e., we use LLLMA to do synonymous rewriting of generated expressions. The prompt we use for synonymous rewriting is provided in Sec. I. We observe that for different objects in one image, the expression for one object may be similar to that of others. These expressions are ambiguous since we can not refer to a unique object based on their referring. Nevertheless, we transfer these expressions to refer to multi-objects since they express set of objects in one image. This transfer further augments multi- object referring expressions. Finally, we collect these remaining expressions after post processing as instructions. Together with corresponding object bbxs and images, we construct our InDET dataset by incorporating diversified object detection instructions encompassing user intentions.
4 DATASET ANALYSIS
Our InDET dataset contains images from MSCOCO (Lin et al., 2014), Flicker (Plummer et al., 2015), and Objects365 (Shao al., 2019). There are 120.6K images with 908. 908.4K referring object sets in total. Together with original expressions, there are 3.6M instructions in total, making InDET the largest real-world REC dataset present. The average instruction length is 6.2 words and the vocabulary size is 63k words, which surpasses existing automatically annotated datasets in terms of instruction quantity, richness, and vocabulary breadth. We split the images into training, validation, and testing sets, with the corresponding instruction amount of 3139K, 240K, and 247K, respectively. In the following, we first propose a guideline that represent common user intentions and divides existing instructions into 6 groups. Then, we analyze all the instructions in InDET according to this guideline show how our InDET advances REC scenario compared to existing datasets.
Instruction Guideline. The instructions in InDET dataset describe objects from various perspec- tives. We observe that these descriptions all focus on object category, attribute, and relations, but with different emphasis extent. Based on expression complexity, we establish guideline that di- vides all instructions into groups. Each group reflects one level of emphasis on category, attribute, and relations. Table shows our guideline and examples. The first four groups are for single object and the last two groups are for multiple objects. In the first group (GI), there is only one single phrase to describe object category, which is similar to the traditional object detection task. From G2 to G4, more phrases are involved to describe the target object. For G5, we construct a spliced form to combine instructions from different single objects. In G6 the instruction is general description

---

Published as a conference paper at ICLR 2024
Table 1: Instruction guideline and samples. Our guideline contains 6 aspects and covers common user intentions. These aspects are built upon object category, attribute, and relations with different emphasis levels. We use and ** to indicate the description complexity of different aspects.
Aspect
Cutegory
Attribute
Relation
Examples
pencil;; two childrow, soccer hall; city street
2
shirts with Englich lettery ved and white airplane
men in blse okies baffeey on scuee.
Single Object
3
a mas in gray rwonter and black jeans yas forming a skateboarding trick:
people who sitting under unbrella:
*
**
womun sitting cruss feggedi the couch with her back faring the viewer has whkite shirt black pant; the glauses are sitling top of some lind of papor folder
4
a book and lantern nest
Multiple Objects
5
Single object combination
tlue ruck in ralks
a hlark hot on a mon's head and red umbrella and
6
Commonality generalization
every object on table; Side playing with the blond bay
Amouat (e))
Percentage %
Amount (e')
12
WOFT nefocco Mived
1.0
10
Raltsso Muad
0.4
Ficter
8
0.8
0.6
0.3
0.4
0.2
0.2
0.1
Num of
Cosine 1.0 Similarity
10
15
20
25
Words
0.6
5 6
Group
(e) Length distribution
Diversity distribution
(c) Group distribution
Figure 5: Dataset analysis of expression length, diversity and group distributions.
of commonality between multiple objects. To this end, the instructions from G1 to G6 gradually introduces semantic understanding, visual language grounding, and logic reasoning for ROD. After guideline construction, we use LLaMA to assign each instruction into our groups by using in-context learning that let LLaMA to understand assigning principles and in-context assigning examples. The detailed usage of LLaMA for instruction assign shown in Sec. F.
Instruction Length, Diversity, and Aspect Ratio Distributions. We analyze our InDET from the instruction length, diversity, and ratio distribution in our guideline groups The RefCOCO and Flicker datasets are introduced for comparison. Fig. 5(a) shows the number of word distribution where the instruction of InDET contains more words than the other two datasets. Moreover, there 100K instructions in our InDET consist of more than 10 words, while other datasets do not contain such informative expressions. In Fig. 5(b), we show diversity comparison where we use CLIP to map all instructions into semantic space. Then, for the same target objects compute average pairwise cosine similarity. The results show that our InDET contains lower value than other datasets, which indicates that our instructions are more diverse when describing the same target object. In Fig. 5(c), we show aspect ratio distribution of expressions assigned by our guideline. For existing datasets, user expressions commonly exist in GI and G2. In contrast to the user expressions that seldom exist from G3 to G5 for Flicker, and seldom exist in G5 and G6 for RefCOCO, the instructions in our InDET exist normally in all groups This distribution shows that our InDET is more effective to encompass common user intentions, especially for multiple objects. By leveraging our InDET, the ROD model becomes more practically applicable.
5 REFERRING OBJECT DETECTION
In this section, we illustrate our model design for ROD task. We notice that ROD shares little difference with visual grounding (VG). First, ROD produces uncertain number of object bbxs (i.e., 0. 1, or multiple) based on one input instruction, shown in Fig. 1. Second, ROD supports abstract and summarized object descriptions (e-g. "all objects on the table")) that do not clearly refer to specific objects such "bottle", "'rrngge,, and "knife". As recent VG models (Zhang et al., 2022; Liu et al., 2023c) require a one-by-one verification between visual objects and expression words, they not able to execute such instructions. Motivated by the difference, we set up conventional framework from DETR-bare based VG models (Zhu et al., 2022; Yan et al., 2023). Fig. 6 shows an overview of our DROD model. We illustrate key steps follows:
Given an image with text instruction, we use visual and text encoders (Dosovitskiy et al., 2021; Devlin et al., 2018: 8; Ge et al., 2023) to obtain their embeddings. Then, a bi-directional cross
7

---

Published as a conference paper at ICLR 2024
attention module to perform multi-modality embedding fusion. For the fused visual embedding, sent it to the transformer encoder and decoder structure (Zhu et al., 2020) with N learnable queries as position priors (Meinhardt et al., 2022; Ge et al., 2021). Then, the decoder produces N instance proposals for further selection. For the fused text embedding, we pass it through a global average pooling and MLP for text2visual embedding space mapping. Finally, we use cosine similarity to match proposals and mapped text embedding, During the training stage, we use confidence loss and localization loss via supervised learning. During the inference stage, we select proposals whose matching scores are above a predefined threshold, which allows our model to produce arbitrary number of bbxs for diversified instruction execution. More details are shown in Sec. G.
EXPERIMENTS
We evaluate the ROD performance standard VG benchmarks (i.e.. RefCOCO, RefCOCO+, and RefCOCOg) and our InDET dataset. As illustrated in Sec. 4, the images with marked objects of our InDET dataset are collected from existing datasets while the instructions are significantly enriched. We split the training and test set of InDET following RefCOCO/g/+ where the test set contains 6.5k images with an increased number of instructions to 315K. Moreover, these instructions are assigned to groups according to our guideline. The perfor- mance on each group reflects how VG methods perform when processing different aspects of user instructions. The comparing methods in our experiments are from recent VG methods including MDETR (Kamath et al., 2021), Grounding-DINN (Liu et al., 2023c) and UNINEXT (Yan et al., 2023). Due to page limit, show the evaluation results fled expression, and standard benchmarks. Model training, and ablation studies on partial LLaVA finetuning and visual prompt selection are provided in Sec. G and Sec. H. We also provide visual comparison results of these methods in Sec. A. A video demo showing the practical usage of our DROD model is in our webpage.
Farbeckingg
a blonde lady wearing u long white scarf
PwdtMLP
Cress Atizetion
Dransfomer
Decoder
Tharsformer
Encodar
Image
Emhedding
Figure 6: An overview of our diversified referring object detection (DROD) model.
InDET, InDET with shuf-
Table 2: Evaluation results on our InDET and shuffled InDET test sets. We show the object bbx average precision (AP) values (%) of these two test sets with a slash (*'' separation.
AP by Group
Method
Backhone
AP
G1
G2
G3
G4
G5
G6
MDETR G-DINO
ResNet101
34.86/31.21
47.44446.61
46.79/42.55
34.14/28.13
23.22/ 16.86
25.91/23.52
28.17 23.66
SwinB
35.96/30.43
47.10/45.91
47.17/42.56
35.29 27.22
26.84/
27.95/23.74
27.61/2.55
UNINEXT
ResNet50
43.37/37.61
54.49/53.09
$4.52/49.91
/5.49/35,59
28.33
32.01 28.28
/2.352
DROD (Ouns)
ResNet50
62.24/53.78
67.14/65.08
67.34/6156
60.89 /48.22
55.10/41.50
64.15/
74.22/67.11
DROD (Ouns
WT-H
66.90/57.32
72.53 / 69.79
72.47 /5..44
52,50
59,86 / 46.01
73.34/67.82
75.92/ 68.73
In our InDET test set, compare DROD model other methods under the evaluation metric of object bbx average precision with a threshold of 0.5. On the other hand, we investigate whether these methods have truly comprehended the meaning of instruction, or they perform ROD only based on the key words (c.g., noun) without comprehending the whole expression. So we shuffle the InDET test set by randomly ordering the words in each instruction. We produce results of existing VG methods on our InDET test set without assuming object numbers in advance. For one method, its performance drops more on the shuffled data, this method is shown to better comprehend the meaning of instruction.
Table shows the evaluation results. Overall, UNINEXT achieves a higher AP than MDETR (i.e., 43.37 v.s. 34.86) in our InDET test set, while decreasing more than MDETR (i.e., 37.61 v.s. 31.21) in shuffled data. This indicates that UNINEXT is more effective than MDETR for ROD and bet- ter comprehends instruction meaning. Meanwhile, UNINEXT achieves a higher AP value than Grounding- Frounding-DINO. In comparison, our DROD largely surpasses UNINEXT (62.24 v.s. 43.37) on the overall AP comparison, and using a VIT encoder further increases our performance. This indicates that our DROD is more effective to comprehend generalized instructions for ROD. Meanwhile, we observe that our performance drop is larger than UNINEXT (8.46 v.s. 5.76), which shows that
8

---

Published as a conference paper at ICLR 2024
Table Evaluation results on the RefCOCO/g/+ datasets. We follow evaluation protocols report AP values (%) of comparing methods. We use the notations "CC", "VG", "OI", "0365", "RIGame", for COCO, Visual Genome, Openlmage, Objects365, ReferltGame, respectively.
RefCOCO
RefCOCO+
RefCOCOg
Method
Backbone
Data
val
TestA
testB
val
testA
testB
val-u
ReFTR
ResNet101
VG VG,RIGame I RefC
85.65
68.73
81.16
77.55
82.26
68.99
79.25
80,01
SeQTR
DarkNet53
87.00
90.15
83.59
78.69
84.51
71.87
82.69
83,37
MDETR
ResNet101
GoldGi,CCRRtt"
86.75
89.58
81.41
79.52
84.09
70.62
81.64
80.89
G- G-DINO
SwinB
0365,CC,ReIC GoldG.etc
83.95
87.79
79.16
72.91
80.91
62.96
76.98
76.76
UNINEXT
ReaNet50
0365, Re.Ref
87.64
90.35
83:49
78. 14
83
68.71
80.96
81.86
DROD (Ours)
ResNet50
88.92
78.27
71.04
83.01
AP
abject ohat pensect
plater
rbinz people
Rellneo-Mixed Ficker InDET
MDETR
G DINO
UNINEXT
(a) Numerical Results
(b) Visual Comparisons
Figure 7: Our InDET dataset improves logic reasoning of ROD models. In (a), existing models trained with our InDET dataset show superior results compared to other datasets. In (b), we show visual comparisons by using the same DROD model but with different training datasets.
model better comprehends different expressions. Specifically for the results in each group, we notice that our performance drop is little in GI, and becomes larger from G2 to G4. This is because more and more words are introduced from G1 to G4 for object description. A random order gradually af- fects our model comprehension. For G5 and G6, we note that our method largely outperform other methods. The multi-object instructions incorporated in the dataset improves our performance.
Besides evaluating our InDET test set, we compare our DROD model with existing VG meth- ods (Zhu et al., 2022; Li & Sigal, 2021) on the standard VG benchmarks RefCOCO/g/+. Table 3 shows the evaluation results. Overall, our DROD model achieves favorable performance on these datasets. This is because our DROD model utilizes InDET dataset where diversified instructions im- prove model generalizations. By using a conventional ROD model, we improve the VG performance from the data diversity perspective.
In addition to the overall precision comparisons, we evaluate how our dataset improves logic rea- soning and instruction comprehension of existing models. Specifically, we select 2k test samples from our InDET test dataset where logic reasoning on instructions is required for object detection. For each model (i.e., MDETR, G-DINO, or UNINEXT), we train it by using different datasets (i.e.. RefCOCO, Flicker, or InDET) and show the performance comparison on our 2k test samples. Fig. 7(a) shows the evaluation results where each model trained with our InDET dataset outperforms the same model trained with other datasets. In Fig. 7(b), we show visual comparisons by using our DROD model but with different training sets. It shows that using original datasets, the model tends to ground keywords rather than preform modal reasoning based on instructions. In compar- ison, by training with our InDET, the model well interprets instruction meaning and conduct logic reasoning across languages and visual images.
CONCLUDING REMARKS
We aim push ROD into practical usage from a data-centric perspective. On one hand, we notice that current REC expressions are insufficient to encompass user detection intentions. On the other hand, foundation models have shown promising generalizations to simulate manual understanding and description abilities. To this end, we develop InstructDET that leverages foundation models to produce human-like expressions in REC, which tends to incorporate common user intentions into ROD training. As a result, our DROD model achieves favorable performance compared to existing VG methods. In the future, we can combine our method with open-set object detectors to fully explore in in-the-wild images (e.g., Internet images) for comprehensive user expression generation. We expect our DROD model to generalize as much as existing foundation models, and thus take a huge step towards completely solving ROD task.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This paper is supported by the National Natural Science Foundation of China under Grants (62073245, 01173248,62233011 Shanghai Science and Technology Innovation Action Plan (2251 I 104900). Shanghai Municipal Science and Technology Major Project (201IHHZZZX0100) and the Fundamental Research Funds for the Central Universities.
REFERENCES
Hassan Akbari, Svebor Karaman, Surabhi Bhargava, Brian Chen, Carl Vondrick, and Shih-Fu Chang. Multi- level multimodal common semantic space for image-phrase grounding. In JEEF/CVF Conference on Com- puter Vision and Pattern Recognition, 2019.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, 2020.
Zhenfang Chen, Peng Wang, Lin Ma, Kwan-Yec K Wong, and Qi Wu. Cops-ref: A new datasct and task on compositional referring expression comprehension. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yee Kenneth Wong, Joshua B. Tenenhaum, and Chuang Gan. Grounding physical concepts of objects and events through dynamic visual reasoning. International Conference on Learning Representations, 2021.
Jiajun Deng, Zhengyou Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. Transvg: End-to-end visual grounding with transformers. In JEEE/CVF International Conference on Computer Vision, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. ArXiv preprint arXiv:1810.04805, 2018.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey on in-context learning. ArXiv preprint arXi::2301.002334 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Repre- sentations, 2021.
Martin Ester. Hans-Petee Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based based algorithm for discovering clusters in large spatial databases with noise. In Special Interest Group on Knowledge Discovery and Data Mining, 1996.
Shangqian Gao, Burak Uzkent, Yilin Shen, Heng Huang, and Hongxia Jin. Learning to jointly share and prune weights for grounding based vision and language models. International Conference on Learning Representations, 2023.
Chongjian Ge, Youwei Liang, Yibing Song, Jianbo Jiao, Jue Wang, Ping Luo. Revitalizing can atten- tions via transformers in self- supervised visual representation learning. In Advances in Neural Information Processing Systems, 2021.
Chongjian Ge, Jiangliu Wang, Zhan Tong, Shoufa Chen, Yibing Song, and Ping Luo. Soft neighbors are positive supporters in contrastive visual representation learning. In International Conference on Learning Representations, 2023.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without train- ing. In IEFF/CVF Conference on Computer Vision and Pottern Recognition, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2016.
Drew Hudson and Christopher Manning, Compositional attention networks for machine reasoning. In Interna- tional Conference Learning Representations, 2018
Aishwarya Kamath, Mannat Singh, Yann LeCun. Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. Mdetr- modulated detection for end-to-end multi-modal understanding. In IEFF/CVF International Conference on Computer Vision, 2021.
10

---

Published as a conference paper at ICLR 2024
Sahar Kazemzadeh, Vicente Ordonez. Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Conference on Empirical Methods in Natural Language Processing, 2014.
Gaoussou Youssouf Kebe. Padraig Higgins, Patrick Jenkins, Kasra Darvish, Rishabh Sachdeva, Ryan Barron, John Winder, Donald Engel, Edward Raff, Francis Ferraro, and Cynthia Matuszek. A spoken language dataset of descriptions for speech- based grounded language learning. Advances on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Alexander Kirillov, Erie Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, arXiv:2304.02644 2023.
Segment anything. ArXiv preprint
Noriyuki Kojima, Hadar Averbuch-Elor, and Yoav Artzi. A joint study of phrase grounding and task perfor- mance in vision and language models. ArXiv preprint arXiv:2309.0269) 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vixion, 2017.
Alina Kuznetsova, Hassan Rom, Neil Alidrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tusce Shahab Kamali, Stefan Popov, Mattco Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification,, object detection, and visual relationship detection at scale. Interna- tional Journal of Computer Vision, 2020.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with co-tontext instruction tuning. ArXiv preprint arXiv:2305.0337 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 22ip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ArXiv preprint arXiv:2301.12597, 2023b.
Liunian Harold Li, Pengehuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang. Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022a.
Muchen Li and Leonid Sigal. Referring transformer: A one-step approach to multi-task visual grounding. In Advances in Neural Information Processing Systems, 2021.
Xiang Li, John Thickstun, Ishaan Guirajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. Advances in Neural Information Processing Systems, 2022b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv preprint arXiv:2304.0844) 2023a.
Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang. Ravi Kumar Satzoda, Vijay Mahadevan, and R Man- matha. Polyformer: Referring image segmentation as sequential polygon generation. In Proceedings of the IEEEJCVF Conference Computer Vision and Puttern Recognition, pp. 18653-18663, 2023b.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ArXiv preprint arXivv2303.0544 050999, 2023c.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
Junhua Mao, Jouathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy, Genera- tion and comprehension of unambiguous object descriptions. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. Trackformer: Multi-object tracking with transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
OpenAl. Gpt-4 technical report. 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedhack. In Advances in Neural Information Processing Systems, 2022.

---

Published as a conference paper at ICLR 2024
Bryan Plummer, Nikoli Dryden, Julius Frost, Torsten Hoefler, and Kate Saenko. Neural parameter allocation search. In International Conference on Learning Representations, 2022.
Bryan A Plummer. Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-tor testtto-pppee -o-paasss correspondences for richer image-to-sentens models. In IEEF/CVF International Conference on Computer Vision, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Ob- jects365: A large-scale, high-quality dataset for object detection. In IEEE/CVF International Conference on Computer Vision, 2019.
Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about red circle? visual prompt engineering for vlms. ArXiv preprint 2.0..:::300.66711 2023.
Wei Su, Peihan Miao, Huanzhang Dou, Gaoang Wang. Liang Qiao, Zheyang Li, and Xi Li. Language adap- tive weight generation for multi-task visual grounding. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
Weijic Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-linguistic representations. International Conference on Learning Representations, 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anee Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Frie Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv preprint arXiv:2302.1339 2023.
Thomas Wang. Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung. Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective works best for zero-shot generalization? In International Conference on Machine Learning, 2022.
Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. ArXiv preprint arXiv:2303. 0467000044711.. 2023.
Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Lingfeng Yang. Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. Fine-grained visual prompting. ArXiv preprint arXiv:2306.04335 2023.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara Berg. Modeling context in referring expressions. In European Conference on Computer Vision, 2016.
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang. Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language un- derstanding. In Advances in Neural Information Processing Systems, 2022.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe Le, et al. Least-to-most prompting enables complex reasoning in large language models. ArXiv preprint arXiv:2205. 222.2222...22255. 10625, 2022.
Chaoyang Zhu, Viyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, and Rongrong Ji. Seqtr: A simple yet universal network for visual grounding. In European Conference on Computer Vision, 2022.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. ArXiv preprint 2rXiv20010.04159, 2020.
12

---

Published as a conference paper at ICLR 2024
APPENDIX OVERVIEW
We provide an overview to present a clear understanding of this section.
In Sec. A, we present visual comparisons of recent VG methods under various instructions.
In Sec. B and Sec. C, we describe the process generating expressions from foundation models in both global prompt and local prompt pipelines, respectively.
In Sec. D, detail the aspects of visual prompting and visual-textual matching.
In Sec. E, we explain our approach to designing prompts that enable LLaVA to extract commonalities among multiple objects for summarized expression generation.
In Sec. F, discuss strategy for designing prompts that allow LLaMA to assign gen- erated instructions to our predefined groups.
In Sec. G, we provide an overview of our model architecture and implementation specifics.
In Sec. H, we present ablation studies investigating LLaVA fine-tuning in local prompt pipeline and visual prompting selection.
In Sec. I, we provide the prompt we used in post processing.
In Sec. J, we add supplementary evaluation results to Table. and Table.
Sec. K, we add the time consumption of main steps in our instruction generation proce- dure.
A VISUAL COMPARISON RESULTS
Instruction:
Instruction: person who holding remote
person not holding
DROD XT
DROD
DROD
DROD
Instruction
not holding knife
Instruction
red phone not reflection
PERODAT
DROD NINEXT
Figure 8: Visual comparison results. Our DROD well executes detection instruction compared to Grounding-DINO and UNINEXT.
13

---

Published as a conference paper at ICLR 2024
Instruction:
Instruction:
item that might not be the healthiest option
a food preparation container with the contents obscured by hand that over
UNINEXT
DROD
UNINEXT
ROD
Instruction:
hair that needs to be dried
Instruction:
the white area in the middle of the cake
UNINDROD
UNIN DROD
DING
DROD
fool
UNINEXT TO KO
SERGEE
Instruction:
far right baby yellow bib, kids chair
Instruction:
and black coat
UNINEXT
DROD
DROD
UNINEXT
DROD
DROD
Figure 9: Visual comparison results. Our DROD well executes detection instruction compared to Grounding-DINO and UNINEXT.
14

---

Published as a conference paper at ICLR 2024
B EXPRESSION GENERATION VIA LOCAL PROMPT PIPELINE
B. PROMPT AND GENERATION DISPLAY
We show the text prompts we use below in Table 4. For the LLaVA input, We randomly select one text prompt together with one image marked with object bbox. The inputs and samples generated from VA are shown in Fig. 10.
Describe the objects the red box."
Take a look this image and describe What's the red box."
Please provide description of the object the red box."
Could you describe the contents in the red box of the image for me?"
Use one sentence to index the objects in the red box."
"Output a sentence describing the objects in the red box, so that people can locate the objects without ambiguity through this sentence."
Look carefully at the objects in the red box and describe them in one sentence to distinguish them from other objects."
Table 4: Text prompts use for LLaVA inputs.
Rekerinu Object
Beferring Object Pronpts
Destriz objecta
imaae
hoc. Gescrnted Insrrctions
Theee is woren shtin cos-lagee on coach bucne vereer
toeel honging bualruene.
pants
Reierriaa Obbject Frompts
Reforias Cltject Prampo
Creld desenie hes the imugee for mu?
Iedimetiins
Ou vide the preaue is sna wealing
cheracerr in the maage hoide wear paoplo
Referrigg Object Prospts
Reforiar Citject Prampo
Ploae poridce doceggrce the nod bet,
Clepun descnbieg
ddeves bas, poslic localc the obyyce wrihect anthagary thoough this scriceez.
Generated Inetructien
Aosibell bike usvang at bal.
Generated teatractioen
Figure 10: Examples of text prompts, images with marked object bboxs, and LLaVA outputs.
B.2 FINE-TUNING SETTINGS
Dataset Construction We employed the entire dataset from ReiCOCO/+/g for our fine-tuning endeavors, which primarily comprises triplets of images, bounding boxes, and instructions. The detection boxes are overlaid onto the images, serving as the visual input for LLAVA, while the instructions from the dataset used supervised outputs. The prompts from Table 4. The overall dataset consists of 386k instructions paired with 66k images. We chose fine-tuning objective solely directed not utilize additional REC datasets for fine-tuning,
randomly selected
15

---

Published as a conference paper at ICLR 2024
towards enabling LLAVA comprehend our task and acquire the ability to describe targets within red detection boxes.
Firstly, is crucial emphasize that in this paper, LLAVA does refer the model presented in the original LLAVA paper. Instead, it broadly denotes a multimodal paradigm where image information abstracted into several tokens and inputted into a Large Language Model (LLM) for cross-modal attention with textual tokens. Our LLAVA architecture utilizes Q- Q-Former to extract 32 tokens from the VIT features of the image, and these 32 tokens are subsequently mapped to the same feature space as textual tokens through linear layer. During fine-tuning, only the final mapping linear layer trained. This fine-tunin approach enables the model learn the salience of the target object within the red bounding box in the input image while ensuring that the overall model avoids catastrophic forgetting. We initialize our model with the weights of MiniGPT-4.
Training Process
Hyperparameter Configuration and Implementation Details We employed Vicuna 13B Large Language Model (L..M). The batch size and number of epochs were set to 32 and 30, respec- tively. The initial learning rate was 3 x 10-5, Model training was conducted using threads on a single A100 80GB GPU.
B.3 CHAIN-OF-THOUGHT AND REFLECTION
After fine-tuning, the LLAVA model still exhibits a certain degree of hallucinations, manifesting the generation of imaginary objects. Taking inspiration from the concept of thought chains in large language models, we manually constructed thought chain specifically tailored for the task of describing target objects. In Table 5, illustrate the prompts at each step of the thought chain: 1) Inquire about the category of the target object. 2) Inquire about the attributes possessed by the target object. 3) Inquire about the objects surrounding the target object. 4) Inquire about the relationship between the target object and its surrounding objects. Through these four steps, we can attain a comprehensive understanding of LLAVA's perception of the target object. However, hallucinations may still persist in the thought chain; therefore. in step 5), we prompt the model to reexamine the image and correct any errors. Finally, in step 6), we prompt LLAVV to produce the ultimate description of the target object.
1) "What is the object inside the red bounding box?"
2) "What are the attributes of the object within the red bounding box?"
3) "Which objects are around the target object in the red box?"
4) "What the relationship between the object inside the red bounding box and the surrounding objects?"
5) "Please review the image once again, and if there are any inaccuracies in your previous answers regarding the object's attributes and relationships, kindly correct them."
6) "Look carefully at the objects in the red box and describe them in one sentence to distinguish them from other objects."
Table 5: Text prompts for guiding chain-of thouuht and reflection.
16

---

Published as a conference paper at ICLR 2024
C EXPRESSION GENERATION VIA GLOBAL PROMPT PIPELINE
We choose an image from Objects365 as an example to illustrate the instruction generation pipeline via our global prompt pipeline. This pipeline consists of inputs, steps, in-context samples, and LLaMA outputs, which are illustrated one-by-one in the following:
Inputs
Our input is an example image with object bbox coordinates, which is shown in Fig. 11.
category:
"Street lights"
bboxes:
[0.20, 114.75, 19.15, 19.13]
category:
"Fire Truck"
bboxes:
130.94, 307 41, 110.05, 81.51).
(126.45, 298.61, 61, 124.33, 95 95.20],
[225.59, 280.94, 187.80, 114.51].
[379.37, 293.37, 122.69, 102.08),
[494.54, 291.74, 185.18, 98.48)
image
annotations
Figure 11: An input example. The left part is image from the Objects365 dataset. The right part the object category names and corresponding bounding boxes coordinates.
Output We send the final text prompt shown in Table to LLaMA. Table 6 shows the output expression of the inputs shown in Fig. 11. The detailed steps to prepare text prompts illustrated as follows:
[Fire Truck]
(I) vehicle, emergency vehicle, fire engine, parked outside the fire station, an essential part of the fire station's s resources, essential part of the fire station's 's resources
(2) lined up in a neat row, ready for use, object parked in the row with other fire trucks, object with ladders and equipment
[Street Lights]
(1) light fixtures, outdoor lighting, two lights visible, providing illumination, source of illu- mination
(2) objects providing illumination, objects in the parking lot, objects providing clear rep- resentation of the overall setting, objects providing light for the parking lot
Table 6: LLaMA output for the inputs shown in Fig. 11.
Figure 11 and Table 6 depict the original input and final output of the global prompt pipeline, with the intermediary processing steps outined below. Initially, the generation of the desired caption description for images lacking dense captions is required (Step 1). Subsequently, with the image fully textualized, we employ detailed prompt design and seed example design instruct LLAMA in outputting instructions describing the target object (Step 2).
17

---

Published as a conference paper at ICLR 2024
Step We use LLaVA to generate image captions. Table 7 shows the text prompt list use. One random text prompt from this list is combined with object content description (i.c., referring expression category name) for LLaVA imput. For each image, send these prompts into LLaVA, twice and obtain two diverse text descriptions. These two text descriptions are both utilized in the following steps. Examples of generated text descriptions are shown in Table 8.
Describe the following image in detail"
Provvde a detailed description of the given image"
Give an elaborate explanation of the image you see"
Share a comprehensive rundown the presented image"
"Offer a thorough analysis image"
EEpplai the various aspects of the image before you",
"Clarify the contents of the displayed image with great detail"
"Characterize the image using well-detailed description"
Break down the elements of the image in detailed manner"
"Walk through the important details the image"
"Portray the image with rich, descriptive narrative"
"Narrtee the contents of the image with precision"
"Analyze the image in comprehensive and detailed manner"
"Illustrate the image through descriptive explanation"
"Examine the image closely and share its details"
"Write an exhaustive depiction of the given image"
Table 7: Text prompts we use for LLaVA. These prompts follow the author usage in the original LLaVA work.
Prompt for LLaVA
Provide a detailed description of the given image, including objects: Street lights, Fire Truck
Output from LLaVA
Output 1st The image displays parking lot outside fire station where several fire trucks are parked. total, there are five fire trucks of varying sizes, all lined up neatly in the lot. Additionally, there are two street lights visible in the scene, providing illumination for the area. The fire trucks appear ready for use and are an bessential part of the fire station's s resources.
Output 2nd The image depicts a parking lot outside fire station, where several fire trucks parked in neat row. There ready for use. In addition to the fire trucks, there are two street lights visible the scene, providing illumination for the parking lot. The overall setting gives clear representation of the organized and prepared nature of the fire station.
total of five fire trucks of varying sizes, all aligned and
Table 8: Generated examples of L.LVVA based random text, prompt from Table 7.
We write text prompt for LLaMA. In this step, the text prompt contains three parts including task descriptions, in-context examples, and text description of image (i.e., Table 8). An example text prompt for LLaMA is shown in Table where in-context sample shown in Table 10.
Step 2
18

---

Published as a conference paper at ICLR 2024
task description f"n
Task description prompt
## Establishing fundamental roles and task localization for LLM.
You are an Al visual assistant that can analyze a single image. ## Elucidating the form and significance of input information.
User will give you several sentences, describing the same image you are observing. In addition, specific interested object locations within the image are given, along with detailed coordinates. These coordinates are the form of bounding boxes, represented as (xl, yl, x2, y2) with floating numbers ranging from 0 to 1. These values correspond to the left top left top y, right bottom x, and right bottom y. ## Explicating the output content and the encapsulated information.
Using the provided caption and bounding box information, give descriptions about the visual content of each interested objects as if you seeing the image, assistant:
(1) give descriptions about the object itself, including object types, object functions, object counts, object locations, object actions, etc. (2) give descriptions about the object and other objects, including the relative positions between objects, the interaction between objects in the image, etc.
## Emphasizing common output issues.
Descriptions should be series of phrases, not whole sentence. Give descriptions for specific interested objects only, not centered on other objects. Again, give descriptions centered on specific objects only.
image2text g"""
# Image description of LLaVA. the following two paragraphs are from Table 8
The image displays parking lot outside fire station where several fire trucks parked. In total, there are five fire trucks of varying sizes, all lined neatly in the lot. Additionally, there are two street lights visible in the scene, providing illumination for the area. The fire trucks appear ready for use and are an bessential part of the fire station's 's resources.
The image depicts parking lot outside fire station, where several fire trucks are parked in neat row. There are total of five fire trucks of varying sizes, all aligned and ready for use. In addition to the fire trucks, there are two street lights visible in the scene, providing illumination for the parking lot. The overall setting gives a clear representation of the organized and prepared nature of the fire station.
Street lights: [0.0, 0.23, 0.03, 0.26]
Fire truck: [0.05, 0.6, 0.21, 0.76), [0.19, 0.58, 0.37, 0.77], [0.33, 0.55, 0.61, 0.77], [0.56, 0.57, 0.74, 0.77], [0.72, 0.57, 1.0, 0.76]
# Python code together with above text prompts are directly sent to LLaMA
messages [["role"; system, "content": task_ descriptionn]
for in context Lenteex_eexampeeeeeeeeee in context.........ooo wowtextt_exmppeess. messages aassaees.apeend append roolee: user, "content": __cootext__xxmpplee
content "1])
messages append (( ("role": assistant, "content":
in context example ['response]]]
messages append (( "role": user, "content"; image2text))
Table 9: An example of prompt generation for LLaMA input. For each image, we first obtain text description in Table Then, we organize the above prompts and python code, together with in-context examples for LLaMA input. An in-context example is in Table 10.
19

---

Published as a conference paper at ICLR 2024
Example image2text
Two children, girl and boy are practicing their writing. Two children sit on a small seesaw in the sand.
Two children sitting on teeter totter.
2 kids playing on seesaw.
Two kids on seesaw.
2 kids/two kids/two children: [0.09, 0.59, 0.49, 0.94]
girl: [0.09, 0.59, 0.23, 0.92]
boy: [0.34, 0.62, 0.49, 0.94] scesaw/small seesaw/teeter totter: [0.06, 0.83, 0.57, 0.93] sand: [0.01, 0.61, 1.0, 1.0]
Example Response
[2 kids/two kids/two children]
(1) persons, children, two kids, children learning, kids playing, kids sitting, children practicing their writing
(2) 2 kids playing on seesaw, Two kids sitting on a seesaw, girl and boy, two person on the teeter totter, kids by the sea [girl]
(1) girl, kid on the left of the image, girl sitting, girl playing
(2) kid playing with the boy, girl sitting the small seesaw, girl playing in the sand, girl reading a book, a friend of the boy on the right, the taller kid [boy]
(1) boy, kid on the right, boy sitting, boy playing, boy practicing his writing
(2) kid playing with the girl, boy sitting on teeter totter, friend of the girl the left, boy playing in the sand [seesaw/small seesaw/teeter totter]
(1) small seesaw, teeter totter, item to be played on, common facilities in parks and playground, game of two people
(2) seesaw in the sand, item the kids are sitting item the girl is sitting on, item the boy is playing on [sand]
(1) common by the sea, the background of the scene
(2) item on which the seesaw is placed, item on which the kids are standing
Table 10: An in-context example. We manually prepare three in-context examples and show one here. On the top block there are image captions and object bbox coordinates. Note that the displayed image is only for image caption reference and is not used in practice. The bottom block shows our expected expressions for each object. The expressions listed in (1) focus on the object attribute itself, and listed in (2) focus on relationship between current object and other objects this image.
The in-context example in Table 10 follows the task description in Table and primarily serves to standardize the output format of LLAMA. The desired output format intended to resemble the Example Response presented in Table 9. [name] denotes the target currently being described, where the descriptions (1) are relatively simple, pertaining only to the inherent attributes, and those in (2) entail more complex descriptions involving surrounding objects. This facilitates accu- rate correspondence between each description and the object it pertains to when parsing the output descriptions.
noteworthy that the prompt design is manually crafted. It draws significant inspiration from the prompt framework used by LLAVA, with specific modifications and supplements tailored to the requirements of our task. The entire prompt design underwent numerous iterations based on the output results, ensuring the stability of the output format and style.
20

---

Published as a conference paper at ICLR 2024
D
VISUAL PROMPTING AND VISUAL-TEXTUAL MATCHING
fr(-)
Reverse
SAM
Target Bounding Box
Visual Prompts Overlay
Red Cirele
X
fc(-)
Referring Expression
Y
CLIP
blonde lady wearing long white scarf
CLIP
Prompi Enhancenent Score
Sg
Global Score
S.
S S
Local Prompt Score
S
|
a1
Final Score 5
A detailed pipeline of visual prompting and visual-textual matching. We use visual prompting to emphasize target object in the image. Then, we use the CLIP model to compute the scores for expression filtering
Figure 12:
Motivated by recent applications (Yang et al., 2023; Shtedritski et al., 2023) of visual language pretraining (VLP) models in solving zero-shot REC tasks, we discover that coupling VLP with visual prompting enables robust pairing of image regions and text the basis of potent generalization. The efficacy pairing is largely influenced by the configuration of visual prompting, which can be divided into two types as follows:
(i) Shape division emphasized areas: box, circle and contour.
(ii Highlighting methods for emphasized areas: crop-based, line-based, mask-based, grayscale version and blur reverion based image editing operations.
As shown in Fig. 12, after various combinations of experiments, find the superposition of line- based circle prompt and blur reverse contour prompt yields the optimal visual prompting strategy. The image X after visual prompting can be expressed as:
X (X){ with 5₂(-) BR(: SAM(, B)) fo(-) = RC(: B)
where B represents the box coordinates of the target, BR and RC' represent the reverse Gaussian blur for the mask area and the inscribed ellipse for the target bounding box, respectively. SAM de- notes Segment Anything Model (Kirillov et al., 2023), which makes visual prompting more refined.
Once obtain image and visually prompted X, send both of them to the CLIP model together with the generated expressions Y from global prompt and local prompt pipelines. Then, we can compute the global score S, and local score S,, and thus obtain the final score Sy illustrated in equation for expression filtering.
21

---

Published as a conference paper at ICLR 2024
E INSTRUCTION GENERATION FOR MINING CLUSTERS OF MULTI-OBJECTS
In order to find and summarize the common attributes among multiple objects in the clusters from DBSCAN (Ester et al., 1996), use LLaMA for further analysis. Table 11 shows the text prompt of task description and in-context examples we use for the LLaMA inputs. Then LLaMA produces expressions for qulti-objects. as shown Fig.
Task Description
You are an AI language assistant that can analyze phrases and sentenses. User will give you descriptions of several objects in an image. Descriptions of each object are several phrases or short sentences.
The given objects expected to have similar properties. Based on the descriptions. find the common properties between given objects and summerize precisely as an assistant: common properties between objects include same types, same functions, same color components, same poses, same relationships with other objects, engaging in the same activity, etc.
there are no common properties between given objects, just tell that there are no common properties. Your summery should also be phrases. Do repeat.
Give similarity between all given objects, contrary properties like different positions or different colors of clothes should not be included together in your descriptions.
One In-context Example
Prompt: Objects and their descriptions:
## object 2: girl sitting bed, girl with toy, girl sitting on bed ## object man looking down, boy sitting on the bed, man sitting on bed Please find an summarize the similar properties of given objects.
Response: Summary of common properties of given objects: ## people bed; person sitting on bed; people playing bed; who sitting bed;
Table 11:
Task description and an in-context example for multi-objects instruction generation.
22

---

Published as a conference paper at ICLR 2024
F INSTRUCTION GROUPING
We use LLaMA to analyze different extents of description based on object category, attribute, and relations. For each instruction of single object in the InDET dataset, we use LLaMA assign into one of the preset groups. Table 12 shows the text prompts of task description and in-context examples we use for instruction grouping. For instructions of multiple objects, we assign them to GS there is the combination (e.g., "and") of single instructions, or we assign them to G6 if the instructions are generated without concatenation.
Task Description
You are an Al language assistant that can analyze the language complexity of sentences or phrases.
User will give you phrase sentence describing specific object in image, which could be composed nouns, adjectives, verbs, etc.
Grade the description according to its language complexity Al language assistant.
The language complexity of phrase sentence describing specific object in an image can be graded into four levels:
Level 0. A single noun is used to give the object's name or type.
Level 1. A phrase with one or more nouns, verbs and abjectives is used to describe simple attributes of the object itself, such as its color, its function or purpose, location in the image, or actions.
Level A phrase with one more nouns, verbs and abjectives is used describe the object by referring other objects in the image and describing their relationship, such as their relative positions or interactions.
Level 3. A long phrase sentence is used to describe attributes of the object and also refer to a few other objects detail, or describe a complicated or comprehensivv/implict relationship between multiple objects.
The level of descriptions increase as the language complexity and length increase, and also increase the phrases sentences become more descriptive and more abjectives and nouns to describe the object.
One In-context Example
Prompt: Grade description: people who are sitting under an umbrella.
Response: My grading for description people who are sitting under an umbrella: This phrase referring to the object of people, and gives simple object action sitting and object rela- tionship with the umbrella. The level of this description is: level 2.
Table 12: Task description and in-context examples for single-object instruction grouping.
23

---

Published as a conference paper at ICLR 2024
G MODEL AND IMPLEMENTATION DETAILS
In this section, we illustrate our model architecture and training configurations.
G.1 MODEL ARCHITECTURE
Fy
blonde lady wearing
Bert
y
Confidence Loss
Localization Loss
long white scarf
Pool MLP
Fy
Cross
Attention
F
Res/ViT
Transformer Encoder
Transformr Decoder
F²
F
Figure 13: A detailed overview of our diversified referring object detection (DROD) model.
In the main text, We briefly introduced the DROD model architecture we use. Here, we introduce the overall model structure in Fig. 13.
Visual Textual Feature Fusion We use general text encoder (Devlin et al., 2018) and vision encoder (Dosovitskiy et al., 2021) to obtain text features Fy and image features Fz. In order to im- prove the attention of visual contents on the described regions by the text expressions, we conduct multi-modality feature fusion. Specifically, we use a bidirectional cross attention (Bi-XAtt) module to enhance image and text features through information transfer between modalities. The enhanced image and text features are then added to the original features. This process can be formulated as follows:
= Fzzu: Fy
Fy
Fx2y
with Fgzu Fazy Bi-XAuu(F2,F,
Target Discovery and Retrieval With the enhanced visual and language representations, we need to extract the targets referred to by the expression from the image features. Our DROD model applics the encoder-decoder architecture of Deformable DETR (Zhu et al., 2020), which allows for more flexible query retrieving. Here we provide a detailed introduction to our internal module.
The transformer encoder receives multi-scale visual features after text enhancement. Multi-scale Deformable encoder utilizes flexible attention at different scales and spatial shapes to obtain hier- archical characteristics for instances. In addition, following the design of two-stage Deformable DETR, we add an auxiliary prediction head for reference points. The top N points are input into the decoder the position prior.
The Transformer decoder uses learnable queries to retrieve instance-related information from en- coded multi-scale visual features. The design of the query is critical for ensuring stability and efficiency in training, based on previous works (Meinhardt et al., 2022; Wang et al., 2021). The N reference points served by encoder act the position priors of the N queries. The content part of each query is a static learnable vector. Moreover, following DINO, we add denoising queries to make the decoder's convergence more stable and faster. Through Deformable attention, N queries efficiently retrieve instance embedding F: IR from expression-awee visual features.
Finally, we need select the instances referred to by expression from the N instance proposals. As shown in Fig. 13, we use global average pooling and MLP to map the visual-aware expression feature F to the semantic space of the query embedding to obtain F, I Thus, the matching
24

---

Published as a conference paper at ICLR 2024
score between each query proposal and the expression can be expressed by the cosine similarity
between vectors. S F, Fy In the inference stage, proposals with scores above threshold A are selected. This flexible selecting strategy allows our architecture to output any number of results that satisfy user requirements. In the training stage, the overall model is supervised by combination of confidence loss and localization loss.
G.2 TRAINING CONFIGURATIONS
Generation Engine ay in the expression filter responsible for balancing referentiality and seman- tic accuracy is set to 0.5. While generating multi-target expressions, DBSCAN clustering method has two hyperparameters: neighbourhood radius eps is 1.5 and minimum neighbors of the core point minPts is 2. The temperatures of LLaMA are set to 0.7.
DROD Model We use ResNet-50 (He et al., 2016) and ViT-Huge (Dosovitskiy et al., 2021) as visual encoders and Bert (Devlin al., 2) as the encoder. The transformer encoder-decoder architecture consists of a six-layer encoder and a six-layer decoder. The number of object queries N set to 900. Our DROD model is initialized by weights pretrained on Objects365 released by UNINEXT (Yan et al., 2023). The optimizer we use is AdamW Loshchilov & Hutter (2019) with a learning rate of 2e-4, weight decay of 0.05 and the warm-up steps are 400 with an initial learning rate of 4e-5. The model is trained on 32 and 16 V100 GPUs for pretraining and finetuning, respectively.
25

---

Published as a conference paper at ICLR 2024
H ABLATION STUDIES
In this section, we study the necessity of finetuning LLaVA during expression generation from local prompt pipeline. Then, investigate how different visual prompting strategies affect the CLIP model to maintain our expected expressions.
LLaVA Finetuning In our InstructDET, we do not finetune LLaMA in global prompt pipeline, but finetune LLaVA in local prompt pipeline. There are two main reasons illustrated as follows:
No Finetune
With Finetune
Good Instructions Ratio
11.4%
80.2%
Cuapar cernace dacotbiig the ebjure the people kekkkbkkkk wille untiggutty tmough actience
Original
nu
Artan cundin outs arpinao ground
vias body wit precn mrrpps the wigss There gaicn hackpood
TBD 22
REC Finetuned Wrights
Amsizcyd: diree vitk
Could desuriee in hot
Original Wiights
age living
sakuni lugh. The Eocring is caade hardleed and thene les
und plana comer The diiiwanker The nome ypaiinn farralle
foac. The ramn velldl
Weights
Dhmakdooe of thec ohanal becrnc thradh
Figure 14: Comparison expressions generated by LLaVA before and after our finetuning. Blue words indicate incorrect descriptions.
(i) Asking questions directly to LLaVA will get the answers with dense description, rather than the exact expressions we expect for the specific target objects.
(ii) Due to the limited number of visual tokens output by Q-Former, it is difficuul to fully display all visual features the entire image. With limited visual information, LLaVA produces lengthy descriptions and thus lead to massive hallucinations (blue words in Fig. 14). The generated expres- sions in this way becomes unstable.
Based on above analysis, partially finetune LLaVA the linear layer that that transforms visual features into the semantic text embedding space. To this end, the model updated to understand which area aim emphasize (i.e., marked in the red box in Fig. 14) and which visual features are important for target object description. Fig. 14 shows that after finetuning the linear layer using RBC datasets, the probabilities of generating our expected high-quality expressions increases from 11.4% to 80.2%.
Visual Prompting Selection The selection visual prompting (VP) is crucial for expression filtering. We evaluate the affect of various visual prompting methods based on the retrieval perfor- mance of CLIP model in Table 13. First, we define new metric called expression retrieval ratio, which indicates the proportion of correct expressions that can be retrieved based on the visual-textun matching of CLIP model. During testing, each minibatch contains correct target expressions, and the remaining expressions negative samples. We take the expressions with the top Clip score
26

---

Published as a conference paper at ICLR 2024
Table 13: Visual Prompting (VP) Evaluation. Different VP methods make differences in the ability of CLIP to retrieve high-quality expressions.
Visual Prompting Shape
Expression Retrieval (%)
Strategy
Tool
Easy t
Hard
Crop
31.52
24.09
Gray
39.66
31.85
3
Box
Line
50.12
42.30
Mask
48.29
39.93
5
Blur
52.75
44.30
Line
52.78
44.33
Circle
Mask
51.01
42.87
8
Blur
54.13
46.22
Line
52.34
44.05
Contour
Mask
51.90
43.44
Blur
55.79
47.78
12
VP3 + VPII
56.81
48.97
VP6 VP11
58.29
50.99
the retrieved expressions. Finally, the proportion of correct expressions in the retrieved expressions the expression retrieval ratio. Under the Easy setting, the negative samples in the minibatch come from other images. Under the Hard setting, the negative samples in the minibatch may come from different targets in same image. Table 13 shows that cropping (1) and grayscale reversion (2) methods achieve poor results. Because cropping loses all surrounding environment information, and grayscale reversion loses all color features. The best single prompt is the contour blur reversion. The best combination prompt is the contour blur reversion and circle line. Circle line can indicate the rough areas that needs attention, and contour blur reversion can highlight the target object in a fine- grained manner that eliminates background interference. Inevitably, the prior knowledge in the CLIP model also important to the difference in visual prompt effects. In CLIP's pre-trained web-scale dataset, images with red circles often indicate that the targets in the red circles more important and need to be noticed. A large amount of photography also exists in the web-scale dataset, which employs "Bokeh" blur the background and highlight the subject.
27

---

Published as a conference paper at ICLR 2024
I SYNONYMOUS REWRITE
In post processing, we utilize LLLMM to do synonymous rewriting to further diversify the generated expressions. The prompt we used is shown in Table 14.
Synonymous Rewriting Prompt
I want you to act as synonymous expression provider. I will give you text of phrase short sentence, which an expression that describes main object while mentioning some other objects. And you will reply to me with meaning and describe the same main object as the provided expression. The new expressions should also be phrases or short sentences no longer than 25 words. Do write explanations on replies. Do not repeat.
expression that have the same semantic
Table 14: Synonymous Rewriting Prompt for LLaMA.
J SUPPLEMENT COMPARISONS
J.1 EVALUATION ON INDET
Table Evaluation results on our InDET, We show the object bbox average precision (AP) values
(%).
AP Group
Method
Backbone
AP
G1
G2
G3
G4
G5
G6
Re<Nei101
47.44
23.22
25.91
SwinB
47.10
26.84
27.95
UNINEXT
ResNet50
43.37
54.49
54.52
44.49
37.17
31.41
32.01
DROD (Ous))
ResNet50
62.24
67.14
67.34
60.89
55.10
70.15
74.22
DROD (Oun))
ResNet50
62.34
67.22
68.04
61.09
55.42
68.60
72.91
DROD (Ours)
VIT-H
66.90
72.53
72.47
66.42
59.86
73.34
75.92
For comparison, our DROD model Table only utilizes Re/COCO/+/g datasets but with diversified instructions, which partial of InDET.
2 Here we add evaluation results DROD model trained on full InDET
J.2 EVALUATION ON REFCOCO/G/+
Table 16: Supplementary Evaluation results on the RefCOCO/g/+ datasets. We follow evaluation protocols report AP values (%) of comparing methods. We use the notations "CC", "VG", "OI", "0365", "RIGame", for COCO, Visual Genome, OpenImage, Objects365, ReferltGame, respec- tively.
RefCOCO
RefCOCO+
RefCOCOg
Method
Backhone
Data
val
lestA
lestB
val
lestA
lestB
val-u
test-u
RefTR SegTR
ResNer101
VG
85.65
88.73
81.16
77.55
82.26
68.99
79.25
80.01
DarkNei53
VG,RIGame.FFie Reic
87.00
90.15
83,59
78.69
84.51
71.87
82.69
83.37
MDETR
ResNet101
Re.fG,CC.RefC
86.75
89.58
81.41
79.52
84.09
70.62
81.64
80.89
G-DINO
SwinB
Da65.OC.RetC.GoolGG.a
83,95
87.79
79.16
72.91
80.91
62.96
76.98
76.76
PolyFormrr
Swinl.
GoldG.
Referli,RRet..
90.38
92.89
87.16
84.98
89.77
77.97
85.83
85.91
UNINEXT
5e
O3fF.CC
87.64
90.35
83.49
78.14
83.22
68.71
80.96
81.86
(Ours)
C..D,CC,InDET
85.57
83.39
71.04
82.91
DROD (Ours)²
ResNet50
0365.CC..DET
89.85
92.03
87.24
80.50
85.87
73.61
83.93
84,73
DROD (Ours)
VIT-11
0365.CC.mDET
92.93
94.47
91.13
86.20
89.82
80.86
88.62
89.46
For fair comparison, DROD model in Table only utilizes RefCOCO/4/g datasets but with diversified instructions, which part of InDET.
add evaluation results DROD trained full
DROD here with VIT-H as backbone also utilizes Re1COC0//g datasets with diversified instructions.
We have supplemented the comparisons with specialized models, such Polyformer(Lin et al., 2023b). With comparable magnitudes in the backbone network, our model continues to exhibit advantages.
28

---

Published as a conference paper at ICLR 2024
K
TIME CONSUMPTION
Table 17: Time Consumption Statistics. We use "instr:" as the short for "instruction"
Pipeline
Step
Main Model
BatchSize
Tine
Avg
Image Caption
LLaVA
10:71 a/butch
2.68 shimage
Global Prompt
Instr. Generation
LLaMA
17.25 s/image
0.63 s/instr.
Local Prompt
Instr. Generation
LLaVA
2.47 s/aath
0.15 sinstr.
filtration
shmage
0.016
Post- Processing
Multi-Object Instr.
DBSCAN
shmage
Instr. Grouping
LLaMA
1.43e-06 s/instr.
Table 17 shows the time consumption of the main steps in our instruction generation procedure. The machines. Obviously, the most time consuming steps are
data is obtained from those who depend on large models. Except for the limited hardware performance, another reason why the instruction generation step global prompt pipeline is very time consuming that have very long LLaMA prompts which include three in-context examples and we also desire long response which could include more diversified instruction for each object in image. So this step requires multiple machines to generate simultaneously. While the text received and generated in each forward pass relatively short the local prompt pipeline, the generation efficiency significantly constrained by the fact that only address this limitation by maximizing parallelism through the increase in batch size, allowing the simultaneous generation of multiple instructions.
instruction can be generated per forward pass. Therefore,
29


Reviews:
[{"review_id": "3mzrxlY7aS", "reviewer": "Reviewer_Vtb3", "summary": "The authors propose InstructDET for referring object detection (ROD) that localizes objects based on user instructions. They leverage diverse instructions generated by vision-language models to create the InDET dataset. Based on InDET, their InstructDET outperforms existing methods on referring expression comprehension datasets and their own InDET test set.", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- As the authors claim, InDET is the largest real-world REC dataset. It is expected that InDET will push the study of REC and its application.\n- The dataset covers many real-world scenarios, including multiple object instruction.\n- The proposed DROD model is simple yet effective. And it helps prove the quality of InDET.", "weaknesses": "- More experiments are expected. For instance, previous REC models should be trained on InDET and evaluated on RefCOCO/+/g.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "avGNkc4y4f", "reviewer": "Reviewer_uAXw", "summary": "This paper introduces InstructDET as new method for referring object detection (ROD). The paper identifies the lack of diversity in the user prompt in existing ROD datasets as a critical limitations. To address this limitation, a data pipeline based on query of foundation models, such as LLaVA and LLaMA is built. This pipeline creates accurate and diverse object reference via a combination of prompt engineering and heuristics. It first create object and image level expressions via LLaVA, then reject hallucination via CLIP, and finally diversify the expressions via LLaMA. The resultant expressions can then serves as training data for the ROD task. The paper shows that the dataset curated from the proposed procedure can significantly improve the model performance.", "questions": "* Could you include more details in terms of the prompts that are use, the exact procedure to generate them (number of queries, the services used, costs etc.)? I think these are interesting information for the community that should be included to the paper wherever appropriate.\n\n* Do you plan to release the dataset curated for this paper?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* This paper provides an interesting research direction in referring object detection. While many existing works attempt to improve ROD via better model design, this work suggests that it could be fruitful to enrich existing dataset via a combination of data mining and/or prompt engineering from existing foundation models. It would be of great interests to the community in general.\n\n* The method taken from this paper seems to quite novel. While the use of foundation models as a data curation tool is not new (as fully acknowledged in this work, and is the practice used in LLaVA which is in return used heavily in this work), this work has explored some novel ideas to improve this procedure. For example, the \"vision and language mapping\" via CLIP as a verification step is particularly interesting.\n\n* The resultant model trained on the curated dataset seems to outperform prior arts by a large margin, even with the same network architecture.", "weaknesses": "* There are many language errors throughout the submission, including strange use of words, grammatical mistakes etc.. It is still possible to comprehend the ideas despite those writing issues, but this submission can use additional proofreading. Part of the issue is with the naming of concepts. The \"pathways\" are easily confused with the terminology used to describe parallel branches in a deep network. The \"dropout\" can be confused with the regularization method. These names are unfortunate in the sense that they are in fact significant distractors for readers in this field, and not really descriptive of the concepts.\n\n* I find the definition of \"single\" versus \"multi\" modality pathways confusing. To my best understanding, the \"single modality pathways\" refers to a sub-process in the data generation that uses LLaVA to create output texts descriptions for an input image. As such, it needs both an image and some text prompts to obtain text result. The \"multi modality pathways\" refers to a procedure that similarly use LLavA to create results from image & text prompts. The two pathways have the same form of inputs and outputs. Why one is \"single modality\", while the other is \"multi modality\"? It is also hard to see, at least from the discussions provided in the submission, why the \"single modality\" case can use an out-of-box LLaVA model, whereas the \"multi-modality pathways\" must use a finetuned version.\n\n* While the main contribution from this paper is the proposed data generation step procedure, the details are not included in a way that would allow other researchers to reproduce the results. For example, Section 3.1 (combined with appedix C) describes the \"single modality\" pathway. However, critical details such as the procedure to generate prompts are left in a rather abstract level. I don't think it would be possible for reader to re-produce the method described. Similarly, in Section 3.2. the paper does not precisely describe the procedure. For example, while the paper mentions fine-tuning LLaVA on some REC dataset, it does not describe the exact dataset nor the exact training setting for this. Another example is in the \"post-processing\" section. It is not at all clear from reading the paper how the expressions are \"verify and remove\", and how LLaMA is used to \"diversify\" the expressions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mVYCJVSzgt", "reviewer": "Reviewer_nJxy", "summary": "The paper introduces \"InstructDET,\" a novel method for referring object detection (ROD) that identifies and localizes target objects in images using user instructions. This approach is an advancement from the traditional referring expressions (REC) and places emphasis on generating a wide variety of user-centric instructions linked to object detection tasks using VLM and LLMs\nThe InstructDET dataset is a compilation of images, their associated bounding boxes (bbxs), and the generalized instructions curated from VLM and LLM. Models trained on the InDET dataset demonstrated enhanced performance, outclassing existing methods on traditional REC datasets and the new InDET test set. InstructDET showcases the potential to expand by integrating any image accompanied by object bounding boxes.", "questions": "Can you please address weakness 1?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper proposes a novel and interesting way to produce rich visual descriptions of objects and their relationships using LLM and VLM\n2. I like the approach for removing hallucination from the VLM model using vision language matching for CLIP.\n3. This approach is scaleable which means that the results can continue to improve with more bbox data.", "weaknesses": "1. It's not clear from the paper that the method comparison in table 2 is fair because it's not clear if the models were trained on the same images and bounding boxes\n2. One additional interesting experiment that would make this a stronger paper - run SAM on web scale images to get bboxes for images and then scale up this approach further to understand if the benefits saturate", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qPbDEOP8c1", "reviewer": "Reviewer_n3DU", "summary": "This paper introduces a new grounding dataset, which is created with LLM (LLAMA) and VLM (LLAVA). They propose to leverage LLAMA in the single modality pathway, which receives the text only descriptions and bounding boxes to create annotations, in the same spirits to LLAVA. And LLAVA is leveraged in the multi modality pathway. After creating the dataset automatically, they leverage CLIP to drop out the annotations with low quality. A simple baseline model has been proposed to showcase the effectiveness.", "questions": "See weakness", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The proposed dataset may be a good resource for future research in visual grounding tasks.", "weaknesses": "Weaknesses: \n1. The major concern I have is regarding the quality dataset. The author uses the open-source model LLAMA, which is not instruction-tuned, in the single modality pathway. As opposed to GPT4 used in LLAVA’s instruction tuning dataset. In addition, LLAVA is used in multi-modality pathways. I have doubts about the quality of the dataset since both LLAVA and LLAMA have problems of their own, such as hallucinations and incapability to describe the objects of small scales.\n\n2. The experiments are insufficient. The author only presents the result in their own proposed dataset and the original ref coco benchmark. I am not certain how much the dataset actually helps the performance of the model, or helps equip the model with new capabilities. Specifically, the performance on refcoco is not extraordinarily high and is not compared with SOTA methods such as poly former.\n\n3. What does the author mean by “initializing LLAVA with minigpt4 weights”? I think LLAVA finetunes the entire LLM, while minigpt4 only finetunes a linear layer, and the architectures are different.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)