Ferret: Refer and Ground Anything Anywhere at Any Granularity
Authors: Haoxuan You
Haotian Zhang
Zhe Gan
Xianzhi Du
Bowen Zhang
Zirui Wang
Liangliang Cao
Shih-Fu Chang
Yinfei Yang
Venue: ICLR 2024 spotlight (2024)
Keywords: Ferret
Multimodal Large Language Model
Referring
Grounding
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Ferret, a new Multimodal Large Language Model (MLLM) capable of understanding spatial referring of any shape or granularity within an image and accurately grounding open-vocabulary descriptions.

Abstract: We introduce Ferret, a new Multimodal Large Language Model (MLLM) capable of understanding spatial referring of any shape or granularity within an image and accurately grounding open-vocabulary descriptions. To unify referring and grounding in the LLM paradigm, Ferret employs a novel and powerful hybrid region representation that integrates discrete coordinates and continuous features jointly to represent a region in the image. To extract the continuous features of versatile regions,  we propose a spatial-aware visual sampler, adept at handling varying sparsity across different shapes. Consequently, Ferret can accept diverse region inputs, such as points, bounding boxes, and free-form shapes. To bolster the desired capability of Ferret, we curate GRIT, a comprehensive refer-and-ground instruction tuning dataset including 1.1M samples that contain rich hierarchical spatial knowledge, with an additional 130K hard negative data to promote model robustness. The resulting model not only achieves superior performance in classical referring and grounding tasks, but also greatly outperforms existing MLLMs in region-based and localization-demanded multimodal chatting. Our evaluations also reveal a significantly improved capability of describing image details and a remarkable alleviation in object hallucination.

Published as a conference paper at ICLR 2024
FERRET: REFER AND GROUND ANYTHING ANY- WHERE AT ANY GRANULARITY
Haoxuan You't, Haotian Zhang"1 Zhe G Xianzhi Du², Bowen Zhang Zirui Wang", I Liangliang Cao', Shih-Fu Chang', Yinfei Yang* CColubia University, ?Aplee AI/ML hacxuan, youluss cowumbi edu, (haotian, zhangg, zhe. gan, sannyyineeiryappllee..oo
Input Referring
Ferret
Output Grounding
What animal Is in area (raglond]?
[boxO].
boxt
boxo
Cool, what is the animal [region1]? Are they the same one?
No, the animal [borf]] is ferret, which is differant ane from the one (baxd]. They are not the same ferret.
How to make the food [regronO] with available ingredients?
Boxst
To makee a sandwich [box0] with available ingredients,
pnloci
1. You can propare the sandwiches placing the sliced turkey [box]] on one bread, and the sliced cheese [box2] another.
2. assamble sandwiches,
the two halves together
box
box5
the sandwich together, [bax3] your hands [baxd] needed. Once you have complete sandwich, you can usa the knife to cut into quarters or halves, and serve on plate [oox5].
you
Figure 1: Ferret enables referring and grounding capabilities for multimodal large language model
(LLM). In terms of referring, user can refer to a region or an object point, box, or any free-form shape. The regionN in the input will be replaced by the proposed hybrid representation before being fed into the LLM. In terms of grounding, Ferret is able to accurately ground any open-vocabulary descriptions. The boxN in the output denotes the predicted bounding box coordinates.
ABSTRACT
We introduce Ferret, a new Multimodal Large Language Model (MLLM) capa- ble of understanding spatial referring of any shape or granularity within image and accurately grounding open-vocabulary descriptions. To unify referring and grounding in the LLM paradigm, Ferret employs novel and powerful hybrid re- gion representation that integrates discrete coordinates and continuous features jointly to represent region in the image. To extract the continuous features versatile regions, we propose a spatial-aware visual sampler, adept at han- dling varying sparsity across different shapes. Consequently, Ferret can accept diverse region inputs, such as points, bounding boxes, and free-form shapes. To bolster the desired capability of Ferret, we curate GRIT, comprehensive refer- and and-ground instruction tuning dataset including 1.1M samples that contain rich hierarchical spatial knowledge, with 95K hard negative data promote model robustness. The resulting model not only achieves superior performance in clas- sical referring and grounding tasks, but also greatly outperforms existing MILMS in region-based and localization-demandee multimodal chatting. Our evaluations also reveal a significantly improved capability of describing image details and remarkable alleviation in object hallucination. Code and data are available at https://gthub..om/aapll/mmiiif
Work done during internship at Apple. Equual contribution.

---

Published as a conference paper at ICLR 2024
INTRODUCTION
In vision-language learning, how to enable spatial understanding in models is a fundamental research problem. Two desired capabilities stem from this problem: referring and grounding. Referring demands that the model can accurately comprehend the semantics of specific given regions (Krahmer
& Van Deemter 2012 Kazemzadeh al. 2014] Mao et al. 2016 Yu al.) 2016; Zellers et al.
2019), whereas grounding necessitates that the model to localize the region in accordance with the
given semantic description (Luo & Shakhnarovich 2017: Nagaraja al., 2016: Yu al. 2017
Kamath et al. 2021).
Essentially, referring and grounding demand the same type of knowledge: alignment of spatial information and semantics. Despite this, existing works mostly learn referring and grounding indi- vidually (Li et al. 2022; Wu 2022 Yu et al. 2017). In comparison, humans can learn from one task and generalize the shared knowledge to the other task effortlessly, and are able to seamlessly integrate
capabilitiis with daily dialogue and reasoning (Zellers et al. 2019). Inspired by the above gap. in this paper, we study three main questions: (i) How to unify referring and grounding in one framework, and will they benefit each other? (ii) How to represent versatile types of regions that humans usually use for referring, such point, box, scribble, and even free- form shapes? (iti) How to make referring and grounding open- apen-vocabulary instruction- followin,, and robust, which are crucial for practical applications?
Targeting these three questions, we introduce Ferret, a novel refer-and--round Multimodal Large Language Model (MILMM. First of all, choose MILLM the bedrock of Ferret to leverage their powerful vision-language global understanding capability (Zbu et al.] 2023a Liu et al.. 2023b: Li al. 2023c). To unify referring and grounding, Ferret first represents the coordinates regions in natural language numerical form as illustrated in Figure 3 However, it is inefficient to use single point or box coordinates to represent versatile shapes of regions, such as strokes, scribbles, or com- plex polygons. These shapes are essential for more universal and precise human-model interaction. To solve this problem, we further propose a spatial-aware visual sampler to acquire the visual fea- tures for regions in any shape, taking care of the varying sparsity in those shapes. Then, the discrete coordinates and the continuous visual features are combined together to represent the visual regions in the input, composing hybrid region representation in Ferret. Equipped with above methods, Ferret can deal with input that mixes referred regions with free-form text, and is able to ground the mentioned objects in its output by seamlessly generating the coordinates for each groundable object along with generating text. To our best knowledge, Ferret is the first work that is able to process free-formed region inputs in MLLMs.
In order to make the refer-and-ground capability in Ferret open-vocabulary, instruction-following and robust, we collect GRIT, Ground-and- Refer Instruction-Tuninn dataset with 1.1M samples. GRIT contains multiple levels of spatial knowledge, covering objects, relationships. region descrip- tions, and complex reasoning. It includes both text-in location-out (grounding) and location-in text- out (referring) data, as well as data that mixes location and text in both input and output. The ma- jority of the dataset is converted from existing vision (-angu--a tasks like object detection (Krishnal et al. 2017) and phrase grounding (Yu et al.) 2016 Plummer et al.) (2015) with carefully designed templates to make it instruction-followee
Additionally, 34K refer-and- grouud instruction-tuninn conversations are collected via the help of ChatGPT/GPT-4 (OpenAl 2023b) to facilitate training an mnstruction-followinn and vocabbbuyyy refer-and and-ground generalist. Moreover, we conduct spatial-aware negative data mining, which further promotes model robustness.
Ferret subsumes strong open-vocy capabilities of spatial understanding and localization. When evaluated on conventional referring and grounding tasks, it achieves superior performance. More than that, we believe and-r-udd capabilities should be integrated into daily conversa- tions of humans, e.g., people refer something they don't know and ask what it is used for (like Figure To evaluate this new capability, we introduce Ferret-Bench, covering three new types of tasks: Referring Description, Referring Reasoning, and Grounding in Conversation. We benchmark existing MLLMs and observe that Ferret can outperform the best of them by 20.4% on average. Moreover, Ferret demonstrates an intriguing property of alleviating object hallucinations.
In summary, our contributions are threefold. (i) We propose Ferret, that uses a hybrid region rep- resentation equipped with a novel spatial- aware visual sampler, to enable Ane-grained and open-
Note that there additional vocabulary or position encoders introduced in Ferret model.
2

---

Published as a conference paper at ICLR 2024
vocabulary referring and grounding in MLLM. (ii) We construct GRIT, large-scale ground-and- refer instruction tuning dataset, for model training. It also contains additional spatial negative sam- ples to enhance model robustness. (tái) We introduce Ferret-Bench, evaluate tasks jointly quiring referring/grounding, semantics, knowledge, and reasoning. Our model exhibits superior performance in wide range of tasks and reduces object hallucination.
2 METHOD
We start with detailing the proposed hybrid region representation depict regions of various shapes and formats, Then, we present the model architecture of Ferret.
2.1 HYBRID REGION REPRESENTATION
When referring to specific regions, three primary formats are generally used: point, box, and free-form shapes. While the point and box for- mats can be succinctly represented by coordinates (e.g.. [x, y] for point, and [T'min Mmin; mmxx Mmax for a box) as in Peng et al. (2023); Chen et al. (203bb), the free-form shape more versatile, encompassing a variety of region types such as scribbles, polygons, and masks. The advantage of free-form shape is straightforwardly illustrated in Figure 2 Depict- ing free-form shapes through coordinates is computationally expensive and obscure, and its complexity hinders the model learning to establish a clear correlation between the provided coordinates and the corresponding re- gions.
regioni
regiond
aaject (regiont]
object [region1]]
The object (otyo] and the object [obs2] knife
To generalize across all three distinct formats, propose hybrid region representation that synergizes discrete coordinates with continuous visual features refer a particular region, which shown in the top-left Fig- ure For coordinates, following Chen et al. (021); ):; Yang et al. (0022), we quantize each coordinate into one of the Thans discrete bins Regarding continuous visual features, for a given region R. we first construct a 2D binary mask M of the same size as the image, marking a value of in- side the targeted region and O outside of the region. Then, the binary mask M, jointly with the extracted image feature map Z, is sent into our pro- posed spatial-aware visual sampler s--), which will be detailed in Section 2.2 to extract the visual continuous feature f = s sM, Z).
Figure 2: Bounding box
These Free-from Shape. objects have almost the same bound- ing box, causing ambigu- ity when relying on the box to refer to. Equipped with hybrid region repre- sentation, Ferret can sep- arate them.
Finally, we represent point with {z, y, fR, where the region Rp is circle centered in {z, y with fixed radius F A box free-form shape can both be represented by {aimin: Ymin: Tmax; Imax f R } where / max denotes the minimum/maximan z-axis coordinate of the region, and so forth for g-axis. Rboz denotes the input region.
2.2 MODEL ARCHITECTURE
As illustrated in Figure 3 Ferret is mainly composed of (f) image encoder to extract image embeddings, (ii) the proposed spatial-aware visual sampler to extract regional continuous features, and (iit) an LLM to jointly model image, text, and region features.
Input. We feed the image into a pre-trained visual encoder, to extract the image embeddings Z E #x xC' For text, we tokenize the text sequence using the pre-trained LLM's tokenizer and project them into text embeddings T E LxD As for referred regions, append the coordinates and a special token placeholder for continuous features after the name of the region: "(region name) (coordinates) (SPE)". For example, "a cat [100, 50, 200, 300] (SPE)". If the name is unknown or hard to describe because multiple objects are included, just use "region" or "area" as the "(region_me "(region_name)" In this way, referred regions can be well mixed with ordinary texts to form complete sentences.
(Radford et al.) 202
"hins = 1000 by default. The value is input invariant, which means for any input image size, the original coordinate will be mapped to the new coordinates. This makes the model robust to different input resolutions. Radius set to 5 by default.
3

---

Published as a conference paper at ICLR 2024
Hybrid Region Representation
Ferret Model
Region Name [Coordinates] cfeatures
It's a cat tailiac, 320)
Large Language Model
Point
Box
Free form Shape patygamo
Sampling
Fusion
Image Encoder
Neighbbrr
apatial-Aware Visual Sampler
Embedding
Block
Block
Projection
Flatten
What's in region|[100, 800, 500, 900] <SPEN ?
Spatlal-Aware Visual Sampler
Faatures
Image Input
references
Figure 3: Overview of the proposed Ferret model architecture. (Left) The proposed hybrid region representation and spatial-aware visual sampler. (Right) Overall model architecture. All parameters besides the image encoder are trainable.
Spatial-aware Visual Sampler. The shape of the referred regions can be quite varied, not limited to just points or rectangle boxes. Grid-based processing like convolution or patch attention cannot handle irregular shapes. Similar to our cases, 3D point clouds are also in irregular shape and show varied sparsity in the 3D space. Inspired by existing works in 3D point cloud learning (Qi et al., 2017a
Ma et al. 2022 Wang et al., 2019) we propose a spatial- apare visual sampler.
Given extracted image feature map Z E "x and the binary region mask M, we first randomly sample N positive points inside M. For each point, its feature obtained by bilinear interpolation. The N points are fed into cascade of blocks, where each of them includes three steps: sampling, gathering, pooling. (1) Sampling:
points are sampled from N points via farthest point sampling (FPS) algorithm (Qi et al., 2017b which can guarantee sufficient coverage. (2) Gathering: For each of the sampled points F,, search its & nearest neighbors from the pool of previous N points,
and obtain a group of points { I22, ..., Tik J. Then, inspired by PointMLP (Ma et al. 2022), for each group, we fuse the features of sampled point zy and it neighbor points by:
h4s ([(((Z();)) Z(x;): C(ax)) - C(x:)): ((x)):(()))
(1)
where Fik is one of the neighbors of vú, Z(x) denotes the point x's feature (in the first block, it is interpolated from feature map Z: in the succeeding blocks, it is the output feature from the previous block), (zz) denotes the 2D coordinates of point x, [:] means channel-wise concatenation of multiple vectors, is implemented by a linear layer adapt the relative local features, and o also a linear layer to fuse each local feature from neighbors with sampled point feature. (3) Pooling: max pooling conducted to fuse k neighbor features into one feature as the representation of the sampled point:
max (= JKNNS of x;
har
After the three steps, we obtain fewer points but more dense feature space since it incorporates the local neighbor features as well as their relative positions. k=24, and cascade two such blocks, which in the end outputs 32 points with their features. Similar to ROIAlign (He et al. 2017). we flatten the point features into a single vector and project it to the dimension of LLM embeddings. The final feature is used to replace the (SPE) token in the input.
experiments, we set N=512, r=4 and
The above region denotations are used in Ferret input to refer to specific regions. In Ferret output, to achieve grounding, we generate the box coordinates right after the corresponding regions/nouns in the text response. For instance, "There is a dog [100, 150, 300, 200) in the figure." With this data format, image and what their locations are.
Output.
model is expected implicitly learn what is groundable in the current
LLM. We consider Vicuna (Chiang et al. 2023) our language model, a decoder-only LLM (Brown et al. 2020) that is instruction-tuned on top of LLaMA (Touvron et al. 2023a). Prior to being fed into the LLM, the image embeddings undergo transformation via an additional linear layer to match the embedding dimension of the text tokens.
*FPS starts from a random single point sampled from N points. In each iteration, it samples one point from the rest points such that it is the farthest from the of already sampled points. See detail in et al. (207h).
4

---

Published as a conference paper at ICLR 2024
Object:
Input
What the class of object
Output
corgi.
Visuel Genome RefCOCDs Flickr30k
[region0)?
Input
Output
Where the corgi?
corgi [bxx)].
Relationship:
Input
you figure
Output
relation of objectt [region and object [region2]?
Visual Genome
the floor.
GLL
Region:
Input
Output shori,
Describe (region3]] short phrase.
Genome
Detail +
Input
Output
Reasoning:
Do you know when the object [region4] was invented?
The invention of TV [box]] cradit to John Logio Baird, He
GPT Generated datt
inventedd the early 1920s.
Robustness:
Input
Output
No, there dog [box0] image.
Spatial Negative Mining
there image?
Figure 4: Overview of the GRIT dataset for Ferret model training, It contains three types of data: (i) public datasets that are converted into instruction-followwe following format (the top-3 rows): (it) data generated via prompt- ing ChatGPT and GPT-4 (the 4th row); and (ii0) negative data to enhance model robustness (the last row).
3 GRIT: GROUND-AND-REPET INSTRUCTION-TUNING DATASET
In this section, we present GRIT, Ground-and- Refer Instruction-Tune Dunning dataset containing around 1.IM multimodal dialogues for model training. GRIT consists of three types of data: (1) public datasets that are converted into instruction-followii data generated via ChatGPT and GPT-4 (Section 3.2); ; and (iii) additional data from spatial negative mining for enhancing model robustness (Section 3.3
format (Section (6i) instruction-tuning
3.1 HIERARCHY
Spatial understanding can be characterized by varying levels of granularity and task formats. During our dataset creation, we look into the following categories based on two dimensions:
In terms of granularity, we identify four main categories: (i) individual objects, (ii) relationships
among objects, (iid) descriptions specific regions, and (iv) region-based complex reasoning.
In terms of task format, we further divide the data into three distinct types: (i) Region-in Text-out data, (ii) Text-in Region-out data, and (iti) Text- Region combined data
We compiled an extensive set of public data focusing on the aforementioned dimensions and con- verted them into instructionnfollooiing format using carefully designed templates. A more in- depth view of these templates available in Appendix C.1
Individual objects. To achieve visual understanding the object level, select object detection datasets such as Visual Genome (Krishna et al.] (017)) Object365 (Shao et al. 2201)) and visual grounding datasets including RefCOCOs (Yu et al.) 2016; Lin al. [014]] Nagaraja al. 2016) and Flickr30k-Entite
(Plummer et al. 2015). The converted Visual Genome object data follow Region-in Text-out format. Additionally, to enable Ferret to understand free-form shapes, we apply SAM (Kirillov et al,, 2023) to Visual Genome object data to obtain a segmentation mask for each object, which fed into the spatial-aware visual sampler to extract continuous region feature during training. The visual grounding datasets and Object365 data adhere to Text-in Region-out format. This section has in total 678k data.
Relationships among objects & descriptions of regions. We selected data pertaining to object relationships and region captions from Visual Genome (Krishna et al.) 2017) to address these two facets, respectively. Both datasets employ Region-in Text-out format and 177k data are obtained. Similar to Visual Genome object data, we also extract segmentation masks objects in Visual Genome relationship data via SAM.
Foor Region-in Text-out data, the input highlights a specific region, prompting queries about it. For Text-in Region-our out data, the input comprises textual descriptions. and the task is to pinpoint or ground the relevant region in its response. The combined Text-Region data integrates both text and region within single sequence, which can he present in the input, output, both.
5

---

Published as a conference paper at ICLR 2024
Region-based complex reasoning. Regarding complex reasoning centered specific regions, constructed a novel dataset with the help of ChaGGPT/GPT44 It adopts a combined Text-Region format, and is detailed in the subsequent section.
3.2 GPT-ASSISTED VISUAL INSTRUCTION DATA GENERATION
Besides converting existing datasets by templates, dialogue instruction tuning data is proved to be critical for MLLM to understand human intention and generate fluent, natural, and long-fon re- sponses (Liu et al. 2023b Zhu et al. 2023a Li al. 2023d). Few-shot prompting is widely used to obtain visual instruction tuning data, where textual scene descriptions of images and human- annotated dialogues are provided few-shot demonstrations, and ChaIGPT/GPT4 are prompted to generate new dialogue based on the new image's 's textual scene descriptions.
However, previous instruction tuning data mainly focus on describing the entire image without ex- plicitly specifying spatial-related information. To collect refer-and-groume instruction tuning data, we emphasize region-based spatial knowledge in the following three steps. (i) Besides objects and global captions usually used as before, our symbolic scene description additionally includes physical relationships between objects and region captions along with coordinates of them. (ii) In human- annotated dialogues, we add coordinates after the groundable regions or objects either in input or output or both, and the dialogues are typically focused on specific regions. helps to implicitly prompt ChatGPT/GPT4 to follow similar patterns when generating new dialogues. (mi1 The gen- erated dialogues sometimes cannot follow the rules and patterns we wrote in system prompts and few-shot examples, which might be due to that the context of LLM imput is too long to handle all the details. To alleviate it, we propose to use ChatGPT/GPT-44 again to refine the initially generated dialogues, whose context length is only 10% of the data generated from the first round on average. To save cost, we use ChatGPT in the first round of generation and GPT-4 for refining. 34k dialogues in total are collected.
Additionally, to exploit existing instruction-tuning data such as those in LLaVA (Liu et al. 2023b we apply open-vocabulary object detector, GLIPv2 (Zhang et aT., 2022), LLaVA-158K data to localize groundable nouns in the text. Then, we append the bounding boxes after the corresponding nouns, forming a pseudo-grounded grounded LLaVA instruction data that are also used for training Ferret.
3.3 SPATIAL NEGATIVE MINING
As highlighted in prior studies (Li et al. 2023e Liu et al. 2023a), MILM exhibits a propensity to hallucinate in response to yes/no questions. We observed a similar occurrence when inquiring about detailed regions. To address this, we also conduct negative sample mining by following two ways: (i) condee-onddiiooed Category Localization, and (ii) Armantics-conditioee Category Localization. They both ask the model to localize specific object categories, thereby enabling the model's ability to discern and potentially recognize the absence of certain objects. They differ in how to select the negative category. For (1), Object365 data are employed and we randomly select the object class from the vocabulary that is not shown in the given image. For (ii), Flickr30k data are used and negative categories are sourced by utilizing ChatGPT/GPT4 to find entities that are most analogous to the original class, attribute, or quantity, e.g., 'man' vs. 'woman', 'bluc' vs. 'yellow', 'two' vs. "three".
We curate the data to maintain equilibrium between positive and negative samples for each of the two types F 95k data are collected. A more comprehensive elaboration is provided in Appendix C.2
EXPERIMENTS
First of all, we illustrate the training details of Ferret. Then in evaluation, we start with evaluating Ferret on conventional referring and grounding benchmarks (Sec. [4.] and 4.2). Then, we demon- strate the power of Ferret in more complex multimodal chatting with refer-and-groum capability in Sec. 4.3 For a detailed visualization of each, kindly check Appendix We further ablate key
components in Ferret (Sec. 4.4). analyze the object hallucination of Ferret (Sec. 4.5) and discuss Ferret v.s. GPT-4V (Sec. ??).
'We observed that even though we don't It collect other data specifically for training, Ferret demonstrates the capability to generalize robustness across diverse categories like relationships, events, etc. We attribute this versatility to the potent compositional capabilities inherent L.I.M.

---

Published as a conference paper at ICLR 2024
Table I: Results of referring object classi- fication three different referring types, in- cluding point, box, and free-form shape. 'x' means no such capability.
Table 2: Results of grounded image captioning on the test
set of Flickr30k Entities. BLEU@4, METEOR, CIDEr, and SPICE are used for the caption evaluation. Flan and Fl are used for grounding evaluation. '-' means not reported.
Models
LVIS (Ace ())
Models
Caption Eval.
Grounding Eval.
Point
Box
Free form
B@4
M
C
Flait
Flo
Random Guess
50
50
GVD (Zhon et al 2019
27.3
22.5
62.3
16.5
7.55
22.2
LLaVA
50.1
50.3
X
Cyclicar bar et u. 2020
26.8
22.4
61.1
16.8
8.44
22.78
Kosmos-2 Pong ct al. 2023
60.25
POS-SCAN 2hou o 2020
30.1
22.6
69.3
16.8
7.17
17.49
Shikra 7B
Chen er al
20230
57,82
67.71
X
UniTAB (Yangreedi 200ce
30.1
23.7
69.7
17.4
12.95
34.79
GPT4-RO1
Zhaang ot 2023
61.76
Shikra- 13B (Chen er Mizsh
73.9
Ferret-7B
67.94
79.42
69.77
Ferret-7B
35.1
24.6
74.8
18.0
15.02
37.62
13n
68.35
800.66
70.96
Ferret-33B
37.0
25.5
76.1
15.12
38.03
Training Details. We initialize the image encoder with CLIP- VEL---VI--/11@2333 336p, the LLM with Vi- cuna, and the projection layer with LLa's's first-stage weights, leaving the visual sampler ran- domly initialized. After the initialization, Ferret is trained on the aforementioned GRIT data for three epochs, optimized by Loshchilov & Hutter (2017) with learning rate of 2e 5 and batch size of 128. The training takes ~52.55 days on 8 A00 GPU for Verret-13B/7B During training, when input refers to regions, we randomly choose either the center points or the bounding boxes (or segmentation masks if available) to represent the regions. We perform de-duplication in training data to remove the samples that are in downstream evaluations.
4.1 INPUT REFERRING
The model's capability of understanding referring is reflected in that, given a referred region in the question, how accurately the model can understand the semantics of the referred region. To measure it, we start with the most basic semantics, object, as it is fundamental and clear to define. To be more specific, the task we evaluate on is Referring Object Classification: the question refers to a specific region in the image, and the model needs to classify the object in the region. Since Ferret and MLLMs usually generate free-form text responses, it is inaccurate match the predicted class with the ground- truth class if directly asking the model to classify without constraints. Alternatively, make it binary-choice question in the format of "Is the object (location) (class A) (llassB))?0 ?". We feed the binary-choice question and image into the MILLMs to obtain the response, and then detect if the response matches the ground-truth (GT) class by some rule [
To prepare the data, we used the validation split of LVIS dataset (Gupta et al. 2019) covering over 1000 object categories, and sampled 2667 objects the GT objects. Then, we randomly choose a different object category in the same image whose central point is close to the GT object as the negative object, and replace (class_A) and (class_B) with those two randomly form 2667 questions. Additionally, to mimic the versatility of referring in human life, we replace the (location) with three different types: point, box, and free-form shape. For point, we randomly sample a point inside the GT object that is also near the GT object's boundary. For box, we use the GT bounding box provided by LVIS For the free-form shape, we randomly generate some strokes inside the GT object to simulate that. Results on all three types of referring are summarized in Table [ Ferret can significantly outperform previous models (Peng et al., 2023 Chen et al. 2023b) and handle all types of referring, capability notably absent in previous works.
4.2 OUTPUT GROUNDING
Ferret performs well in referential dialogue, allowing for its integration into various VL tasks, no- tably those with grounding outputs. To rigorously assess the grounding capability, we first subject Ferret to benchmark visual grounding tasks in a generative paradigm. Then, to measure the align- ments between words and regions, we further evaluate Ferret on grounded captioning task.
Visual grounding. Visual grounding aims to ground language queries into aligned image regions. We experiment on the sub-tasks of referring expression comprehension (REC) with three renowned benchmarks: ReíCOCO (Lin et al.] 20144), RefCOCO+ (Yu et al.) 2016). and RefCOCOg (Mao et al. 2016), and phrase grounding with Flickr30k Entities dataset (Plummer et al. 2015). REC task involves a question or description about specific area in an image, with the model expected to predict just one bounding box. Phrase grounding, conversely, seeks to associate all the noun
Sooeetmes both GT class and negative class appear in the answer, e.g., "TTh object is (class_GT), not (class Neg)". Our rule removes the substring in-between "not" and comma/period, and then detects GT class.
7

---

Published as a conference paper at ICLR 2024
Table 3: Performance comparison (Acc@0.55) on referring expression comprehension (RefCOCO. Ref- COCO+, RefCOCOg) and phrase grounding (Flickr30k Entities) tasks. indicates that the method is specifi- cally fine-tuned in the second stage.
RefCOCO
RefCOCO+
RefCOCOg
Flickr30k Entities
Models
val
testA
testB
val
testA
testB
val
test
val
MAtNee YYu 2018
76.40
80.43
69.28
64.93
70.26
56.00
66.67
67.01
OFA-L
Wang ct al. 2022b
79.96
83.67
76.39
68.29
76.00
61.75
67.57
67.58
VG
Deng
81.02
82.72
78.35
64.82
70.70
56.94
68.67
67.73
79.10
UNITER
Chen el al.
20:20
81.41
87.04
74.17
75.90
81.45
66.70
74.02
68.67
VILLA
Gan et al.
2020
82.39
87.48
74.84
76.17
81.54
66.84
76.18
76.71
UniTAB
Yang a1.
2022
86.32
88.84
80.61
78.70
83.22
69.48
79.96
79.97
78,76
79.58
MDETR
Kamath et al.
202
86.75
89.58
81.41
79.52
84.09
70.62
81.64
80.89
82.3*
83.8*
Shikra-7B Chen al.
2023b
87.01
90.61
80.24
81.60
87.36
72. 12
82.27
82.19
75.84
76.54
Ferret-BB
87.49
91.35
82.45
80.78
87.38
73.14
83.93
84.76
80,39
82.21
Shikra-13B
(Cenn et al.
2023b
87.83
91.11
81.81
82.89
87.79
74.41
82.64
83.16
77.41
78.44
F3rret-13B
89.48
92.41
84.36
82.81
88.14
75.17
85.83
86.34
81.13
84.76
phrases in the input sentence with corresponding boxes, requiring the model to predict these boxes and the word-box connections. For both tasks, we utilize uniform prompts, represented as "What are the locations of <querr>><<hrruses>>>>>>>>>>>>>>> ?", where <query> denotes the textual referring expression, while <phrases>> stands for a "comma-delimited" aggregation of the given phrases. The model is trained to output in "<query>> [box]." format. The generated bounding box is considered correct its intersection over union (IoU) with the GT box is greater than 0.5. As shown in Table 3, Ferret achieves an outstanding performance on all metrics, and is comparable to specialized fine-tuning approaches (Kamath et al., (202)).
Grounded captioning. The grounded captioning task requires the model to generate a caption and ground all generated noun phrases image regions. The final predictions generally consist of three parts, ie., the text caption, visual regions as boxes, and the grounding alignments between words and boxes. Following the established benchmarks on the Flickr30k Entities dataset, we evaluate captioning and grounding separately with the captioning metries and grounding Fl scores, respec- tively. Flat evaluates grounding as multi-label classification problem. We also report Flaoc that only computes the grounding score on correctly predicted object words. Results are summarized in Table [ and Ferret achieves state-of-the-oe
4.3 FERRET-BENCES MULTIMODAL CHATTING WITH REFERRING AND GROUNDING
Multimodal chatting has been an emergent ability of MLLMs. Previous benchmarks (Liu et al., 2023b) mainly evaluate conversation, detailed description, and complex reasoning via GPT-4 as a judge. Yet, a gap exists as no dataset currently evaluates multimodal chatting that necessitates referring or grounding actions, e.g., instances where individuals reference an unfamiliar object and inquire about its purpose. To benchmark this intriguing and practical capability, we introduce Ferret- Bench that covers three kinds of region- questions evaluating referring and grounding capa- bility: (i) Referring Description: models are asked to describe a referred region based on its in- teraction with surrounding objects. (ii) Referring Reasoning: models need to reason on top of one more referred regions correctly. (Hii) Grounding in Conversation: models required to reason correctly and accurately ground/localize the objects/regions necessary for the reasoning. For the ease of benchmarking other methods, we represent the regions with boxes instead of points or free-form shapes.
Specifically, we randomly sample 40 images from the COCO validation set for each type of question, and generate the questions and GPT--''s answers following the instruction generation pipeline in Sec. 3.22 Following Liu et al. (002b)). we feed the question and image into MLLMs to obtain the predicted answer, and then prompt GPT-4 to rate the predicted answer and pseudo answer from GPT-4 based on the ground-truth textual scene description (object, relationship. region caption, global caption). GPT-4 evaluates both the precision of referring understanding, object grounding, and correctness of semantics. The rating score ranges from to 10, in which higher means better. We calculate the ratio of the predicted answer's score and the GPT-4 answer's score, which is then presented as a percentage to measure the performance of MLLMs. We also asked GPT-4 to give a comprehensive review for the rating and found that GPT-4 is good at measuring the degree of spatial precision, such as how much the predicted bounding box diverges from the GT box coordinate. We refer the readers Appendix D for further elaboration.
8

---

Published as a conference paper at ICLR 2024
Table 4: Results on LLaVA ---eeenchh and the proposed Ferret Berret-Bench GPT4-as aa----s---Jude evaluation.
LLaVA-Bende
Ferret-Bench
Conversation
Detail Description
Complex Reasoning
Avg.
Referring Description
Referring Reasoning
Grounding in Conversation
Avg.
LLaV 85.4
68.3
92.1
81.9)
41.4
31.7
28.8
34.0
Kosmos-2
71.7
63.4
74.9
70.0
51.8
33.7
48.4
44.6
Shikra-7B
80.6
70.7
88.1
79.9
46.0
41.6
50.1
45.9
Ferret-7B 84.4
79.4
96.3
86.7
68.7
67.3
57.5
64.5
Ferrer 85.2
80.9
96.4
87.5
70.6
68.7
59.7
66.3
Table 5: Ablation study on the mutual benefit of grounding data and referring data. We evaluate Ac- curacy for LVIS referring and @ I for grounding.
Table 6: Ablation study the effectiveness of the pro-
posed spatial-aware visual sampler. Accuracy used evaluate LVIS referring.
Referring (I.VS))
Grounding
Model
Referring (LVIS)
Module
Point
Box
Flickr30k
Point
Box
Free-form
Ferret
67.9
79,4
80.4
Spatial-aware Visual Sampler
67.9
79.4
69.8
w/o Grounding data
65.4
75.6
X
Visual Sampler in SHEM
67.1
77.2
68.9
w/o Referring data
x
X
79.8
We use LLaVA- Bench (Liu et al. 2023b) and the proposed Ferret-Bench to compare Ferret with
previous models, including LLaVA (Liu al., 2023b), Shikra (Chen et al. 2023b), and Kosmos- 2 (Peng et al. 2023). Results are summarized all types of tasks, boosting the score for the detailed description category from 68.3 to 80.9, and especially excels at the three new tasks demanding referring and grounding abilities.
Table Ferret achieves superior performance in
4.4 ABLATION
In the ablation studies below, in default, we ablate Ferret-7B and mainly evaluate in referring object classification and grounding tasks on Flickr30k Entities validation set.
Mutual benefits of grounding and referring. As shown in Table [ grounding and referring, as two main capabilities emphasized in this paper, can actually benefit each other. Particularly, when adding grounding data into training, the referring performance gets improved, and vice versa.
Spatial-aware Visual Sampler. We ablate the effectiveness of the spatial-aware visual sampler by replacing it with the visual sampler in SEEM (Zou et al. 2023). where they average the features of all the sampled points as the region feature. As we can see in Table [6] ours can outperform the previous visual sampler in all three referring tasks.
LLM model size. We study how much LLM model size influences the performance of referring and grounding. As seen in Table having a larger L.M backbone can generally help.
4.5 ОВJЕCТ HALLUCINATION
Attribute to the incorporation of fine- fine-grained spatial knowledge and negative mining, Ferret also exhibits strong power against the hallucination problem. We evaluate object hallucinations on the POPE benchmark (Li et al. 2023c). Results summarized in Table Ferret has exhibited perfor- mance comparable to Shikra (Chen et al. 2023b), and far surpasses recent popular MLLMMs
CONCLUSION
We present Ferret, a new multimodal large language model adept at referring and grounding. Fer- ret can refer image regions in any free-form shape, and automatically establish grounding for text deemed groundable by the model. We have curated the GRIT dataset for model training, and the Kerret-Bench dataset for evaluation. Ferret, like most MILMs,, may produce harmful and counter- factual responses. For future work, inspired by LISA (Lai et al. 2023). we plan to enhance Ferret to be able output segmentation masks in addition bounding boxes.
'The result LLaVA-Bent is obtained by evaluating LLaVA released checkpoint. The slight discrepancy might be due evolving GPT4 APIs. For Ferret-Bench, employ the same conversation template Ferret, providing I.I.VVA with a predefined input size, resizing coordinates accordingly, and generating a response. Unlike other methods, Ferret refrains from relying VQA. This decision stems from our observation that VQA answers tend be concise, and this brevity can restrict the conversational capabilities of L.MMs.

---

Published as a conference paper at ICLR 2024
REFERENCES
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, al. Cm3: A causal masked multi- modal model of the internet. arXiv preprint 2rXiv:2201.055200 2022.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanaty, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo. March 2023. URL https:
//doi. 0t.pg//1..2811/// 5201...211111odll 2 7733589
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Chi Chen, Ruoyu Qin, Fuwen Luo, Xiaoyue Mi, Peng Li, Maosong Sun, and Yang Liu. Position- enhanced visual instruction tuning for multimodal large language models. arXiv preprint orXiv:2308.13437 2023a.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306. /.5 195, 2023b.
Ting Chen, Saurabh Saxena, Lala Li, David Fleet, and Geoffrey Hinton. Pix2seq: language modeling framework for object detection. arXiv preprint arXiv:2 10852, 2021.
Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Yi Lin, David Fleet, and Geoffrey Hinton. unified sequence interface for vision tasks. arXiv preprint arXiv:2206.0760: 2022a.
Xi Chen, Xiao Wang. Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Danicl Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, al. Pali: A jointly-scabe multilingual language-image model. arXiv preprint arXiv:2209.06799 2022b.
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305. 0at.:2335./8555 18565, 2023c.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprini arXiv:2204.025:: 2022.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint 2.05:230000000 2023.
Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Hougiang Li. Transvg: End-to- end visual grounding with transformers. In ICCV, 2021.
Danny Driess. Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM-E: An embodied multi- modal language model. arXiv preprint arX 2023.
10

---

Published as a conference paper at ICLR 2024
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-len -ad-aang representation learning. NeurIPS, 2020.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmen- tation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5356-5364, 2019.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cmn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Car- ion. Mdetr-modulated detection for end-to-end multi-modal understanding. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1780-1790, 2021.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgam:: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods natural language processing (EMNLP), pp- 787-798, 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Ya Yen Lo, et al. Segment anything. arXiv preprint ar XxXi:23304.02665 2023.
Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal language models. arXiv preprint urXiv:2305.172111 2023a.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried, Grounding language models to images for multimodal generation. arXiv preprint arXiv:2301.13823 2023b.
Emiel Krahmer and Kees Van Deemter. Computational generation of referring expressions: A survey, Computational Linguistics, 38(1):173-218, 2012. 2012.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen. Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsoureed dense image annotations. International jouraall of computer vision, 123:32-73, 2017.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. arXiv preprint arXiv:2308.008: 00692, 2023.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, al. Obelisc: An open web-scale filtered dataset of interleaved image-text documents. arXiv preprint arXiv:2306.1653 16527, 2023.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726. 2023a.
Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, and Jianfeng Gao. Multimodal foundation models: From specialists general purepoe assistants. arXiv preprint arXiv:2309. 10020, 2023b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023c.
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang. Jingjing Xu, Xu Sun, et al. M A large-scale dataset towards multi-modal multilingual instruc- tion tuning, arXiv preprint arXiv:2306. 04387, 2023d.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Li- juan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language- -mage pre-trate training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10965-10975. 2022.
11

---

Published as a conference paper at ICLR 2024
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint S.kv:2305..1335 2023e.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv::304. 08485, 2023b.
Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arxiv:1711.05001, 2017.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and mudti-modal tasks. arXiv preprint arXiv:2206.08916, 2022.
Ruotian Luo and Gregory Shakhnarovich. Comprchension-guidee referring expressions. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7102-7111, 2017.
Chih-Yat Yao Ma, Yannis Kalantidis, Ghassan AlRegib, Peter Vajda, Marcus Rohrbach, and Zsolt Kira. Learning generate grounded visual captions without localization supervision. ECCV, 2020.
Xu Ma. Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: A simple residual mlp framework. arXiv preprint arXiv:2202.07123, 2022.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 11-20, 2016.
Varun Nagaraja, Vlad I Morariu, and Larry Davis. Modeling context between objects for referring expression understanding. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-4, 2016, Proceedings, Part IV 14, pp. 792-807. Springer, 2016.
OpenAl. GPT-4 technical report.
2023a.
OpenAl. Gpt-4 technical report. arXiv, 2023b.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furn Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 148244 2023.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region-to- to-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference computer vision, pp- 2641-2649, 2015.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 652-660, 2017a
Charles Ruizhongtai Qi, Li Yi, Hao Su. and Leonidas J Guibas. Pointnet++: Deep hierarchical fea- ture learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017b.
12

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana llié. Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 2022.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4430-8439, 2019.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint arXiv:2307. 05222, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205. 241100205..4100, 2022a.
Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence-to-sequenee -oquueeeccee learning framework. In ICML, 2022b.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint 2rXiv:2305. 11175 2023.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. ACM Transactions on Graphics (tog), 38(5): 1-12, 2019.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In ICLR, 2022c.
Jialian Wu, Jianfeng Wang, Zhengyuan Yang. Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-texs to-text transformer for object understanding. arXiv preprint arXiv:2212.002880 2022.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded rision-language modeling. In European Conference on Computer Vision, pp. 521-539. Springer, 2022.
Qinghao Ye. Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. /4/78, 2023.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amster- dam, The Netherlands, October 1 11-14, 2016, Proceedings, Part II 14, pp. 69-85. Springer, 2016.
Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. A joint moakeer-istenee--einnee for referring expressions. In Proceedings of the IEEÉ conference on computer vision and pattern recognition, pp. 7282-7290, 2017.
model
13

---

Published as a conference paper at ICLR 2024
Licheng Yu. Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mat- tnet: Modular attention network for referring expression comprehension. In CVPR, 2018.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint 23Xiv:2309.0251,, 2023.
Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy. Contextual object detection with multimodal large language models. arXiv preprint arXiv: 18279, 2023.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6720-6731, 2019.
Haotian Zhang, Pengehuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision- language understanding. Advances in Neural Information Processing Systems, 35:36067-36080, 2022.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601, 2023.
Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. Bubogpt: Enabling visual grounding in multi-modal 1mms. arXiv preprint arXiv:2307.0858// 2023.
Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J Corso, and Marcus Rohrbach. Grounded
video description. In CVPR, 2019.
Yuanen Zhou, Meng Wang, Daqing Liu, Zhenzhen Hu, and Hanwang Zhang. More grounded image captioning by distilling image-text matching model. In CVPR, 2020.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint vrXiv:2304.10592 2023a.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Young- jae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion- scale corpus of images interleaved with text. arXiv preprint arXivv23004.663 06939, 2023b.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jac Lee. Seg- ment everything everywhere all at once. arXiv preprint 0rXiv:2304.06718, 2023.
14

---

Published as a conference paper at ICLR 2024
Table Results on the object hallucination benchmark using the POPE evaluation pipeline (Li et al.
2023ep.
Datasets
Metrics
Ferret
Shikra
InstructBLIP
MiniGPT4
LLaVA
MM-GPT
mPLUG-Owl
Accuracy (1)
90.24
86.90
88.57
79.67
50.37
50.10
53.97
Precision (1)
97.72
94.40
84.09
78.24
50.19
50.05
52.07
Rundom
Recall (1)
83.00
79.26
95.13
82.20
99.13
100.00
99.60
Score (t)
89.76
86.19
89.27
80.17
66.64
66.71
68.39
Yes
43.78
43.26
56.57
52.53
98.77
99.90
95.63
Accuracy ())
84,90
83,97
82.77
69.73
49.87
50.00
50.90
Precision (1)
88.24
87.55
76.27
65.86
49.93
50.00
50,46
Popular
Recall (t)
80.53
79.20
95.13
81,93
99.27
100.00
99.40
FI Score (1)
84.21
83.16
84.66
73.02
66.44
66.67
66.94
Yes
45.63
45.23
62.37
62,20
99.40
100.00
98.57
Accuracy (
82.36
83.10
72.10
65.17
49,70
50.00
50.67
Precision (1)
83.60
85.60
65.13
61.19
49.85
50.00
50.34
Adversarial
Recall (f)
80.53
79.60
95.13
82.93
99.07
100.00
99.33
F1 Score (f)
82,00
82.49
77.32
70.42
66.32
66.67
66.82
Yes
48.18
46.50
73.03
67.77
99.37
100.00
98.67
Table Comparison of Ferret v.s. recent MLLMs integrating spatial awareness. 'Convention' refers to comprehensive collection of publicly available data that has been transformed using templates, "ePT-GGeerate' signifies the generated refer/ground datasets employing GPT, and 'Robustness' de- notes datasets aimed at mitigating ballucination and improving robustness. Section explains more details about each.
Input Types
Output Grounding
Data Construction
Quantatitive Fval. of Refer/Ground w. Chat
Model
Point
Box Free-form
Convention GPT-Generate
Robustness
BuboGPT
x
X
X
Vision-LLM
X
X
X
Kosmos-2
X
X
X
Shikra
X
X
GPT4-ROI
X
X
PVIT
X
X
Ferret
A RELATED WORK
Multimodal large language models (MLLMs). Large Language Models (LLMs), including
GPTs (Brown et al. 2020; OpenAl 2023a). PaLM (Chowdhery et al. (022), BLOOM (Scao al. 2022). and LLaMA (Touvron et al. 2023a b). have revolutionized research in NLP, spurring sig- nificant advances in multimodal language models well. Early models primarily focused large-scale image-text pre-training. Notable examples include SimVLM (Wang et al. 2022c), 2022b), PaLI-X (Chen al. 2023c), PL (Li PaLM-E (Driess et al. (2023). CM3 (Aghajanyan Flamingo, in particular, pioneered the integration
GIT (Wang et al., (0022a). PaLI (Chen al.
et al. 2023c), Flamingo (Alayrac et al. 2022
2022 and CM3Leon (Yu al.) 2023).
of pre-trained CLIP image encoder with LLMs through gated cross- attenntton blocks, showcasing emergent multimodal in-context seo- learning capabilities. Its open- sourced variants, such as OpenFlamingo (Awadalla et al.) 2023) and IDEFICS (Laurençon et al., 2023), have garnered sig- nificant attention. Typically, these models undergo pre-training using millions even billions of image-text pairs and interleaved image-text datasets (Zhu et al. 2023b)
On the other hand, recent research has increasingly focused on using pre-trained LLMs for visual instruction tuning. Prominent examples include LLaVA (Liu et al.) 2023b), MiniGPT-4 (Zhu et al. 2023a), mPLUG-Owl (Ye et al.) 2023). Otter (Li et al.) 2023a), InstructBLIP (Dai et al. 2023 to name few. In addition to text generation, recent models like FROMAGe (Koh et al. 2023b GILL (Koh et al. (023a), Emu (Sun et al. 2023), have also enabled MLLMs for image retrieval and image generation. Please refer Chapter of LiLi al. (202bb) for detailed review.
15

---

Published as a conference paper at ICLR 2024
MLLMss for referring and grounding. In the realm of existing literature, works such Kosmos-
2 Peng et al.) 2023) and Shikra (Chen et al. 2023b), closely resemble ours as they also enable
MLLMs for fine- line-grained image comprehension and open-world referring and grounding. Addi-
tional works in this direction include GPT4ROI (Zhang et al. 2023), PVIT (Chen et al. 2023a
BuboGPT (Zhao et al. 2023), VisionI.LM (Wang et al. 2023), and ContextDET (Zang al. 2023
Nevertheless, pivotal distinctions set our model apart. First, prior endeavors supported only bound- ing boxes (and points in Shikra) as input. Conversely, due to Ferret's innovative hybrid region repre- sentation, we accommodate broader range of free-form shapes for referring, encompassing points, boxes, sketches, scribbles, polygons, and more. Second, we meticulously curate an extensive refer- and-ground instruction tuning dataset. Third, we introduce Ferret-Bende to facilitate forthcoming research and enhance evaluation benchmarks in this direction. Lastly, our model exhibits superior performance compared to previous works, notably mitigating object hallucination to significant extent. A more straightforward side-by-side comparison is shown in Tab. g
Unifying grounding and VL understanding, Our work is also related to previous work that aims to unify text and bounding box output for vision-language (VL) models, such as UniTAB (Yang
et al., 2022p, OFA (Wang et al., 2022b), and Unified-IO (Lu et al., 2022). which also represent bounding boxes using a set additional discrete tokens as proposed in Pix2Seq (Chen et al. 2021 2022aa). Ferret is unique in that (i) our model is built upon LLMs,, marrying the power of LLMs and grounding, thus unlocking new capabilities such as grounded instruction tuning, and (ii) we handle bounding box coordinates as regular text tokens, avoiding the need for extra specialized tokens dedicated to representing boxes.
B DISCUSSION ON LIMITATION AND FAILURE CASES
We acknowledge certain specific failure scenarios and limitations for our models, which detailed as follows:
Failure Scenarios: (1). Referring to many objects (more than 3) in one question might not be as accurate as referring to each of them in separate conversations. This is likely due to a relative scarcity of training data that mentions too many objects. (2). The referring and grounding very small objects is less accurate than large or medium objects. It's a common challenge in object detection. However, we think further improving input image resolution is able to help.
Limitations: (1). Not good at other languages because the training dataset is curated only in English. Although Ferret shows some emergent referring and grounding capability in other languages, its performance in other languages is still worse than in English. Future incorporation of multilingual training data could potentially mitigate this. (2). Similar to many large language models, Ferret has the potential to generate harmful or factually incorrect responses. (3). Ferret is not designed for segmentation tasks requiring mask outputs.
C DETAILS OF DATASET
C.I TASK TEMPLATES FOR PUBLIC DATASETS
In Section
3.1 mentioned using carefully designed task templates to convert public datasets such as Visual Genome into instruction-following format. The task templates we used are provided in For simplicity, we only list three examples for each task.
Table 9
C.2 DETAILS ON SPATIAL NEGATIVE MINING
In Section we conducted negative sample mining for two aspects: (i) Image-conditioned Cate- gory Localization, and (ii) Semantics-condiiionee Category Localization. They use the same tem- plate to convert the original data, which falls into the task of object hallucination in Table Specif- ically, for the negative categories in (ii), we prompt ChatGPT/GPT-4 to generate entities that are most analogous to the original class, attribute, or quantity, e.g. 'man' vs. 'woman', 'blue' vs. "yel- low", "two' vs. 'three'. The prompt feed into ChatGPT/GPT-4 encompasses all the entities extracted from 5 captions associated with one single image. We show the exact prompt template in Table TO
16

---

Published as a conference paper at ICLR 2024
Table 9: Examples of task templates Ferret used transfer different public data types into the instruction-following format.
Task
Three randomly chosen examples from many.
What the class of the object <location within the image?
Referring Object
Classify abject <locaaion>>
image.
Identify the object <location>- in the image.
What does cobjectt > > <locationl > do to <ohject2>> <location2> the image?
Relering-Reeltioon
What physicul relation between <object|>> <location<>> and <objeet2>> <locution>>?
Can you figure out the goometric relation of the <object1>> <locationI>> and <object2>> <location2>?
Describe region <location>> in a short phrase.
Referring Region
What in the region <location>>? Describe phrase.
Capture in phrase: what's 's near region <location'> in the picture?
Where <object>> in the image? What are the coordinates for the given <object>> in the image?
REC.
Given the image, could you pleuse tell me where <object>>
What the locations of cobjects>> Could you provide me with the exact locations of <objects>>?
Phrase Grounding
Please indicate the positions <objects>> the image?
Detect all objects among <class>> the image.
Object Detection (0365)
Perform object detection given the image within <class>,
Given the image
<class>, identify the objects that helong set.
What this photo about? Use concise language. Describe the overall picture in just few words.
Grounded Captioning
What do you happening this image? Provide the short.
Is there <object>> in the image?
Object Hallucination
Are there <cobject>> Please tell me whether <object>> exists in the image?
image?
Table 10: In this example, we provide the prompt to generate the spatial negative sets.
messages [ ("role": "system", "content" : You are an AI visual assistant that can analyze single image. You receive several entities given by list, each describing the objects in the image you are observing.
For each entity mentioned, change them with the most mislcading entity name (may belong the same category but are actually different) (nonexistent objects: man + woman, nonexistent attributes: brown + yellow, nonexistent quantities: two - three, etc.). The instructions should contain interrogative and declarative sentences.
The output format needs to be list only which contains the misleading entity names. Please follow the instructions carefully.
The length of output list needs to be exactly equal the input list.
2. Do not explain the reasons.
3. Do mention the input entities, at least the output and input name needs be different.
4. Do not mention something abstract, like älien:
5. When dealing with quantities, focus solety increasing the numbers during revision.
When dealing with words like few", group". "several", "some", etc., try changing the objects (A few men + A few women).
7. Ensure that inclusive words "people," avoid replacing it with genders like "man" or "woman." Instead, consider modifying them to different categories, such as "people" "animals."."
substituted with their specific subsets. For example, if the word
17

---

Published as a conference paper at ICLR 2024
C.3 EXAMPLES FOR GENERATING REFER- AND-GRUNDDD DATASETS
We provide some example prompts to generate refer-and-ground from ChatGPT/GPT-4 Prompt and the in-context example of multiple-round visual conversation data shown in Table TI and Table 12 Prompt and the in-context example of one-round reasoning data are shown in Table 13 and Table
Table 11: In this example, we provide the prompt used to generate the conversation response for refer-and-ground instruction tuning, following the practice of LLaVA (Liu et al. 2023b).
messages ("role": "system", "content": You are Al visual assistant that analyze single image. You receive five global captions, each describing the image you are observing. In addition, specific object locations within the image are given, along with detailed coordinates. These coordinates are the form bounding boxes, represented as (xl, yl, x2, y2) with floating numbers ranging from to 1. These values correspond to the top left x. top left bottom right and bottom right y. Also, the relationships between pairs of objects relationship - subject, where the object/subject from previous object lists as well as ohject names. Also, several region descriptions are given, each describing a box region the image, with detailed coordinates.
provided in indexed by object id
the format of object
Design conversation between you and person asking about this photo. Ask diverse questions and give corresponding answers. The answers should a tone a visual assistant seeing the image and answering the question.
Here some additional requirements about generated questions and answers:
Only include questions that definite answers:
(1) one can see the content in the image that the question asks about and can answer confidently; (2) one can determine confidently from the image that it is not in the image. Do not ask any questions that cannot be answered confidently.
Also include complex questions are relevant to content in the image, for example, asking about hackground knowledge of the objects in the image, asking to discuss events happening in the image, asking about object actions in the context of entire images, etc. Again, do not ask about uncertain details.
3. Provide detailed answers when answering complex questions. For example, give detailed examples or reasoning steps to make the content more convincing and well-organized. You can include multiple paragraphs if necessary.
all samples, either question or answer, you must mention bounding box coordinates refer to the object or regions instead directly saying the object name or describing the regions in text. In answer, explain the region in the context of the scene.
Do not mention that the information source provided Always answer as if you are directly looking at the image.
text/caption/region description.
6. Make the question as diverse possible. Include questions asking about the visual content of the image, including the object types, counting the objects, object actions, object locations, relative positions between objects, object selection, object functions, etc. Make question challenging by less including the visual content details in the question."")
for sample in fewshot.sane sta.hootssmmless:: cessages.appent (f "rolee: "user", "content" :sappl 'context'
ressages append (1 "rzole:; "assistant", "content : sample| 'response"
messages append (( ("role"; "user", "content"; 'In". join (query)
18

---

Published as a conference paper at ICLR 2024
Table 12: One example used in in-context learning to construct GPT-Assisted Refer-and Sonn-Ground Instructon -Tunig... We encourage readers to refer to the codebase for the prompts for other types.
Objects
Object O: chair at 10.596, 0.637, 0.698, 0.997].
Relationships
Object II frame - with Objoctt 9 photo
Region Descriptions
Region Description
0.466, 0.600, 0.529]
white picture frame with black and white photo on it.
Global Caption
White chairs sit around a polished wood dining table while sectional soft sits the background.
Conversation
Question:
What the object (0.212, 0.539, 0.718, 0.820] made of?
Answer:
The dining table over there is crafted from tree trunk.
Question:
Where a good place have my
out locations in image.
Answer:
suggest having dinner the table (0.214, 0,541, 0.720, 0.997], possibly sitting the white chair [0.264, 0.717, 0.524, 0.995).
Also, enjoy the wine (0.242, 0.644, 0,288, 0.682).
Question: In terms comfort resting. which object is better one [0.830, 0,411, 0.978, 0.596] or the one [0.264, 0.717, 0.524, 0.995J?
Answer:
Among the two, the couch (0.830, 0.431, 0.978, 0.596] would be more comfortable for resting. The object [0,264, 0.717, 0.524, 0.995] dining chair, typically comfortable extended
Question:
Is item 10.182. 0,238, 0,342, 0.461 real is it mirror reflection?
Answer:
The television and the firoplane [0. 182, 0,238, 0,342, 0,461] are not real but are actually reflections the mirror.
Question:
Can you describe decorating style the living room shown this image?
Answer:
The living decorated modern chic The image shows white chairs and lamps, gray couch, and wooden table and floor, all contributing to modern aesthotic. The large mirror and built-in wine rack are also characteristic of modern design. the overall decor creates atmosphere elegance sophistication, indicating
style.
19

---

Published as a conference paper at ICLR 2024
Table 13: In this example, provide the prompt used to generate the reasoning response for refer- and-ground instruction tuning, following the practice of LLaVA (Liu et al. 2023b).
messages {"role": "system", "content"; Y"u are visual assistant that can analyze single image. You receive five global captions, each describing the same image you are observing. In addition, specific object locations within the image are given, along with detailed coordinates. These coordinates
in the form of bounding boxes, represented as (x1, yl, x2, y2) with floating numbers ranging from 1. These values correspond the top left x. top left y. bottom right x, bottom right y. Also, relationships between pairs objects are provided, the format of object - relationship - subject, where the object/subject are indexed by object from previous object lists as well as the object names. Also, several region descriptions are given, each deseribing box region of the image, with detailed coordinates.
task use provided image information (objects, attribute, relationship, region description, captions), create a plausible and challenging question about the image, and provide the answer in detail.
Create complex questions that mention specific regions image, but question should require some knowledge-awws aware or high-level commonsense reasoning beyond describing the scene.
To answer such questions,
should first understand the visual content, then based the background knowledge reasoning, either explain why the things happening that way provide guides help user's request. Make the question challenging by not including the visual content details in the question so that the user needs to reason about that first.
Here some additional requirements about generated questions and answers:
In question or answer, you must mention bounding box coordinates to refer to the object or regions, instead of directly say the object name or describing the regions in text. In answers, explain the region in the context of scene. Include details like object counts, position of the objects, relative position between the objects.
2. Don't t ask the question you are not confident to answer. Only include question that have definite answer.
Do mention that the information source provided in text/catpion/region description. Always answer as you are directly looking at the image.
4. Make the question as diverse as possible and as complex-reasoning required possible."")
for sample in nowshot.samplos:
messages. append ({ ("role": "user", "content :onntt"saen sample 'context! 1])
me ressages.appent "role": "assistant", "content" ": samppee sample "response")
messages. append ( (("role"; "user", "content"; 'ln", join (query) }
20

---

Published as a conference paper at ICLR 2024
Table 14:
One example used in in-context learning construct GPT-Assisted Refer-and-Grounl Cnsructon-Tuuing We encourage readers to refer to the codebase for the prompts for other types.
Objects
Object badge [0.802, 0.505, 0.872, 0.581).
Relationships
Object 5 fringe
Objoct 4 flag
Region Descriptions
Region Description at [0.038, 0.029, 0,508, 0.995]
Grinning gentlemen about get
Global Caption
A female military uniform cutting businessman's neck tie.
Conversation
Question:
What uousual about the region [0. 102, 0.230, 0.578, 0.761J?
Answer:
In the specified region, there is an intriguing sight: a pair oversized scissons positioned near man's tie, accompanied by in military attire. The two them share hearty laugh, creating unexpected contrast. The atmosphere appears celebratory, resembling ribbon-cutting ecremony with flags in the background. However, rather than cutting a ribbon. the the situation.
are directed man's adding unexpocted
D EXAMPLES AND PROMPTS FOR FERRET-BENCH
We leverage GPT-4 to generate three kinds of region-based questions evaluating referring and grounding capability: (i) Referring Description, (ii) Referring Reasoning, and (iii) Grounding in Conversation. Here, we only provide the prompt in Table 15 used to generate the referring descrip- tion response. One example of GPT-4 answers shown Table 16
We recommend readers check
out more examples in Appendix E
21

---

Published as a conference paper at ICLR 2024
Table 15: In this example, we provide the prompt used to generate the referring description response.
Y""Yo are an visual assistant can analyze single image. You receive five global captions, each describing the same image you are observing. In addition, specific object locations within the image are given, along with detailed coordinates. These coordinates
messages ("role": "system", "content""
in the form of bounding boxes, represented (x1, These values correspond
x2, y2) top left x, top left y.
with floating numbers ranging from bottom right x, and bottom right y. Also, the relationships between pairs of objects are provided, the format of object + relationship + subject, where the object/subject are indexed by object from previous object lists well the object names. Also, several region description are given, region image, with detailed coordinates.
each describing
The task is to use the provided image information (objects, attribute, relationship. region description, captions), create plausible and challenging question about the image, and provide the answer in detail.
Create questions that refer to coordinates some objects or regions without describing it, and ask about its interaction with surrounding/nearhy objects.
To such questions, should require first understanding the visual content, then based the spatial information provided.
Here are some additional requirements about generated questions and answers:
question, you must mention bounding box coordinates refer the object regions, instead directly say the object name or describing the regions in text. In answers, explain the region the context of scene. Include details like object counts, position of the objects, relative position between the objects.
Don't ask question you are not confident answer. Only include question that have definite answer.
3. Do not mention that the information source is provided in text/catpiinn/rgiion description. Always if you directly looking the image.
4. Don't mention additional coordinates in the answer.
5. Question should be explicitly ask about context/urroundinnnnnayy information//nteraaciion.
for sample in fowshot.sampless
messages.appen (1 ("role": "user", "content" : sample 'context'' '
bessages.appent "role": "assistant", "content "; sample ("response)
messages append (( (["role": "user", "content"; Mnn join (query) }
22

---

Published as a conference paper at ICLR 2024
Table 16: One example of GPT4's answers about referring description.
Conversation
Question:
Describe the region [0. 102, 0.230, 0.578, 0.761] in the context of surrounding objects.
Answer:
It's the region, who standing and tree near him. Also, be in front of some buildings.
skatehoard, performing rail grind metal hand mill There bench
E MORE VISUALIZATION
We provide quantitative results of the predictions under various tasks from Ferret to indicate the model's strength and capability.
Please refer Figure 5 for Referring Object Classification on LVIS with different referring for- mats (point/box/).
Please refer to Figure 6 for Visual Grounding on Flickr30k Entities and Referring Expression Comprehension on ReICOCCRReECOCO--RRRCCOO..
Please refer to Figure for Grounded Captioning Flickr30k Karpathy split.
Please refer to Figure for Evaluating Object Hallucination (POPE) COCO val split.
Please refer to Table 17
for Referring Description in Ferret-Bench.
Please refer to Table 18
for Referring Resoning in Ferret-Benche
Please refer to Table 19 for Grounding in Conversation in Ferret-Bench.
Referring Object Classification (LVIS) point
Referring Object Classificatiin (LVIS) point
theikpodt Jooy ingonn suppour
netigerattor chei??
bu moiggaatt (oont).
Addok SeAm (heads
Referring Object Classification (LVIS) box
Referring Object Classification (LVIS) - free farm shape
(rggin0)
heoddaand
E/ Aarissaade (boo)).
Figure 5: Referring Object Classification on LVIS. The task aims to classify specific region(s) in an image given by soim/bbx/seemeetation inputs.
23

---

Published as a conference paper at ICLR 2024
Phrase Grounding 30k Entities)
Referring Expression Comprehension (RefCOCO)
are bicatinn of man
What the looaoonn canroa the inage?
wood?
erean Boadl lourdooa
cermaaa (ood))
Locate pariully wiille zetea the image??
and concvete cononee wall?
anveraliskkoe concrate wall Boon
particl vouble adee (ood)
Could Inceronccn daps, canall
Find top right
dogs (nced] (boo)) (bocc))
howd [nead)
Referring Expression Comprehension (Refcoco+)
Referring Expression Comprehension (RefcocOg)
locutam moot but thai thed ooanting fiom Foe butto mog aider
What location donut lile neresst ooffee inager?
Ahour with holn neoreet cottee
emrrral Inddion loy ludt lie leid up (sced].
find with koonee plane to las HKW
Locate a hoose with white soicre
cur with vtaire Boorse plate fosics
with strie mule
Talime precire
ueaaller lapin
numbeer (hoa2)
large toddy mumaler (ono])
Figure 6: Phrase Grounding Flickr30k Entities and Referring Expression Comprehension on BefcoCOOReCoCCOHoeeCCcOO The tasks aim to localize specific object(s) in an image described by referring expression/given entity.
Grounded Captioning (Flickr30k Karpathy split)
Write lotel deactppiin the
festures the
Aooree Boodd woottng to brace vollobell (oonc))
bang unty cundder (ooun in
What thipphoo thoutruke
Provide arowerii short
cort Sheppeent
Jeseor bostl a tackted by player [pood].
overal ins fow wordu?
pictust Eogain briely.
Divont and ande fot (boa7) (iliin
Josil and [bo02] andia
Figure 7: Grounded Captioning Flickr30k. The task aims generate caption about the image and ground all generated noun phrases to image regions.
24

---

Published as a conference paper at ICLR 2024
POPE (COCO adversarial)
there microwove
Yea, Uare uirrroowee (bond] image.
No. soti but thene toaster
reneltracitt the
ls trrpe lncodial inlonci the
No. theen diou tannis inae
in the image
POPE (COCO popular)
liforre porsooii inrage?
There char in imag??
YAL, thare ltA prrtion (hool]) tne
there thenica (hootili
Iellare lubee indie maggo?
there [boet]]
inore skotebourd boold) in
POPE (COCO random)
bo Daoe albooool in the inage?
kithere boodl in tnithe image??
You there is trocooll (boxd)
No, Ehano maroonn
lothore conson inage?
No. thene ic pernzc, But thereil
Image
Figure 8: Object Hallucination Evaluation (POPE) on COCO. The task aims to input query inquiring about the existence of an object, and the model is expected generate a response in the form of either "yes/no""
25

---

Published as a conference paper at ICLR 2024
Table 17: Referring Description in Ferret-Bench. Qualitative examples illustrate the difference between various models (LLaVA vs. Kosmos-2 vs. Shikra vs. Ferret (Ours)). For clarity, we have omitted the bounding box outputs in the textual representations.
Question:
What the interaction between object [regionO] and its surrounding objects?
Ground Truth:
In the region, there is metal support column. This column providing support bridge above which a train is passing over. The column also features a red line on it. This region is part of larger scene that includes train track elevated bridge.
LLaVA:
The object is a bridge which appears to be part of train station, as there are traffic lights present in the scene. The traffic lights are likely used to control the train's 's movement and ensure safe passage over the bridge.
Kosmos-2:
The interaction between the object and surroundings that a train passing over a bridge.
Shikra:
The metal pole is standing tall and straight, supporting the train that is traveling the elevated track.
Ferret (Ours):
The object is metal post that is part of bridge. This post is essential because supports the bridge structure, ensuring its stability for the train to safely traverse the bridge. Other elements such the train, and the overhead guard rail, depend this post for support and stability. Without this post, bridge wouldn't able withstand the train's weight and motion, potentially leading a structural collapse. Therefore, this post plays a critical role in maintaining the bridge's integrity and stability.
Question:
What is the relationship between the object [region0] and object [region]] and its surrounding objects?
Ground Truth:
The object a vase, and the object a flower. The flower is in the vase, suggesting it decorative element within the room.
LLaVA:
wooden table with a green bowl placed on it.
Kosmos-2:
The object is with green base.
Shikra:
The answer flowers.
Ferret (Ours):
The object and object are vase and flower, respectively. The vase is holding the flower, suggest- ing it's displaying or supporting it.
26

---

Published as a conference paper at ICLR 2024
Table 18: Referring Reasoning in Ferret-Bench. Qualitative examples illustrate the difference between various models (LLaVA vs. Kosmos-2 vs. Shikra vs. Ferret (Ours)). For clarity, we have omitted the bounding box outputs in the textual representations.
Question:
What the purpose of the object [regionO]?
Ground Truth:
The object is bottle. Bottles are typically used to hold and store different types of liquids. In this context, might he used
heverage for the meal.
LLaVA:
A knife and fork, indicating that it is ready for meal.
Kosmos-2:
The purpose of object attract the birds to the table.
Shikra:
The answer keep birds away.
Ferret (Ours):
The object bottle, generally used to and dispense liquids like water, juice, other bever- ages.
Question:
What the purpose the object [regionO] on bike?
Ground Truth:
The object is shock absorber the bike. Its purpose is absorb dampen shock impulses. does this by converting the kinetic energy of the shock into another form of energy which is then dissipated. In the context of the motorcycle, it is particularly useful in providing comfort and stability for the rider, especially when travelling over uneven or rough terrains.
I.LVVA:
black and silver Kawasaki motorcycle which is likely used for transportation, commuting, leisure rides, or long-distance travel.
Kosmos-2:
The purpose of the object on the motorcycle is to store and protect the motorcycle's fuel tank.
Shikra:
The answer leg.
Ferret (Ours):
The object is shock absorber. This component is essential for the bike's stability, providing a smoother ride by soaking up the impact from bumps and uneven surfaces. It significantly improves the overall performance and comfort of the bike, enabling the rider to maintain control and stability even on rough terrains.
27

---

Published as a conference paper at ICLR 2024
Grounding in Conversation Ferret-Bench. Qualitative examples to illustrate the difference between various models (Kosmos-2 vs. Shikra vs. Ferret (Ours)), For clarity, we only display the bounding box outputs Ferret 's responses.
Table 19:
Question:
What is happening in the field and provide the coordinates of mentioned objects?
Ground Truth:
the image, there is a group of sheep [box0] [box grazing in field [box2]. The field is filled with plant life [box3]. There is also a bush [box4] and trees [box5] present in the field.
Kosmos-2:
In the field, a herd of sheep [box0] is grazing on the lush green grass [box I..
Shikra:
group of sheep [box0] [box1] [box2] [box3] [box4] [box5]
grazing on lush green grass.
Ferret (Ours):
In the field, there sheep [box0] [box1] [box2] [box3] [box4] [box5] grazing the grass [box6].
Question:
What is the man doing and what objects are in front of him? Please provide the coordinates of these objects.
Ground Truth:
The man [box0] is sitting at a table [box1] and looking plate of food [box2]. There is glass [box3] and a bottle [box4]
table front of him.
Kosmos-2:
The man is sitting at dining table [hox0] smiling and looking at the plate of food in front him [box1].
Shikra:
The man [box0] is sitting at a table [box with plate of food [box2] in front of him. He is looking and smiling, possibly staring person [box0] in the restaurant Ferret (Ours):
man [box0] is having meal a table [box]]. front him, there a plate [box2] with a variety food items such as fruit and eggs. There is also a glass [box3] filled with water and a bottle [box4].
28


Reviews:
[{"review_id": "i1252noeBU", "reviewer": "Reviewer_7dkr", "summary": "The paper tackles the challenge of training a Multi-modal Large Language Model to accurately interpret input visual references, such as points, bounding boxes, free-form shapes, with respect to the image (referring) and ground the output text to relevant image regions (grounding). The authors propose a unified framework, called Ferret, for jointly solving the visual referring and grounding problem. They provide a new curated dataset (GRIT) that consists of existing and newly collected data for training, as well as a new benchmark (Ferret-Bench). In the evaluation section, the authors show that the proposed method either exceeds (on the Ferret-Bench and grounded captioning) or is on par with the concurrent SOTA methods such as Shikra. In addition, the proposed method can accept a variety of user input on images as part of referring expressions, including scribble and freeform shapes, in addition to the traditional points and bounding boxes. However, the way how these different visual reference types are processed is conceptually similar to the Visual Sampler proposed in SEEM (Zou et al., 2023), and performs similarly despite the added complexity.", "questions": "Q1. It appears that the proposed method significantly outperforms Shikra on Ferret-Bench, but not much on existing datasets. I wonder if the authors can explain why.\n\nQ2. I think the quality of writing could be further improved. For example, it is not clear to me what the authors are trying to imply by “First of all, we choose MLLM as the bedrock of Ferret due to their powerful vision-language global understanding capability”. I am guessing that they wanted to say that the Ferret is built on top of existing MLLMs to leverage their powerful vision-language capability?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. This seems to be one of the first MLLMs to support a variety of visual reference types, such as point, box, scribble, polygons, and masks. \n\nS2. The authors provide a curated dataset called GRIT that consists of existing datasets and newly collected data for training MLLMs with visual referring and grounding capabilities.\n\nS3.  The authors provide a new benchmark, Ferret-Bench, which covers two new types of evaluation task for visual referencing (description and reasoning) in addition to the conversation grounding task. The key difference to existing benchmarks such as RefCOCO+, RefCOCOg, or PointQA [a] is that the questions include visual references (in forms of bounding boxes). For example, “What is the purpose of the object [x1 y1 x2 y2]?”\n - [a] Point and Ask: Incorporating Pointing into Visual Question Answering, Mani et al., 2022\n\nS4. The paper provides comparison with the SOTA methods and the concurrent methods in the evaluation section.", "weaknesses": "W1. The paper omits any discussion on the limitations or potential failure scenarios of the proposed method.\n\nW2. The significance of the proposed Spatial-Aware Visual Sampler is minimal. The idea of sampling the visual features over the grid is in the same spirit as the Visual Sampler in SEEM (Zou et al., 2023), although the details of how the points features are aggregated and pooled are different. Performance-wise, the Spatial-Aware Visual Sampler is shown to be only marginally better than the Visual Sampler in SEEM as shown in the ablation study section.\n\nW3. While the idea of jointly solving referring (with explicit visual cues, such as markings on the image) and grounding in one unified framework makes sense as the two tasks are interrelated, this idea was also explored in concurrent works (Chen et al., 2023a) and (Peng et al., 2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5F5wI4Dn3k", "reviewer": "Reviewer_6Qgf", "summary": "This paper introduces Ferret, a Multimodal Large Language Model (MLLM) capable of understanding spatial referring within an image and grounding open-vocabulary descriptions. The paper also introduces the dataset GRIT with 1.1M samples and an additional 130K hard negative data.\n\nFerret includes several components:\n1. A powerful hybrid region representation integrating discrete coordinates and continuous features to represent image regions.\n2. A spatial-aware visual sampler for handling various region shapes and extracting continuous features.\n3. Integration with LLM for referring and grounding tasks.\n\nFerret achieves superior performance in classical referring and grounding tasks and outperforms existing methods and it shows improved capability in describing image details and reduces object hallucination.", "questions": "1. Table 5 shows that mutual benefits of grounding and referring. From the results, it seems grounding task can help more for referring task than the other way around. How to interpret this effect?\n2. For section 4.2, how to evaluate quality of the generated data from ChatGPT and GPT4?\n3. You mensioned in the Ferret-Bench is via GPT4 as a judge. But some of the data are collected from GPT4, is a reason it is better than all other models in Table 7.\n4. Can chatgpt take multimodal input? Since when you collecting dataset via LLM, the author mentioned they use ChatGPT first and then use GPT-4 to refine it. I am wondering how ChatGPT can take image as input.\n\nIt is extensive of work. I would like to raise my score if the questions are addressed and the code and data are public.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is presented very well. \n2. The paper shows a reasonable motivation that humans inherently possess the ability to learn from one task and generalize to another between referring and grounding. This underscores the essential need to unify referring and grounding processes.\n3. The hybrid region representation and spatial-aware visual sampler make the framework flexible to take different form of region definition.\n4. The framework shows a good way of utilization of Large Language Model.\n5. Contribution of the dataset.", "weaknesses": "1. No open source code for the code and dataset. I would raise the soundness score if code and dataset are open, either attached in the supplementary or released in the public repo.\n2. The hierarchy of the dataset is a bit complicated. This may not be practical for costume dataset.\n3. Very engineering paper, extensive work, but not much scientific novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DNAMBvxY0W", "reviewer": "Reviewer_dfTo", "summary": "This paper proposes a large multimodal model Ferret, which, compared with prior works, is especially good at referring and grounding. In Ferret, they propose a hybrid region representation and a novel spatial-aware visual sampler to represent the visual and regional input. To train Ferret, they creat GRIT, a large-scale ground-and-refer instruction tuning dataset. They also introduce Ferret-Bench to evaluate the grounding, referring, and reasoning capabilities of LMMs.", "questions": "1. Is the evaluation on Flickr30k grounded caption fine-tuned or zero-shot?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed GRIT dataset is meaningful to the vision and language research. \n2. The proposed Spatial-Aware Visual Sampler and Hybrid Region Representation are well-motivated. \n3. The experiment results show the better capabilities of the trained model on multiple referring and grounding tasks and validate the effectiveness of the spatial-aware visual sampler module.", "weaknesses": "1. The ablation on hybrid region representation is missing.\n2. Not a strong weakness, but whether the model performs well on non-referring or grounding tasks needs more validation. E.g. VQA_v2, MME, general captioning, etc. And it seems the caption evaluation is not as good as InstructBLIP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)