Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models
Authors: Haotian Zhang
Haoxuan You
Philipp Dufter
Bowen Zhang
Chen Chen
Hong-You Chen
Tsu-Jui Fu
William Yang Wang
Shih-Fu Chang
Zhe Gan
Yinfei Yang
Venue: COLM (2024)
Keywords: Multimodal Large Language Model, Referring, Grounding
TL;DR: Ferret-v2, a significant upgrade to Ferret, which excels in detailed vision-related tasks without compromising their proficiency in global understanding.

Abstract: While Ferret seamlessly integrates regional understanding into the Large Language Model (LLM) to facilitate its referring and grounding capability, it poses certain limitations: constrained by the pre-trained fixed visual encoder and failed to perform well on broader tasks. In this work, we unveil Ferret-v2, a significant upgrade to Ferret, with three key designs. (1) Any resolution grounding and referring: A flexible approach that effortlessly handles higher image resolution, improving the model's ability to process and understand images in greater detail. (2) Multi-granularity visual encoding: By integrating the additional DINOv2 encoder, the model learns better and diverse underlying contexts for global and fine-grained visual information. (3) A three-stage training paradigm: Besides image-caption alignment, an additional stage is proposed for high-resolution dense alignment before the final instruction tuning. Experiments show that Ferret-v2 provides substantial improvements over Ferret and other state-of-the-art methods, thanks to its high-resolution scaling and fine-grained visual processing.

Published as a conference paper at COLM 2024
Ferret-v2: An Improved Baseline for Referring and Ground- ing with Large Language Models
Haotian Zhang", Haoxuan You Philipp Dufter', Bowen Zhang', Chen Chen',
Hong-You Chen', Tsu-Jui Fu², William Yang Wang', Shih-Fu Chang?, Zhe Gan', Yinfei Yang
1 AApple AI/ML, CCllumiaa University, PUC Santa Barbara
{haotian whang2,aann2ammm zhe. gan, yin.yyineeiyyeaplll.. com, haoxuan, www.eo..oo.ubiaaaa..ooooooooaaa. edu, tsu-jui fu@ucsb. eddu
Referring:
Benchmark
Whe wort is strow in aws [regiondy
Performance:
Forret
The rogion orow The word
The word "Groat is daplayed In
F
37.5
Perb000
Grounding:
la there any stuft the image to help peocle cool down summer?
13.
YIA-ABI BTT
Yen, there large trock (bood)
thad has taited windonss for cod whects
Famat
Great
ABTOMOБNЛи C NPOGEROM
NPOOAAA OBMEN KORHCCKA Duasyn AUTO
8815
[[boei)) (bos2), (boady (boodd
Yos there conditoners
61-7
(boutsi
acoling people in the
Ferret-v2 Ferret
(8) Qualttaive Exampie
(b) Quanttative Resul
Figure 1: (a) The comparison showcases Ferret-v2's superior referring and grounding abilities over Ferret, particularly in identifying objects and texts within small regions. (b) Ferret-v2 notably exceeds Ferret's performance in tasks requiring detailed regional and global reasoning and understanding (all w/ 7B models).
Abstract
While Ferret seamlessly integrates regional understanding into the Large Language Model (LLM) to facilitate its referring and grounding capability, it poses certain limitations: constrained by the pre-trained fixed visual en- coder and failed to perform well on broader tasks. In this work, we unveil Ferret-v2, significant upgrade Ferret, with three key designs. (1) Any resolution grounding and referring: A flexible approach that effortlessly handles higher image resolution, improving the model's 's ability to process and understand images in greater detail. (2) Multi-granularin visual encod- ing: By integrating the additional DINOv2 encoder, the model learns better and diverse underlying contexts for global and fine-grained visual informa- tion. (3) A three-stage training paradigm: Besides image-caption alignment, an additional stage is proposed for high-resolution dense alignment before the final instruction tuning. Experiments show that Ferret-v2 provides substantial improvements over Ferret and other state-of- state-of-the-art methods, thanks to its high-resolution scaling and fine-grained visual processing.
1 Introduction
Multimodal Large Language Models (MLLMs) (Koh et al., 2023; Wu et al., 2023a; Yang et al., 2023; Liu et al., 2023d; Wu et al., 2023c; Li et al., 2023d; Ye et al., 2023; Wu et al., 2023b; Li et al., 2023a; Wang et al., 2023a; Gao et al., 2024; McKinzie et al., 2024) have increasingly become pivotal in the recent surge of advancements in AI, serving as foundational elements
*Equal contribution.

---

Published as a conference paper at COLM 2024
in the development of versatile general-purpose assistants. However, these methods were built on coarse image-level aligmments, which suffer from fine-grained understanding (such as region description and reasoning). To this end, Peng et al. (2023); Chen et al. (2023b); You et al. (2023) integrate the grounding abilities and unlock the referential ability in dialogue, i.e. enable the user to point to the object region input, and the model response with spatial coordinates of bounding boxes. Such capability fulfills fine-grained vision tasks, which is great progress in MLLMs.
While grounding and referring MLLMs exhibit strong performance, there are still many chal- lenges that remain unresolved. For example, the aforementioned methods use CLIP (Jiang et al., 2023) or its variants (Sun et al., 2023) as the vision encoder. As the pre-trained image encoders normally adopt a relatively low image resolution, e.g., 224x224, it severely hinders fine-grained visual comprehension for MLLMs. Though some task-specific MLLMss (Lv et al., 2023; Hong et al., 2023; Ye et al., 2023) have explored strategies for upscale processing, these approaches are marred by undue complexity for their own domains and cannot per- form well on traditional MLLM benchmarks. Thus, the scenario prompts a critical inquiry: how can enhance the capabilities of MLLMs to excel in detailed vision-related tasks without compromising their proficiency in global reasoning?
To answer this question, we explore the potential from three aspects, i.e., higher-resolution scaling, multi-granularity visual encoding, and model training recipes. We choose Fer- ret (You et al., 2023) as the robust baseline since it has two advantages: (i) mutual benefits between referring and grounding, and (ii) more versatile referring capability (strokes, scribbles, or complex polygons). Firstly, we conduct a careful investigation into higher- resolution scaling, and evaluate the performance of two mainstream methods, "direct upsampling" (Wang et al., 2023a; Bai et al., 2023) and "any resolution" (Gao et al., 2024; Liu et al., 2024), on (i) Visual detail analysis (ROC (You et al., 2023) & REC (Kazemzadeh et al., 2014)), (ii) Resolution-critica OCR tasks (TextVQ (Singh et al., 2019)), and (iii) Reasoning MLLM benchmarks (Ferret-Bend) (You et al., 2023)). Our analysis indicates that the "any resolution" approach outperforms "direct upsampling" in harnessing image details while retaining the knowledge acquired during pre-training for efficient scaling. This positions "any resolution" as a superior strategy for tasks requiring advanced visual comprehension.
By adopting the "any resolution" method, which involves dividing the image into sub- patches for processing by the CLIP encoder, we observed that incorporating both global context and high-resolution patches into visual embeddings introduces nuanced com- plexity. This is because two types of images exhibits distinct characteristics. To mitigate this gap, we propose the integration of a DINOv2 encoder (Oquab et al., 2023). Renowned for its proficiency in delineating finer details pertaining to local objects, DINOv2 promises to bolster the model's ability to perceive fine-grained aspects. Additionally, we employ separate MLP projectors for each vision encoder to facilitate a deeper exploration of the varying contexts presented by global and fine-grained visual information, aiming for a more comprehensive understanding and representation.
Furthermore, the model is strategically trained in three stages, enhancing resolution han- dling while maintaining vasion-language alignment in a "coarse-to-fine manner. Initially, the model trained on low-resolution images for efficient image-caption alignment. Subse- quently, we recognize the gap that several downstream tasks demand more accurate and thorough spatial understanding and go beyond just the broad semantics, so we specifically design the 2nd stage to align every possible local object of the image with detailed semantics with dense referring and detection data. Finally, the model undergoes visual instruction fine-tuning to better interpret user intent.
The contributions of this paper are summarized as follows: (i) We provide a thorough analysis of higher-resolution scaling, and found that the ""nny resolution" method consis- tently outperforms "direct upsampling". (ti) Based on "any resolution", we further propose multi--raaullaiity visual encoding, where the low-resolution image is encoded via CLIP, while the high-resolution sub-patches are encoded via DINOv2. This strategy fosters deeper understanding of both global and fine-grained visual contexts. (iii) F2 is trained in a three-stage process, where an additional stage is proposed for high-resolution dense alignment before the final instruction tuning. Extensive experiments on a wide range
2

---

Published as a conference paper at COLM 2024
of tasks, including referring and grounding, visual question answering, and modern MLLM benchmarks demonstrate the superiority of Ferret-v2 over existing works (see Fig. 1).
2 Background
Coarse-level MLLMs. Motivated by the advanced reasoning abilities demonstrated by LLMs (OpenAI, 2022; Chowdhery et al., 2022; Touvron et al., 2023a;b; Zhang et al., 2022b; Wei et al., 2021), there a growing interest in extending these skills to visual understanding, leading to the emergence of multimodal LLMs. For example, Flamingo (Alayrac et al., 2022) utilizes cross-attention mechanism to enhance visual context awareness, enabling more sophisticated context-aware visual learning. Models such as LLaVA (Liu et al., 2023b;a) and MiniGIT-44 (Zhu et al., 2023) focus on synchronizing image and text features before applying instruction tuning. Additionally, BLIP-2 (Li et al., 2023d) and mPLUG-OWL (Ye et al., 2023) offer methods for incorporating image features using a visual encoder, which is then combined with textual embeddings in the LLM architecture. Nonetheless, despite their advancements, these MLLMs, including the latest GPT-4V (OpenAT, 2023), are limited to producing text outputs, restricting their application in scenarios that demand rich region- level visual perception.
Region-level MLLMs. In recent investigations, there has been growing focus on the convergence of foundation models and the tasks related to dense visual perception. For example, Li et al. (2023c); Zou al. (2023); Koh et al. (2023) leverage the CLIP pre-trained foundation models to enable open-world detection, but they are unable to handle complex instructions. Differently, VisionL.LM (Wang al., 2023c) combines a range of vision-centric tasks by utilizing instruction tuning with LLMs. However, it may fall short of fully harness- ing the potential of LLMs for handling intricate reasoning tasks. In parallel research efforts, grounding capabilities and open-vocabularies detectors are leveraged by Kosmos-2 (I'ng et al., 2023), Qwen-VL (Bai et al., 2023) and DetGPT (Pi et al., 2023), enabling aser-guided detection. Moreover, GPT4Rol (Zhang et al., 2023b), Shikra (Chen et al., 2023b), LLaVA- G (Zhang et al., 2023a), and Ferret (You et al., 2023) introduce spatial boxes as input and train the model using region-text pairs, offering regional image understanding. However, all the above methods utilize low-resolution image encoders and thus limit the capability of perceiving more detailed analysis.
3 Methods
We first revisit the design principles of Ferret in Sec. 3.1 and present the investigation into siigher-esolutionnscllllle in Sec. 3.2. Subsequently, in Sec. 3.3, we delve into advancements in the model architecture, including techniques for grounding and referring at any resolution, as well as visual encoding with multiple granularities. Finally, we introduce an enhanced training methodology aimed at refining the model's proficiency in aligning global and local elements in Sec. 3.4.
3.1 A Revisit of Ferret
In recent investigations, there has been a growing focus on the convergence of models (Zhang et al., 2023b; Chen et al., 2023b; Peng et al., 2023; Lai et al., 2023; Zhao et al., 2023; You et al., 2023) and the tasks related to visual perception. Ferret (You al., 2023) distinguishes itself from other MLLMs by excelling in spatial referring and grounding within natural images of diverse shapes and levels of detail.
To refer to various types of regions, such as points, boxes, or free-form shapes, Ferret de- veloped hybrid region representation, where each region is referred to by a combination of discrete coordinate tokens and continuous region features, as well as region names if available. The coordinates are normalized into the range from to 1000, and a point or shape is respectively expressed by [x,y] or [Xminz Amax /max The continuous region mmax,
feature is extracted by a spatial-aware visual sampler that samples and aggregates fea- tures of the region. Ultimately, region is represented by "(region name) (coordinates)
3

---

Published as a conference paper at COLM 2024
(a) ROC (LVIS-box).
(b) REC (RefCOCOg).
(c) TextVQA
(d) Ferret-Bench.
Figure 2: Performance of "direct upsampling" and "any resolution" w/ 448x448 image
resolution in ROC, REC, TextVQA, and Ferret-Bench. (* indicates the encoder is frozen during fine-tuning. denoted as vanilla Ferret w/ image resolution of 336x336.)
(continuous fea)" and fed into the model for referring, e.g. "What in the region [100, 50, 200, 300] (continuous fea)?". To achieve grounding, Ferret generates the box coordinates right after the corresponding regions/nous 150, 300, 200] in the figure."
the text response, e.g., "There a dog [100,
Ferret encodes the image with a pre-trained visual encoder (CLIP-ViT-L/14) (Radford et al., 2021) and then feeds the image feature as additional tokens alongside the text input (and hybrid region representation if any) into a decoder-only language model (Vicuna (Zheng et al., 2023)). The training contains two stages, image-caption alignment pre-training and iustrucioon-uning updated with next-token-prediciion loss.
While Ferret boasts flexibility and superior performance, it is hindered by the limitations imposed by the fixed resolution of its pre-trained encoder, which restricts its ability to fully exploit the advantages of enhanced region referring and localization accuracy. Motivated by this, we initially delve into identifying the most efficacious methods for uigh-resolution scaling. Subsequently, we unveil Ferret-v2, substantial extension of the Ferret series, aimed at examining broader and more inclusive multimodal learning framework.
3.2 Analysis of Higher Resolution Scaling
For further analysis, conduct series of controlled experiments using different high- resolution scaling methods, i.e., "direct upsampling", and "any resolution" "(Liu et al., 2024). The overall architecture and training process follows Ferret (You et al., 2023) but with simple modification from linear layer to a two-layer Multi-Layer Perceptrons (MLPs). Additionally, to enable the model better handle short-form answers and perform on more benchmarks, we follow LLaVA 1.5 (Liu et al., 2023b) and add additional task-oriented datasets for VQA (Antol et al., 2015) and OCR the existing GRIT (You et al., 2023), which was previously used in Ferret. To streamline our study, we choose representative tasks: ROC (LVIS: box), REC (RefCOCOg), TextVQA, and Ferret-Bende and measure the capability of the trained models comprehensively.
Direct upsampling v.s. Any resolution. For uniformity in our experiment, we standardize on a target resolution of 448 1 which is upscaled from 336 as the vision encoder's 's pre- training resolution for both scaling methods to ensure identical image tokens are input into the LLMs. In the case of "direct upsampling", positional embedding interpolation is applied, and the CLIP backbone is adjusted this new resolution during the fine-tuning phase. For ann resolution",
predefined set of resolutions to support up to six grids first select the optimal resolution by prioritizing fitting the original image's 's aspect ratio and size as closely as possible while minimizing wasted resolution, and resize the input image to the optimal resolution and split the image into these grids. All image patches are encoded by the CLIP encoder separately, and their features are input into
Given image,
'The numbers of tokens are dynamic given different input image resolutions, but the maximum number tokens 1280. We chose 448 with computational overhead mind.
We grid configurations of [1x1, 1x2, 1x3, 1x4, 1x5, 1x6, 2x2, 2x3, and their transpose)

---

Published as a conference paper at COLM 2024
Ferret-v2 Model
It's a magnifier [310, 15, 350, 320) held by a ferret (150 71, 512, 953).
Large Language Model
000
000
000
Fietten
Hi
H1
Spatial-Awar Visual Sampler
Interpalato
Merge
R
Projector 1
Projector 2
CLIP Encoder
DINOv2 Encoder
Embedding
Referred
Can you describe the region [300, 20, 360, 300] <continuous>
Resize
Low-resolution Global Image
Text w/ references
Find Grid Conig & Resize & Spii
Local Patches
Figure 3: Overview of the proposed Ferret-v2 model architecture.
LLMs as image tokens. We trained the models using both frozen and unfrozen encoder configurations.
As highlighted in Fig. 2, our comparative analysis revealed that the "any resolution" scaling method not only demonstrated significant improvements across all tasks over the vanilla Fer- ret but also outshined the "direct upsampling" approach. Another interesting observation is that in "any resolution", updating the vision encoder always brings a boost over freezing it, whereas in "direct upsampling" freezing the vision encoder is sometimes even better (as shown in the TextVQA result). As for the reason behind those findings, we hypothesize that "direct upsampling" forces the VIT to adapt to a higher resolution, which brings much longer token lengths deviated from its pre-training data. However, the scale of fine-tuning data is usually much smaller than the pre-training data of the vision encoder (1.3M vs. 400M in our setting), which disturbs its pre-training knowledge. On the contrary, "any resolution" " crops the high-resolution image into patches, and the vision encoder processes local patches in similar token length to its pre-training procedure. Overall, "any resolution" has proved to be more optimal strategy that balances leveraging high-resolution images and preserving valuable pre-training knowledge for effective scaling.
3.3 Model Architecture
Multi-Granularity Visual Encoding. With the adoption of the "any resolution" scaling method, yet another problem arises naturally: there is granularity difference between global low-resolution image Ig and local split image patches [In, Ip,,.,, (IN), i.e,, the global image lg sees the entire scene but in coarse resolution, while each local patch I5 can see only part of the scene but in precise detail.
To deal with this issue, we explore encoding those two types of images with distinct visual encoders. Specifically, we choose the CLIP (Radford et al., 2021) to encode global images and DINOv2 (Oquab et al., 2023) to encode local split patches. Our motivation behind this comes from the difference in their pre-training paradigms. The image-text contrastive objective used in CLIP, enables these models to capture image-level semantics from captions but tends to neglect the rich pixel-level details due to the limited fine-grained information in the guided captions. DINOv2, trained with self-supervision objectives of both image-level and plteh- can capture more detailed information about local objects such as shape or texture and therefore possess fine-grained perception abilities. Furthermore, we employ separate MLP projectors for the dual vision encoders, aiming to differentiate and learn the
5

---

Published as a conference paper at COLM 2024
Stage Coppge-Cptiin..oom Aligmment
Stage IE: Hiihhresotutionn Dezse Alignment
Stage III: Incett-rhunned Instruction Tuning
Large Language Model
Large Language Model
Large Language Model
?
Project)
Proteter.
Projector.
Viluaal Samplerr
Projector_
Projector
Aaaue Vicual Gaungler
Ensoder_3
Breader
Dreedor
CUPP Enceder
Disav Encoder
Embbadding
Figure 4: Model Training Paradigm. The model is trained from a "coarse-to-fine" manner. 'snowflake' denotes that the module is frozen.
diverse underlying contexts for global and fine-grained visual information: In 6 {Iu, 42,, |IN)
Fxx = CLIP()); Hg = M
Fn = (4)) Hy (FI).
(1)
(2)
Then, the feature maps of local patches are merged into large feature map according to its original arrangement and then flattened into a sequence of image features. The global image's feature map is also flattened. Two sequences are connected and input into LLM as visual "tokens".
Any resolution Referring. The hybrid region representation introduced in Ferret has proved effective and versatile in handling various types of referring such point, box, scribble, etc. What lies at the core of is the extraction of continuous region features, which is performed by a Spatial-Aware Visual Sampler. However, directly feeding global image features into the visual sampler may not be sufficient to recognize the small referred objects in high-resolution images. Inspired by our previous findings about the visual granularity difference, we further propose to integrate the best of both global semantics and local details for more precise referring. To be more specific, after obtaining the encoded features of global image Hg and local patches local patches into large feature map following their original spatial arrangement, and the global image feature map is upsampled via interpolation to align the size of the merged feature map.
H we first merge the feature maps of
H Concat Hn, Hiz. HIN} H'g UpsumplelH.p. (Hg)
(Hu e R HjJ R RRyymmiix .1 N)
(3)
(Hg E Rog Hyg E E (0xxxx
(4)
Then, we fuse the two processed feature maps by adding them channel-wise: H₂= H3 and obtain high-resolution feature map with strong semantics and local awareness. The H, is input into spatial-aware visual sampler (You et al., 2023) to extract continuous region features. Then the continuous feature is combined with discrete coordinates as a hybrid region representation to refer to any region in the image, as shown in Fig. 3.
Any resolution Grounding. By combining visual embeddings from both global image and local sub-patches, our model can more effectively uncover visual details from high resolution and bridge the semantics. Without specific adaptation, our framework aligns seamlessly with the grounding design in Ferret; therefore, similarly, we delineate the output coordinate regions through an intuitive numerical representation and employ the LLM as the principal mechanism for deciphering the intrinsic correlations.
3.4 Training Paradigm
Stage I: Image-Caption Alignment. Feature alignment before fine-tuning has been widely utilized to achieve better training efficiency. We adopt this strategy to connect the pre-trained CLIP encoder with the LLM using 1.4M image-text pairs, converted to instruction-followin data by (Chen et al., 2023c). The low-resolution image encoder and LLM parameters remain frozen, with only the projector trainable. Without any referring in these image-text pairs, the visual sampler doesn't participate in the training of stage I.
6

---

Published as a conference paper at COLM 2024
Stage II: High-resolution Dense Alignment. Although the previous image-caption align- ment pre-training is effective in bridging vision and LLM in coarse semantics, there still exists severe gap between the image-caption alignment and instruction tuning stage. Many downstream tasks, such as referring, grounding. OCR, etc, require a more precise and comprehensive spatial perception of the image, beyond solely coarse semantics.
To alleviate the above-mentioned issue, we propose a novel pre-training stage aimed at high-resolution dense alignment. Specifically,, instead of aligning the entire image with a global caption, this stage aligns every possible local object of the image with detailed semantics. Correspondingly, two types of tasks and input data are designed.
(1) Dense Referring: given the image, the input question refers to regions of all objects one by one, and asks about their categories, the model required to output the predicted classes accordingly. An example is "Question: Please classify the objects in the following locations. 1: (region_l), 2: (region 2). .... Answer: Here are the categories: 1: cat, 2: dog, ... ". (2) Dense Detection: Given the image, the input question asks to localize all the objects. To reduce randomness and incorporate spatial awareness, we forge the answer to list objects in a certain order, such as raster scan order (from top to bottom, from left to right). An example is "Question: Please localize visible objects in the image in a raster scan order. Answer: The objects are: 1: cat (coordinate 1), 2: dog (coordinate 2), ... To ensure the efficient learning of the fine-grained semantics, we collect data from densely annotated object dataset- LVIS (Gupta et al., 2019). On average, each sample includes around 10 object locations, whereas in the instruction tuning stage, referring and grounding datasets mostly have only one or two object locations mentioned per sample.
In terms of the model, we take pre-trained DINOv2 as the visual encoder for local patches, in addition the CLIP encoder for global images, as mentioned in Sec. 3.3. The projector after CLIP is inherited from the image-caption alignment stage, and we further add separate projector after DINOv2, whose weights are initialized from the CLIP's projector for stability. Then we freeze two vision encoders and LLMs, and only update the two projectors as well as the visual sampler in this alignment stage, with the next-token-prediction loss.
Stage III: Intent-Enhanced Instruction Tuning. After the second stage of pre-training, the model acquires the capability for comprehensive global understanding of images, alongside the ability to identify and narrate objects of interest using free-form texts and visually referred regions obtained flexibly. Our aim is to enhance the model's adherence to user instructions while maintaining its high-resolution visual perception abilities. To achieve this, we render the encoders, projectors, region samplers, and the LLM itself trainable. For training, we utilize the GRIT dataset (You et al., 2023) and incorporate additional task-specific datasets for VQA (Antol et al., 2015) and OCR (Singh et al., 2019; Sidorov et al., 2020) from LLaVA 1.5 (Liu et al., 2023b). Furthermore, we identified two additional strategies that contribute to enhanced performance: (i) Data Unification: To facilitate the model's ' seamless transition from a global understanding based on plain texts to a regional comprehension utilizing hybrid representations, we employ an open-vocabulary object detector, GLIPv2 (Zhang et al., 2022a), to localize groundable nouns in the text on VQA datasets, and a public OCR model (Kuang et al., 2021) to get text bounding boxes on OCR datasets. (ii) Task Generalization: In order to diminish ambiguity across tasks that necessitate referring and grounding capabilities and those that do not, we adopt a method similar to LLaVA 1.5, which involves appending the prompt, "Incluee the coordinates for each mentioned object." " to further clarify task requirements.
4 Experiments
4.1 Referring and Grounding Tasks
Referring. Ferret-v2's enhanced understanding of referential queries is evident in its ability to interpret the semantics of specified regions within an image accurately. This is particularly assessed through the task of Referring Object Classification (ROC), where the model is tasked with identifying the object in a region mentioned in a query. Initially, like Ferret, we utilize the validation split of the LVIS dataset, covering more than 1,000 object
7

---

Published as a conference paper at COLM 2024
Models
Point
IVIS (%)
Box
Free-form
Point
SAvrefer (1)
Box
Free-form
Models
Reteering
Ferret Bendh
Grounding in
Arg.
Random Caess
50
50
50
Reasoning
Kosmoe-2
x
60.25
x
x
53.97
X
LLaVA
11,
31.
25.8
51,0
Shikra-78
87.82
67.71
x
54.15
56.82
X
Kosmce-2
51A
33.7
48.4
44.5
GPTSSRLI
Shikra-7B
460
416
50.
45.9
CogVLM-17B
Co-VEMM-171
67.1
51.7
SPUIINX-2k
72.33
$2.97
61.21
63.38
Osprey-78
72.2
678
Femret-7
67.91
79.42
69.77
61.91
62.99
57.74
SPHINX-2K
25.6
702
66.4
64.11
92-76 (Ours)
74.35
56,59
76.33
68.38
68.83
62,07
Terret-7B
68.7
673
575
61.5
Femre-138
68.35
80.46
70.98
63.16
63.35
58.02
Perret-2277 (Ours)
75.6
F-150--2--158 (Duns)
75.09
87.74
76.35
67.38
69.49
62.58
Terret-136 138
70,6
68.7
59.
66.3
Ferret (Ouns)
79.6
65.7
74.9
Table 1: Results of ROC on three different refer- ring types, including point, box, and free-form shape. 'x' means no such capability.
Table 2: Results on the proposed Ferret-Bench via alTd-as-a-Judge evaluation.
RefCOCO
ReCCOCO:
ReKOOCOg
Flickr00k Entities
Models
val
lestA
lestB
val
lestA
testB
val
lest
val
MAtNet et 2018)
76.40
8043
69
64.93
70.26
56.00
67.00
OFA-L 2022)
79.96
83.67
76.39
68. 68.29
76.00
61.75
67.57
67.58
UNITER (Chen et 2020)
81.41
87 04
74.17
75.90
81.45
66.70
74.02
68.67
VILLA (Gan 2020)
62.39
87
76.17
81.54
76.71
UnITAR (Yang al., 2022)
86.32
88.84
80.61
78.70
83.22
69.45
79.96
79.97
78.76
79.
MDETR (Kamath al, 2021)
86.75
89.58
81 41
79.52
84.09
70.62
81.64
80.89
82.3
83.8
G-DINO-L
2023c)
90.56*
82.75*
87 012
Shikra-7B (Chen al, 20236)
87.01
90.61
80.24
72.12
82.27
82.19
75,84
76.54
MinsGPT-<2-7BB (Chen el. 2023a)
88.06
91.29
84.30
79.58
85.
73.32
84.19
84.31
Owen
2023)
88.55
92 27
84.51
82.82
88. 59
76.79
85.96
86.32
SPHINX-22 (Lin et 2023)
91.10
85.51
90.62
80:45
L.aVA-G (Zhang al. 2023a)
89.16
68
84.82
83.03
83:62
VistaLLL (Pramanick al, 2023)
88.1
91.5
83.0
82.9
89.8
74.8
63.6
84.4
Femee-78 (You 2023)
91.35
80.78
67.38
73.14
83.M3
84.7%
80.39
82.21
Ferret-v--7B (Ours)
92.79
94.68
88.69
87.35
92.75
79.3
89.42
89.27
85.52
85.83
Shikra-138 (Chen et al., 2023b)
87,83
91.11
81.81
82.89
87.77
74.41
82.64
83.18
77.41
78.44
Griffen (Zhan et 2024)
89.6
91.8
86.5
81,9
85.5
76.2
85.9
86.0
64.8
MesVLL-Grouddin---
(Wang al. 2023a)
92.76
94.75
90.79
Ferret 13B (You et al., 2023)
89 .4.48
92.41
84.36
82. 81
88. 14
75.17
85.83
86.34
81.13
84.76
Ferret-V2-13B (Ours)
92.64
94.95
88.86
87.39
92.05
81.36
89.43
89.99
85,33
86.25
Table 3: Performance comparison (Acc@0.5) on the REC (RefCOCO, RefCOCO+, RefCOCOg) and phrase grounding (Flickr30k Entities) tasks. indicates that the method is specifically fine-tuned in the second stage.
categories with a majority being "in-domain" images. To further demonstrate Ferret-v2's improved ability to reference smaller objects, we compile an "in-the-wild" evaluation set using partial images from SA-1B (Kirillov et al., 2023) and corresponding human annotations of objects from AS-human (Wang al., 2023b), which contains high-resolution images, open-vocabulary objects and precise masks. In total, we manually verified 700+ high-quality samples with in-the-wild objects and called SA-refer. As shown in Table 1, Ferret-v2 significantly outperforms previous models on LVIS and sets up new benchmark not fully realized in prior Ferret, primarily contributing to high-resolution scaling. SPHINX also uses high-resolutton input images; however, more challenging tasks for SA-1B, Ferret-v2 still outperforms it, indicating the benefits of our special design for any resolution referring.
Grounding. Visual grounding aims to ground language queries into aligned image re- gions. We experiment on the sub-tasks of referring expression comprehension (RFC) with three renowned benchmarks: RefCOCO (Lin et al., 2014), RefCOCÓ+ (Yu et al., 2016), and RefCOCOg (Mao et al., 2016), and phrase grounding with Flickr30k Entities dataset (Plum- mer et al., 2015). As evidenced in Table 3, Ferret-v2 enables the use of high-resolution input images, leading to significant improvements over Ferret (You et al., 2023). Besides, Ferret-v2 outperforms most state-of-the-ale models, including specialist model G-DINO-L (Liu et al., 2023c) and other generalist models, which adopt even larger input image sizes. Our 7B model can achieve comparable results to Cog CogVL.M-Groundin utilizes a 4B vision model and a 6B connection module. These results demonstrate the competitive capability of Ferret-v2 for visual grounding.
(Wang et al., 2023a), which
Ferret-Bench. Ferret-Bench (You et al., 2023) is carefully designed to evaluate and bench- mark the fine-grained capability of multimodal conversational models, particularly in their
8

---

Published as a conference paper at COLM 2024
Method
VQA GQA
VQA"
POPE
MME
SEED
LLaVA LLaVA
MM-Vet Obj-Hal
BLIP-2-138
41.0
41
42.5
85.3
1293.
46.4
38.1
22.4
InstrucIBLIP-7B
49.2
50.1
53.4
60.9
26.2
IDEFICS-9B
38.4
25.9
Owen-VL-7B
78.8"
59.3"
63.8
56.3
Owen-VL-Chat-D
78.2"
37.5*
61.5
1487.5
58.2
43.8/23.0
11VVA-1.5-BB
78.5"
62.0"
58.2
85.9
1510.7
58.6
82.7
63.4
30.5
46.3/22.6
Ferret-v2-7B (Ours)
81.3"
64.7"
61.7
87.8
1510.3
38.7
89.1
67.7
34.9
23,8/14.77
49.5
50.7
78.9
1212.
58.2
25.6
Shikra-13B
77.4"
IDEFICS-80B
60.0
45.2
30.9
1.5VA-1.5-138
80.0"
63.3*
61.3
85.9
1531.3
61.6
83.4
70.7
35.4
LLaVA-1.5-18BHHD
81.8"
64.7"
62.5
86.3
1500.1
62.6
72.0
39.4
Fermet-v2-13B (Ouns)
81.8"
64.8
62.2
88.1
1521.4
61.7
90.7
69.9
35.7
34.7/16.8
Table 4: Comparison with SoTA methods 10 benchmarks. Ferret-v2 achieves comparable performance with others. *The training images of the datasets are observed during training.
Model
Referring IVIS
Grounding RFC
OCR TextVQA
Reasoning Ferret Bench
Resolution
Referring LVIS
Grounding REC
OCR TextVQA
Reasoning Ferret-Bench
SA
CLIP
73.0 67.8
88.5
60.7
72.6
Fixed Res.
68.4 61.9
86.8
542
71.1
DINOv2
AnyRes, Ground
72.2 67.7
88.3
602
722
Stage II
74.6 68.4
89.3
61.7
75.6
Ans. Refer
73.0 67.8
88.5
607
72.6
Table 6: Ablation study on the effectiveness Table 5: Ablation study on any resolution grounding of the multi-granularity visual encoding and and referring.
Stage II Pre-training.
ability to refer to, describe, and reason about specific regions within images, thereby fa- cilitating a more structured evaluation of models' referring and grounding capabilities in multimodal context. We use Ferret-Bench to compare Ferret with previous models, including LLaVA (Liu et al., 2023b), Shikra (Chen et al., 2023b), Kosmos-2 (Peng et al., 2023), and Osprey (Yuan al., 2023). Results are summarized in Table 2. Ferret-v2 demonstrates superior performance in all types of tasks, indicating the strong spatial understanding and commonsense reasoning capability of the model.
4.2 Modern MILM Benchmarks
In pioneering the fields of referring and grounding, Ferret has demonstrated remarkable region reasoning capabilities, as illustrated above. However, it falls short of academic benchmarks that typically demand tasks-oriented datasets. For Ferret-v2, we specifically include pseudo-labeled VQA and OCR datasets and also append the special prompt, as mentioned in Sec. 3.4. This strategic enhancement progressively narrows the gap between task-specific region-level analyses and broader, more generalized tasks, thereby extend- ing Ferret-v2's applicability to encompass both fine-grained and coarse-grained tasks. As presented in Table 4, we benchmark Ferret-v2 against existing MMLMs across compre- hensive suite of 10 benchmarks: 2 et al., 2015), TextVQA(aka.VO)) et al., 2019), GQA (Hudson & Manning, 2019), POPE (Li et al., 2023e), MME" (Chang al., 2023), SEED (Li et al., 2023b), LLaVA and LLaVAW (Liu et al., 2023b), MM-Vet (Yu et al., 2023b), Obj-Hal (Yu et al., 2023a)). Our models achieve on-par performance with the latest state-of-the-art models, particularly excelling in tasks such as VQAv2, GQA, POPE, etc., which demand precise spatial information for accurate responses.
(Singh
Ablation Studies
In all the ablation studies below, we follow Sec. 3.2 and primarily focus our evaluation on the disparate models' performance across the dimensions of referring, grounding, OCR, and reasoning. Additionally, We also explore how Ferret-v2 balances accuracy with efficiency.
Any Resolution Grounding and Referring. We conduct an ablation study on any reso- lution grounding and referring. As illustrated in Table 5, accommodating any resolution

---

Published as a conference paper at COLM 2024
markedly enhances task performance that necessitates comprehensive understanding of higher-resolution details. Integrating the best of both global semantics and local details for more precise and improved precision in referring tasks across both LVIS and SA datasets. Furthermore, this integration also modestly enhances grounding capabilities, suggesting that grounding and referring derive mutual benefits within our proposed framework.
Multi-Granularity Visual Encoding and Stage-Il Pretrain. Our initial ablation study focuses on incorporating an additional DINOv2 encoder for the encoding of high-resolution patches. We utilize the projector weights from Stage I of CLIP for initialization, followed by fine-tuning in Stage ITI. As demonstrated Table 6, the exclusive employment of visual granularity encoding significantly enhances both referring and grounding performance. Furthermore, introducing an intermediary Stage II in the pre-training process yields im- provements across all evaluated metrics.
Efficiency and Scalability. We add detailed analysis of Ferret-v2's performance Referring (LVTS) and Grounding (REC) and a comparison of the computational cost (in terms of inference time and average token per sec.) with the previous Ferret baseline.
In Table 7, we find that accommodating any resolution markedly enhances task performance, particularly for tasks that require a comprehensive understanding of higher-resolution de- tails. Additionally, incorporating an extra DINOv2 encoder significantly improves referring and grounding performance. While this design leads increased computational demands, the performance gains achieved by the model justify the additional computational costs.
Avg. Inference Time (sec/10)
Avg. Tokens (per sec)
Design
Base (Ferrel)
Vicuna 1.3
Referring (LVIS)
67.9
1.4
6.1
Grounding (RFC)
+ Longer Context
Vicuna 1,5
Referring
68.4
1.1
4.8
Grounding
86.8
AnyRes. Grounding
Vicuna 1.5
Referring
72.2
1.7
3.1
Grounding
88.3
0.48
3.2
AnyRes. Referring
Vicuna
Referring
Grounding
88.5
0.5
3.1
(Ferretv2)
Vicuna
Referring
Grounding
89.1
0.54
2.9
Table 7: Performance comparison of different models and settings. All model inference are based on 7B scale and performed with 8 Nvidia A100 GPUs using greedy decoding.
Conclusions
We present Ferret-v2, significant upgrade of the vanilla Ferret model. It features advanced capabilities in handling any resolution referring and grounding, multi-granularity visual encoding, and novel three-stage training pipeline. These improvements enable Ferret-v2 to excel in processing and understanding images with higher resolution and finer detail.
Limitations
Ferret-v2 mitigates harmful outputs by using transparent datasets, enhancing spatial knowl- edge to reduce object hallucinations, and improving instruction-folloiing for unclear queries. However, its referring and grounding capabilities pose real-world challenges, par- ticularly in interactive scenarios, where incorrect contextual understanding and ambiguous references could lead to inaccurate location outputs.
Acknowledgment
The authors would like to thank Yizhe Zhang, Yanghao Li, Liangchen Song, and Keen You for valuable guidance, suggestions, and feedback. Additional thanks go to Jiaming Hu, Mingfei Gao for supporting large-scale training.
10

---

Published as a conference paper at COLM 2024
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. NeurlPS, 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In CVPR, pp. 1425-2443 2015.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. MtXiv:2308.12966 2023.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 2023.
Jun Chen, Deyao Zhu, Xiaogian Shen, Xiang Li, Zechu Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt- v2: large language model as unified interface for vision-language multi-task learning. arXiv:2310.09478, 2023a.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv:2306.15195, 2023b.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.1221 122993, 2023c.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, pp- 104-120. Springer, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale ad- versarial training for dasion-and-language representation learning. In NeurIPS, volume 33, Pp. 6616-6628, 2020.
Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for family of multi-modal large language models. arXiv preprint arXiv:2402.05935, 2024.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary in- stance segmentation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5356 5364, 2019.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et Cogagent: A visual language model for gui agents. arXiv preprint arXiv:2312.08914 2023.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pp. 6700-6709, 2019.
Dongsheng Jiang, Yuchen Liu, Songlin Liu, Xiaopeng Zhang, Jin Li, Hongkai Xiong, and Qi Tian. From clip to dino: Visual encoders shout in multi-modal large language models. arXiv preprint arXiv:2310.08825, 2023.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mddtr-modulated detection for end-to-end multi-modal understanding. In ICCV, pp- 1780-1790, 2021.
11

---

Published as a conference paper at COLM 2024
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Refer- ring to objects in photographs natural scenes. In EMNLP, 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. arXiv:330.026443, 2023.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. In ICML, 2023.
Zhanghui Kuang, Hongbin Sun, Zhizhong Li, Xiaoyu Yue, Tsui Hin Lin, Jianyong Chen, Huaqiang Wei, Yiqin Zhu, Tong Gao, Wenwei Zhang, Kai Chen, Wayne Zhang, and Dahua Lin. Mmocr: A comprehensive toolbox for text detection, recognition and understanding, arXiv preprint Sati:2208.0054333 2021.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.00699 2023.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. MIXiv:230.003726, 2023a.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal ilms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023b.
Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and recognize anything at any granularity. arXiv:2307.04767, 2023c.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2:
Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv:2301.12597, 2023d.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023e.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiy preprint arXiv:2311.07575, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://llawa-v1.e gi thub. thub. httpii/////////// 2000001222---122000 100-4-01------aaa-eee-ee llxva-next.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint atXiv:2303.05499 2023c.
Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang, Shoufa Chen, Qinglong Zhang, Yang Yang, Qingyun Li, Jiashuo Yu, et al. Internchat: Solving vision-centric tasks by interacting with chatbots beyond language. arXiv:2305.05662. 2023d.
12

---

Published as a conference paper at COLM 2024
Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, et al. Kosmos-2.5: A multimodal literate model. arXiv preprint arXiv:2309.11419, 2023.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016.
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis insights from multimodal Ilm pre-training. arXiv preprint 1rXiv:2403.0%611, 2024.
OpenAL. GPT-4 technical report. arXiv:2303.08774,22 2023.
TB OpenAL. Chatgpt: Optimizing language models for dialogue. openai, 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv:2306.14224, 2023.
Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, Lingpeng Kong, and Tong Zhang. Detgpt: Detect what you need via reasoning. arXiv:2305.141c 2023.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In CVPR, pp. 2641-2649, 2015.
Shraman Pramanick, Guangxing Han, Rui Hou, Sayan Nag, Ser-Nam Lim, Nicolas Bal- las, Qifan Wang, Rama Chellappa, and Amjad Almahairi. Jack of all tasks, master of many: Designing general-purpose coarse-to-fine vision-languaag model. arXiv preprint arXiv:2312.12423, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning trans- ferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 9748.8763. PMLR, 2021.
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: dataset for image captioning with reading comprehension. In Computer Vision-BCCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pp. 742 758. Springer, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. 2019.
CVPR, pp. 8317-8326,
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.1538) 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 2023b.
13

---

Published as a conference paper at COLM 2024
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through simple tt-eeuueee- learning framework. In International Conference on Machine Learning, pp. 23318 23318-23344 23340. PMLR, 2022.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.0300 2023a.
Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhenhang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all-seeing project: Towards panoptic visual recognition and understanding of the open world. arXiv preprint arXiv:2308.01907, 2023b.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv:2305.11775 2023c.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In ICLR, 2021.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv:2303.04671, 2023a.
Tsung-Han Wu, Giscard Biamby, David Chan, Lisa Dunlap, Ritwik Gupta, Xudong Wang, Joseph E. Gonzalez, and Trevor Darrell. See, say, and segment: Teaching Imms to overcome false premises. arXiv:2312.08366 2023b.
Xiaoyang Wu, Zhuotao Tian, Xin Wen, Bohao Peng, Xihui Liu, Kaicheng Yu, and Heng- shuang Zhao. Towards large-scale 3d representation learning with multi-dataset point prompt training. arXiv:2308.09715 2023c.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision- language modeling, In ECCV, pp. 521-539. Springer, 2022.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv:2303.11381) 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv:230..14178, 2023.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv:2310.07704 2023.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L Berg. Modeling context in referring expressions. In ECCV. Springer, 2016.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In CVPR, pp. 1307-1315, 2018.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. arXiv preprint arXiv:2312.00849, 2023a.
14

---

Published as a conference paper at COLM 2024
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, arXiv preprint Stt.v:2330.00299990 2023b.
Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. arXiv preprint arXiv:2312.10032, 2023.
Yufei Zhan, Yousong Zhu, Hongyin Zhao, Fan Yang, Ming Tang, and Jinqiao Wang. Griffon v2: Advancing multimodal perception with high-resolution scalinn and visual-language co-referring. arXiv preprint arXiv:2403.09333 2024.
Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Lei Zhang, Chunyuan Li, et al. Llava-grounding: Grounded visual chat with large multimodal models. arXiv preprint arXiv:2312.00249,, 2023a.
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying local- ization and vision-language understanding. Advances in Neural Information Processing Systems, 35:36067-360800 2022a.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv:2307.0366 01..2000,,0061,,. 2023b.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01663 2022b.
Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. Bubogpt: Enabling visual grounding in multi-modal llms. arXiv preprint arXiv:2307.08581, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph F. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt- 4: Enhancing vision-language understanding with advanced large language models. arXiv:2304.10592, 2023.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. arXiv:2304.067113 2023.
15


Reviews:
[{"review_id": "iG9OiXDb6W", "reviewer": "Reviewer_NGY1", "summary": "The authors propose Ferret-v2, an improved baseline of  Ferret. Ferret-v2 has three key designs: any resolution processing, multi-granularity visual encoding, and a three-stage training paradigm. The referring and grounding abilities of Ferret-v2 are strengthened by these designs. Finally, experiments demonstrated the effectiveness of the proposed Ferret-v2 on referring, grounding tasks, and modern MLLM benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback on our paper. We are pleased that you found Ferret-v2 novel and effective and that you appreciated the clarity of our presentation. We address your concerns below.\n\n**1. Efficiency and scalability**\n\nWe appreciate your concern regarding the efficiency and scalability of Ferret-v2. Here, we added a detailed analysis of Ferret-v2's performance on Referring (LVIS) and Grounding (REC) and a comparison of the computational cost (in terms of inference time and average token per sec) with the previous Ferret baseline. Additionally, we discuss the architectural choices that contribute to this balance. (*All model inference is based on a 7B scale and performed with 8 Nvidia A100 GPUs using greedy decoding.)\n\n| Design | LLM | Task | Performance | Average Inference Time (sec/it) | Average Token (per sec) |\n|-------------------------------|-------------|------------|-------------|---------------------------------|-------------------------|\n| Base (Ferret) | Vicuna 1.3 | Referring (LVIS) | 67.9 | 1.4 | 6.1 |\n| | | Grounding (REC) | 83.9 | 0.4 | 5.3 |\n| + longer context length (4k) | Vicuna 1.5 | Referring | 68.4 | 1.1 | 4.8 |\n| | | Grounding | 86.8 | 0.3 | 5 |\n| + AnyRes. Grounding | Vicuna 1.5 | Referring | 72.2 | 1.7 | 3.1 |\n| | | Grounding | 88.3 | 0.48 | 3.2 |\n| + AnyRes. Referring | Vicuna 1.5 | Referring | 73 | 1.8 | 3 |\n| | | Grounding | 88.5 | 0.5 | 3.1 |\n| + DINOv2 (Ferretv2 w/o Stage II) | Vicuna 1.5 | Referring | 73.8 | 2 | 2.8 |\n| | | Grounding | 89.1 | 0.54 | 2.9 |\n\nWe find that \"any resolution\" and adding an extra DINOv2 encoder significantly improves performance. While this design leads to increased computational demands, the model's performance gains justify the additional computational costs. We will include this discussion in the revised paper.\n\n**2. Potential Harmful Outputs and Ethical Considerations**\n\nWe acknowledge the importance of addressing potential harmful outputs and ethical considerations. Ferret-v2 mitigates potential harmful outputs through several mechanisms: it utilizes public, academic datasets for transparency and bias reduction; it integrates fine-grained spatial knowledge to accurately ground descriptions and minimize object hallucinations; and it incorporates instruction-following capabilities from some datasets (e.g., ShareGPT-40k) dataset to handle unanswerable or unclear queries appropriately. We will include a detailed discussion on these points in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors propose an improved Ferret-v2 with three specialized designs to strengthen the referring and grounding capabilities. Ferret-v2 seems novel to me.\n2. The proposed Ferret-v2 is effective and superior on multiple benchmarks, including referring tasks, grounding tasks, and modern MLLM benchmarks.\n3. This paper is easy to follow.", "reasons_to_reject": "1. Efficiency and scalability. How does Ferret-v2 balance accuracy with computational efficiency? The time complexity of Ferret-v2 is suggested to be clarified.\n2. Potential harmful outputs and ethical considerations. The authors are suggested to discuss how Ferret-v2 prevents the potentially harmful outputs.", "questions_to_authors": "please see the ''reasons to reject''", "extra_scores": {}}, {"review_id": "myNN8VovtH", "reviewer": "Reviewer_889e", "summary": "This paper is an improved version of Ferret. The improvement includes high image resolution, multi-granularity visual encoding and three-stage training pipeline. These techniques bring a lot benefit to Ferret and result to leading performance in various multimodal benchmarks,  especially on region-level tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback on our paper. We appreciate your recognition of the effectiveness and performance of Ferret-v2. Below, we address your concerns and comments:\n\n**Incremental Technical Originality**\n\nWhile it is true that individual components have been touched separately in previous works, the novelty of Ferret-v2 lies in the unique design and combination of these techniques to enhance fine-grained multimodal performance comprehensively. Specifically:\n\n**a. Any resolution**: While previous works used “any resolution” strategies and “direct upsampling” for high-resolution images, our analysis across four tasks (ROC, REC, TextVQA, Ferret-Bench) shows “any resolution” optimally balances high-res image use with pre-training knowledge preservation, crucial for scaling.\nWe designed Ferret-v2’s novel “referring” and “grounding” mechanisms. For referring, we integrate global semantics and local feature maps via a spatial-aware visual sampler. For grounding, our framework aligns with the LLM to delineate output regions. \n\n**b. Multi-granularity visual encoding**: We appreciate the reference to [3]. Unlike [3], which focuses on multi-level feature fusion of CLIP and DINOv2 features before feeding into LLMs, Ferret-v2 encodes global and sub-patch features separately with distinct encoders and then feeds them into LLMs simultaneously. This approach: 1) Treats global and local patches differently, respecting their unique characteristics. 2) Uses encoders' pre-training paradigms suited for global (CLIP) and local (DINOv2) patches. 3) Maintains separate tokens for global/local features, improving performance on referring and grounding tasks.\n\n**c. Multi-stage training**: We appreciate the reviewer’s acknowledgment of multi-stage pre-training trends. Our approach has distinct aspects compared to existing works, which often define training stages based on \"dataset scales\" without model changes. \nIn contrast, our training includes a unique Stage II, using a \"coarse-to-fine\" approach by initializing two projectors and a visual sampler, setting the foundation for a nuanced strategy. Stage III unfreezes CLIP and DINOv2, \"a bold move rarely seen in recent multimodal works \" that the reviewer acknowledges and appreciates. \n\nThank you again for your valuable feedback. We believe Ferret-v2 offers significant practical value and demonstrates satisfactory novelty, regardless of its extending series. We look forward to improving our paper based on your insights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper introduces a series of effective techniques to improve the multimodal model, \n- (1) high image resolution: sub-patches and direct upsampling are well studied.\n- (2) multi-granularity visual encoding: taking advantages of CLIP and DINOv2 on global image feature extraction and low-level details preservation, respectively.\n- (3) multi-stage training pipeline: I really appreciate that Stage III unfreezes CLIP and DINOv2, which is a very brave attempt and rarely seen in recent multimodal works.\n\nBy integrating these to Ferret, leading experimental results are produced in both image-level and region-level multimodal benchmarks.", "reasons_to_reject": "The main concern is that this is an incremental paper from the perspective of technical originality. For example, the sub patches-based high resolution has been used in [1][2], the integration of CLIP and DINO as the vision encoder are studied in [3], the multi-stage training strategies are widely used in previous multimodal works.\n\nThis is the first COLM, the acceptance threshold is not clear to me. If technical originality is a highly strict requirement of COLM, Ferret-v2 may not be novel enough to be accepted. However, if a well-constructed engineering paper is recommended, Ferret-v2 is truly a good example. \n\nMy initial score is a conservative score. I will determine my final score after reading other reviewers comments.\n\nReference:\n- [1] LLaVA-NeXT: Improved reasoning, OCR, and world knowledge\n- [2] Towards Open-Ended Visual Recognition with Large Language Model\n- [3] From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PcTRaIymmB", "reviewer": "Reviewer_aBcC", "summary": "The paper introduces Ferret-v2, an enhancement of the Ferret model, aimed at refining the capabilities of multimodal large language models (MLLMs) for high-resolution image understanding. The authors focus on improving the model's referring and grounding abilities, particularly in detailed regional and global reasoning. The introduction of a multi-granularity visual encoding and a three-stage training paradigm are novel and effective.  The results, compared against other state-of-the-art methods, demonstrate substantial improvements.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and constructive feedback on our paper. We appreciate your recognition of the clarity of our writing, the innovative techniques we introduced, and the comprehensive set of experiments provided. We address your concerns and questions below.\n\n**1. Ethical Considerations and potential risks for real-world deployment**\n\nPlease refer to Response #2 for Reviewer NGY1 with shared ethical considerations. Ferret-v2 mitigates harmful outputs by using public, academic datasets for transparency and bias reduction, integrating fine-grained spatial knowledge to minimize object hallucinations, and incorporating instruction-following capabilities to handle unclear queries.\n\nAdditionally, specific real-world deployment concerns arise from Ferret-v2's referring and grounding capabilities, especially in interactive scenarios where the model provides location instructions and interacts with users, including those with disabilities such as blindness or deafness. These concerns include:1) Contextual Understanding: Incorrect contextual interpretation can lead to erroneous location outputs. 2) Ambiguity in References: Users might provide ambiguous or unclear reference points/boxes/scribbles, which may result in inaccurate outputs. We will address these points in the revised paper.\n\n**2. Model training costs and Reproducibility**\n\nFerret-v2 utilizes a multi-stage training paradigm. For Stage-II, Ferret-v2 is trained on densely partial annotated LVIS data (200k) for 1 epoch, optimized by AdamW with a learning rate of 1e-4 and a batch size of 256. we freeze two vision encoders and LLMs, and only update the two projectors as well as the visual sampler in this alignment stage and the training takes \\~7 hrs on 8 40G A100 GPUs. \n\nFor Stage III, Ferret-v2 is trained on aforementioned GRIT data and task-specific VQA and OCR data (~1.6M) for 3 epochs. optimized by AdamW with a learning rate of 2e-5 and a batch size of 256. In this stage, we render the encoders, projectors, region samplers, and the LLM itself trainable. The training takes ~20/27 hrs on 32 40G A100 GPUs for a Ferret-v2 7B/13B.\n\nWe are committed to releasing the training/inference codes and model checkpoints for Ferret-v2. To comply with the double-blind policy, we created a **public anonymous repo** (https://anonymous.4open.science/r/ferret_v2_anno-85AF/). Upon our institution's approval, we will further organize the code and open-source the full training data and pre-trained model checkpoints.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The paper is well-written, and clear, providing extensive empirical evidence to support its claims.\n\n(2) The approach integrates innovative techniques like any resolution grounding, multi-granularity visual encoding, and a novel training paradigm, which are well justified and rigorously tested.\n\n(3) The paper provides a comprehensive set of experiments and comparisons to benchmark datasets, demonstrating the effectiveness and efficiency of Ferret-v2.", "reasons_to_reject": "(1) The paper briefly mentions potential risks of harmful and counterfactual responses from MLLMs but does not thoroughly explore limitations or potential biases of the proposed model.\n\n(2) The complexity of the model and the training process might pose challenges for reproducibility, which are not adequately addressed in the manuscript.", "questions_to_authors": "(1) Could you elaborate on the potential limitations and ethical considerations associated with Ferret-v2, especially concerning its deployment in real-world applications?\n\n(2) How much does the model training costs? Will the code and model publicly released?", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary of paper and contributions:
- The paper proposes Ferret-v2, a improved model over Ferret, for vision-language tasks that requires referencing and grounding of regions.  Ferret uses LLM to process image,  language, and region tokens in a unified way.  This work re-examines the Ferret architecture, and introduces three key improvements: 1) handling higher resolution images with the "any resolution" strategy where the images is divided into smaller patches for processing, 2) multi-granular visual encoder where different visual encoders are used for different granularities (CLIP for the global image, and DINOv2 for local patches) 3) three-stage "coarse-to-fine" training process.  The three strategies together leads to an overall method that can work on higher resolution upscaled images and is designed for finer-grained visual-language alignment.  Experiments are conducted on a variety of vision-language tasks (e.g. grounding references, VQA, etc), and shows that Ferret-2 outperforms prior work.  Analysis and ablations are also conducted to justify the various design choices.     

Overall Assessment:
- All reviewers are positive on this work.  Some reviewers had questions about whether the work was sufficiently novel (889e) and noted that there could be more discussion about the complexity and scalability of the method, which the authors addressed during rebuttal.  The AC believe the contributions are sufficiently novel, clearly explained and substantiated by the experiments, and agrees the work is of interest to the community and recommends acceptance.

Strengths of the work:
- the proposed designs to be novel and interesting (NGY1,aBcC)
- method is shown to be effective an work well for multiple tasks and benchmarks (NGY1,aBcC,889e)
- paper is well written and easy to follow (NGY1,aBcC)

Recommendations for improvements for the camera ready:
- Add discussion of efficiency and scalability (response to NGY1, aBcC)
- Add discussion of limitations, potential harms and ethical considerations (response to NGY1, aBcC)
- Full pass over paper to improve writing and awkward wording at places.
  - the "any resolution" method should be explained more in the introduction, with appropriate references to earlier work that adopted this 
  - For example: 'After devoting to the “any resolution” scaling method' => it is not clear what this means.  Perhaps "With the adoption of the "any resolution" scaling method".
Decision: Accept