Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want
Authors: Weifeng Lin
Xinyu Wei
Ruichuan An
Peng Gao
Bocheng Zou
Yulin Luo
Siyuan Huang
Shanghang Zhang
Hongsheng Li
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Large Language Model
Visual Prompting
Primary area: foundation or frontier models, including LLMs

Abstract: In this paper, we present the Draw-and-Understand framework, exploring how to integrate visual prompting understanding capabilities into Multimodal Large Language Models (MLLMs). Visual prompts allow users to interact through multi-modal instructions, enhancing the models' interactivity and fine-grained image comprehension. In this framework, we propose a general architecture adaptable to different pre-trained MLLMs, enabling it to recognize various types of visual prompts (such as points, bounding boxes, and free-form shapes) alongside language understanding. Additionally, we introduce MDVP-Instruct-Data, a multi-domain dataset featuring 1.2 million image-visual prompt-text triplets, including natural images, document images, scene text images, mobile/web screenshots, and remote sensing images. Building on this dataset, we introduce MDVP-Bench, a challenging benchmark designed to evaluate a model's ability to understand visual prompting instructions. The experimental results demonstrate that our framework can be easily and effectively applied to various MLLMs, such as SPHINX-X and LLaVA. After training with MDVP-Instruct-Data and image-level instruction datasets, our models exhibit impressive multimodal interaction capabilities and pixel-level understanding, while maintaining their image-level visual perception performance.

Published as a conference paper ICLR 2025
DRAW-AND-UNDERSTAND:: LEVERAGING VISUAL PROMPTS TO ENABLE MLLMS TO COMPREHEND WHAT YOU WANT
Weifeng Lin
Xinyu Wei
Ruichuan An²
Peng Gaot Bocheng Zou
Yulin Luo²
Siyuan Huang
Shanghang Zhang' Hongsheng Lif
CCUHK
Peeking University
University of Wisconsin-Madison
Shanghai Al Laboratory
light brown write foce.
SaM Instructions
Open-Source Data
Annotation
answers
<dag The maarket alight haad expoors baaking directly the
Large Language Model
GPT4V
Re- construct
Model
E Brif Castoo
Training Data
Image Encoder
Tokenizer
Oli kooee eng
Prompts
Input Image
Setting
Image
Prompts
Text
(a) Task: visual prompting
(b) Architecture: VP-MLLM
(c) Data: Training Data Generation
Framework. (a) Illustrating the task of visual prompt- ing understanding. (b) The architecture of Visual Prompting MLLM (VP-MLLM), which consists of an image encoder, visual prompt encoder, and an LLM. (c) The data generation process for training, which involves two components: reconstruction of open-source data and data generation assisted by GPT-4V.
Figure 1: Overview of the Draw-and-understee
ABSTRACT
In this paper, we present the Draw-and-Understan framework, exploring how to inte- grate visual prompting understanding capabilities into Multimodal Large Language Mod- els (MLLMs). Visual prompts allow users to interact through multi-modal instructions, enhancing the models' interactivity and fine-grained image comprehension. In this frame- work, we propose a general architecture adaptable to different pre-trained MLLMs, en- abling it to recognize various types of visual prompts (such as points, bounding boxes, and free-form shapes) alongside language understanding. Additionally, we introduce multi-domain dataset featuring 1.2 million image-visual prompt- text triplets, including natural images, document images, scene text images, mobile/web screenshots, and remole sensing images. Building on this dataset, we introduce MDVP- Bench, challenging benchmark designed to evaluate a model's 's ability to understand vi- sual prompting instructions. The experimental results demonstrate that our framework can be easily and effectively applied to various MLLMs, such as SPHINX-X and LLaVA. After training with MDVP-Instruct-Daar and image-level instruction datasets, our mod- els exhibit impressive multimodal interaction capabilities and pixel-level understanding,
'Equal Contribution
I Corresponning Authors

---

Published as a conference paper ICLR 2025
while maintaining their image-level visual perception perforance.. The code and related resources are available at http::///rraawann-uddderrrtttta
1 INTRODUCTION
Recent works [Liu al. 2024b Zhu et al. 2023 Liu al. 2023a Bai al. 2023 Liu et al. 2024a) have enhanced Large Language Models (LLMs) with visual perception, facilitating image-related communication and fostering a deeper understanding of the world. These models primarily focus on interpreting whole images by aligning them with text prompts. However, simple language interactions often fail to capture users' true intentions, especially when users need to highlight specific areas in images that are difficult to describe with words. Consequently, there is growing interest in enabling visual prompting capabilities in Multimodal Large Language Models (MLLMs) to enhance interactivity and pixel-level understanding.
To achieve this, ChatSpot (Zhao et al. 2023) and Shikra (Chen et al., 2023b) first utilize textual represen- tations to specify coordinates within images, thereby enhancing interaction with the models. Some stud- ies (Peng et al., 2023 Zhang et al.] 2023a Zhou et al. 2023) employ positional embeddings
improve
spatial recognition, while others (Rasheed et al. [023] Zhang et al.) 2023a You et al.) 2023 Yuan et al.,
2024a Chen et al. 2023a) focus on extracting Regions of Interest (ROI) to enhance attention specific
areas of images. Additionally, LLaVA-ViP (Cai et al. 2023) introduces visual markers to facilitate more
intuitive interactions between users and models. (More related works are discussed in Sec. B
However, existing methods have several limitations: (i) ROI-based methods (Zhang et al. 2023a Rasheed ct al.] 2023 Yuan al., 2024a] You et al., 2023) are typically designed for specific architectures. For example. they often rely on pre-attached segmentation models or externally provided ground truth masks, which hinders user flexibility and model scalability. Additionally, they require training from scratch, leading
to substantial resource consumption; (di) Most methods (Zhang et al. 2023a Rasheed et al. 2023 Zhao
jet al. 2023 Chen al. 2023b Peng et al., 2023) depend on fixed visual references. such as bounding boxes, which are neither flexible nor user-friendly; (fii) Several methods (Yuan et al. (0024)) fail to support the simultaneous referencing of multiple objects, limiting their flexibility and preventing them from addressing
more complex understandings. such as nuanced interrelations and spatial dynamics with surrounding entities
and backgrounds; (iv) Most methods (Zhang et al. 2023a; Rasheed et al. 2023 Zhao et al., 2023 Chen
et al. 2023b Peng et al. 2023 Yuan al., 2024a You al.] (203) primarily focus on visual prompting understanding but neglect image-level perception performance, which limits their practical applicability.
To address these challenges, we present the www-and-Unndestann framework, solution specifically de- signed to endow multi-modal LLMs with visual prompting understanding. As depicted in [a), we first define the form of the visual prompting understanding task, emphasizing the multi-modal interaction between users and the model. Next, we introduce visual prompting MLLM (VP-MLLM), a general archi- tecture that comprises vision encoder, a visual prompt encoder, and an LLM. This architecture can be ef- ficiently adapted to most mainstream pre-trained MLLMs, enabling them to easily acquire visual prompting capabilities while maintaining their original robust vision-language understanding. Notably, our proposed visual prompt encoder enhances the model's referring capability by embedding the coordinate features of visual prompts. It supports multiple input formats and can process several visual prompts simultaneously.
In addition, instruction tuning data for visual prompting is essential component. To this end, we curate a comprehensive Multi-Domain Visual Prompt Instruction Dataset MIDVV--Istruct-Datta This dataset com- prises mmage-point-test and lion query-answer triplcts. We compiled this dataset by integrating existing datasets containing segmentation masks bounding box annotations, while leveraging GPT-4V's advanced image understanding capabilities for data generation. MDVP-Instruct-Datt provides detailed attribute information for objects identified by visual prompts, including their relationships with surrounding entities and the background. This dataset
region-text pairs, totaling approximately 0.8 million images and 1.2 mil-
2

---

Published as a conference paper ICLR 2025
Providd: the uzr reealls Th Allagien E- markel.
1 Inter Relecionhip:: -aarkt while <Mark peovides closer look dhe
muplie on Lable
I-
and
OREGON
Doncrn vanr
What releniontip
keptop Tagethar
befweze <Mark 3n and <hlark
likely convey namaore person
CRLEROONE FOS
3-11 Can sne logeterr
eEp
work, semp technelogy
Both <Mark 2- and -Martk are photography
gir losking
sain BVS
plittry instumeen.
par bodding pot
equpmere, with <Mark I- being analing commn and
of gin
right
Detail Captien: Towards the
image,
<Daack 3- being lens. They can cacil begehhrr The lote <htark component of camera at chilak suguesiing that they
whate placed
ty
elightly towards
wiiting appears to
scripe. addinng tusch the
photograpte
ARE wwsetie Masi skin.
Figure 2: (Left) Our approach achieves unprecedented performance in OCR tasks and excels at under- standing implicit relationships and conducting complex reasoning. (Right) Our approach demonstrates the capability to perceive objects at pixel level granularity, such as 'a girl holding pen'. Furthermore, it perform detailed captioning and inter-relationship analysis on arbitrarily shaped Regions Of Interest (ROIs).
enhances spatial understanding across various image domains, thereby improving the robustness of model's s responses in diverse visual contexts. With these approaches place, users can interact with the model using their native language and refer to specific actions, such as clicking and drawing, to obtain desired answers about the region of interest. This allows the MLLMs to engage with the physical world more effectively.
To assess the strength and resilience of visual prompting models, introduce MDVP-
benchmark designed to evaluate visual prompting comprehension abilities. MDVP-Bench encompasses a variety of tasks, including point-level and region-level captioning, inter-relationship analysis, and complex reasoning. In evaluating our VP-MLLMs alongside other visual prompting methods on MDVP-Bench, we find that ours consistently outperforms the other methods. We anticipate that MDVP-Bende will provide solid foundation for future research in the field of visual prompting and multimodal models.
2 MDVP-INSTRUCT-DATS
In this section, introduce the Multi-Domain Visual Prompt Instruction Dataset Instruct-tt designed to enhance interactivity and fine-grained image understanding in MLLMs. It primarily consists of two types of data:
(1) Dynamic Multi-Target Target Captioning, In this context, users are assumed to provide N visual prompts to refer regions of interest, and the model generates comprehensive caption describing all N subjects. To create this type of data, we observed that datasets related to Referring Expression Comprehension (REC) and Phrase Grounding can be transformed align with this task. Specifically. these datasets include the ability to localize visual content and ground each entity mentioned by a noun phrase in the caption to a specific region the image. Building on this insight, we inverted the inputs (captions) and outputs (coordinates of subjects), treating the ground truth boxes as input visual prompts, with the corresponding natural language descriptions serving as the answers. The public datasets we utilized include Flickr30K (Plummer et al.) 2015), RefCOCO/+(Yu et al. 2016), GCG(Rasheed et al. (023), and GRIT (You et al.) 2023). as well as GeoChat (Kuckreja et al. 2023) from remote sense domain.
Data. For visual prompting conversations, primarily employ two meth- ods for data construction: (i) Reconstructing from Grounding QA datasets. Grounding QA datasets pro- vides ground truth bounding boxes for various target objects in both the questions and answers. We treat the provided bounding boxes as visual prompt inputs, discarding the original ground truth coordi- nates from the questions to create high-quality instruction data. We utilize grounding QA pairs from
(2) Costruct-Covvrsaatiin
3

---

Published as a conference paper ICLR 2025
HoUJogS
Nature Image
OCR
Rensee
Web Page
Mobile
Document
Multipanel
39%
12%
12%
11%
10%
Object Categorries
Detailed Caption
Dog
collar
Grass
This dog. showing glossy, reflective surface that indicates liveliness and health. The dog's eye is brown with dark pupil in the center Surroun ding the eye, the fur is predominantly black with brown markings that provide natural pattern typinal of this animal's facial features.
<Mark 1-
Brief Caption
~Mark 2>: Black bucidle collar dog's nede.
<Mark 3o: Circalar red frishee with central
on grass.
Inter Rnte--eeattioship Analyze
Conversation & Reasoning
<Mark 1s <Mark 3os The frishee at <Mark 1> is commonly used playing fench, game that involves the dog at <Mark 2r, signilying playful relationship betwvenn the objoct and the animal.
Question Does the environment at <Mark 4- seem suitable for activity relited <Mark 3-? Answer: environment
4x, which
maintained grass, as indicated by <Mark 3x, due to its open space
<Mark 1s <Mark 2= The collar is part of the dog's attire, signitying ownership possibly containing
the breed dog depicted <Mark 1-7 Answer: The dog at <Mark 1> appears to be a corgi based on its distinct physical characteristies such its pointed
Question
identification in case the dog wanders off. The collar is a pet accessory and closely associated with the dog's identification and control.
ears, short stature, and mullticolored coat.
Figure An illustrative example from the MDVP- IeryP-Inttuut-Daaeee This diagram illustrates the distribution of images sourced from various domains. It also highlights GPT-assisted sample, emphasizing the diversity and richness its point-based and region-batt based instruction-folloiin data
2003). Visual 7W Zhu et al. 2016), and VCR (Zellers et al. 2019) datasets. (ii) Constructing with GPT-4V Assistance. To build more comprehensive and diverse cnttruction-oolloiinnn dataset, we collected various multi- images along with their unique annotations from public datasets (Table 5) These include natural images, Optical Character Recognition (OCR) content the wild, docu- ments, webpage screenshots, mobile screen captures, remote sensing imagery, and multi-panel images. For each distinct image domain, we meticulously crafted prompts to facilitate GPT-4V's ability to adopt various roles in generating instructional data. Specifically, we assigned GPT-4V four distinct tasks: brief captioning. detailed captioning, inter-relationship analysis. and complex reasoning generation. Notably, the captions created for each domain exhibit unique characteristics. For instance, in mobile screen captures, objects are identified not only as relevant components- such as icons, text, search bars, and URLs but also include information about potential actions, indicating whether a component clickable and the expected outcome upon interaction. This diversity significantly enriches the dataset while also presenting challenges. Fur- thermore, as shown Fig to enhance GPT-4V's recognition objects referenced by visual prompts, we employed Set-of-Marks (SoM) prompting (Yang al., 2023). This method directly highlights objects in the input images, ensuring a strong association between the generated data and the referenced objects. Addi- tionally, we propose incorporating the category information of each object within the prompts, which greatly enhances the quality of the generated data. (Details about data construction can be found Appendix D
the point-QA Shim
MDVP-Bench. To evaluate the proficiency of MLLMs in visual prompting tasks and their versatility across various domains, we initially curated subset of our HDDVP-nnttruttDall This subset underwent a thorough manual content filtering and refinement process, resulting in the creation of HDVP-Bende MDVP-Bendh
4

---

Published as conference paper ICLR 2025
LLM
RRegion squirell, it's eating pine comees
Eacoder
Pronpe
LLM (LLaMA2
/ LLaMA3 / InternLM2 Qwen2)
Mtge Alignment Predraining
Freegn Lncnder
Stage 2: Segervised Fiactaning
(n) Two-Stage Trininng Strategy
(b) Visual Promgt Focoder
HRP
Vision Encoder
Visual Prompt Encoder
Tokenizer
What animal <Region 1> What doing?
Volldhy identifion
Enceding
(c) Overall Architecture
Figure (a) The proposed training strategy for our VP-MILM. (b) The detail of visual prompt encoder. (c) The overall architecture of the VP-MILM. The Vision Encoder, Tokenizer, and LLM are derived from pre- trained MILMs such as LL (2024), SPHINX (2024), Qwen-VL (2023). and others. By incorporating proposed visual prompt encoder, we can feed the image tokens, visual prompt tokens, and text tokens together into the LLM for training, equipping the MLLM with visual prompting understanding capabilities.
serves as a challenging benchmark that encompasses a wide range of tasks, including concise descriptions, elaborate narratives, analyses of interconnections between regions, and complex reasoning.
For open-ended evaluations, existing methods typically follow LLaVA's approach (Liu et al. 2024b), using GPT4 with textual descriptions to depict image content. However, these descriptions often depend on image annotations, which can lead to situations where GPT-4 fails to recognize unannotated objects or backgrounds. Additionally, inherent domain gaps between textual descriptions and images may result in misunderstandings of the image content. To ensure more robust evaluation, we employ GPT-4V. We directly annotate images using SoM prompting (Yang et al. 2023), submitting both the images and textual questions to GPT-4V for scoring. The scoring follows the LLVA-bend guidelines, with scores ranging from to 10, where higher scores indicate better model performance.
3 VISUAL PROMPTING MLLM
this section, we detail the process of integrating visual prompt understanding into pre-trained MLLMs and transforming them into Visual Prompting MLLMs (VP-MLLMs). We also introduce a training strategy designed to enhance alignment and fine-tuning for VP-MLLMs.
3.1 ARCHITECTURE
Overall Architecture of VP-MLLM. For most existing MILMs, the overall architecture consists of three components: vision encoder, tokenizer (text encoder), and a Large Language Model (LLM). Each modal- ity is processed by its corresponding encoder, and the resulting tokens are concatenated and fed into the LLM for learning. Similarly, to achieve visual prompting understanding. to embed the input visual prompts, as shown in Fig. [c.. This integration allows us to combine the image, visual prompt, and language representations and forward them collectively to the LLM.
incorporate visual prompt encoder
5

---

Published as a conference paper ICLR 2025
Visual Prompt Encoder. As shown in Fig. 4b), we introduce a simple yet effective visual prompt en- coder that focuses on two types of visual prompts: points and bounding boxes. Initially, the encoder utilizes positional encoding([ange Tancik al., 2020) for the coordinates of both points (center) and boxes (top-left and bottom-right corners). It then adds three distinct learnable embeddings for each corner and processes them through a linear layer to obtain unified output embeddings. Additionally, we accommodate dynamic num- ber of visual prompts as input. We first set a fixed number of visual prompt tokens (e.g.. 16). Based on the validity of the actual input tokens, we provide both valid and invalid tokens with a set of learnable vectors to help the model discern their effective features. Finally, we employ linear layer to map the embeddings of different prompt types to the same dimension, thereby unifying the various visual prompt inputs.
3.2 TRAINING STRATEGY FOR VP-MLLM
Image- Visual Prompt-Text Alignment Pre-training. We initially freeze both the pre-trained vision encoder and the LLM, then focus on training the features of visual prompts to align with those of the input image and text. Following the approach in LLaVA (Liu et al., 2024b), we implement an MLP to transform the visual prompt tokens into the latent space of the LLM. We use open-source detection and segmentation datasets to create our stage I training data. These datasets include wide range of objects and label types, such elements of the natural world (e.g., people, animals, objects), remote sensing (c.g.. buildings, roads, vehicles, water bodies), document components (e.g., titles, paragraphs, images, tables), OCR data (e.g., text recognition), and screenshots (e.g., icons, text, search bars). For point visual prompts, we randomly sample pixels from semantic segmentation images, where each point corresponds to pixel- level label annotation. For box visual prompts, directly use the ground truth bounding boxes from detection datasets as inputs, enabling the model to recognize their corresponding labels. With this rich and diverse data for pre-training, the model is well-equipped for visual prompting and object categorization. The datasets used in stage are shown in Appendix (Ta
Stage 1:
Stage 2: Multi-Task Instruction Finetuning. At this stage, we load the weights trained from stage and keep the vision encoder and visual prompt encoder weights frozen. We then fine-tune the visual prompt pro- jector and the LLM. This stage focus on enhancing model's ability to accurately interpret user instructions and handle diverse visual prompting understanding tasks, such as detailed captioning, inter-relationship analysis, and complex reasoning, while maintain the original robust vision-language global understand- ing capability. Table ( posed MDVP--nnttuur--Dtaa, Visual Genome (217). Visual Commonsense Reasoning (219). Visual7w (2016). Osprey-724k (2024a), and multiple open-source image-level instruction datasets.
outlines all the data utilized during stage 2 fine-tuning, which includes our pro-
Simulation Training for Free-Form Visual Prompt Inputs. To support free-form visual prompts, we introduce a noise-based augmentation during the alignment stage to simulate the required input area. For box prompts, Gaussian noise proportional to the box size is applied, resulting in bounding boxes that may exceed partially cover the target, thereby approximating free-form visual prompt's enclosing rectangle. For point prompts, we sample multiple pixels within the target's mask, guiding them toward the same object enable precise point-based referencing. At inference, free-form inputs are pre- meepproessed into bounding boxes, enabling flexible, user-drawn prompts.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
For evaluation. we primarily use two popular MLLMs: SPHINX-X (Gao et al. 2024) and LLaVA-Next- LLaMA3 (Liu et al., 2024a), along with three model sizes: 7B, 8B, and 13B. We transform these models into VP-SPHINX and VP-LLaVA based our proposed framework. We employ AdamW (Loshchilov & Hutter 2017) as our optimizer and leverage flash attention (Ford et al. 2009) to enhance computational
efficiency. During the stage 1 training phase, we set the starting learning rate 4e- 5. In stage 2, the initial
6

---

Published as a conference paper ICLR 2025
Table 1: Results of referring classification on LVIS and PACO, and COCO text. Calculation of Semantic Similarity and Semantic IOU was performed using box visual prompts. We randomly perturb the center positions and scale the dimensions of the box visual prompts to simulate free-form inputs.
2019
(A23)2023
COCO 20x6
Method
Semantic Similarity
Semantic IOU
Accuracy
Semantic Similarity
Semantic IOU
Accuracy
Point
Box
Free-Form
Box
LLaVA-7B 2023b
48.95
19.81
50.1
50.3
42.20
14.56
Shikra-7B(20255 20225
49.65
19.82
57.8
67.7
43.64
11.42
GPT4Rol-77 2023a
51.32
11.99
61.8
48.04
12.08
ChatSpot-777
2023
64.5
31.8
Osprey-7B(2222 2024a
65.24
38.19
73.06
52.72
Ferret-13B 2023
64.96
37.82
68.4
71.0
Ferret-v2-138((233
75.1
87.7
76.4
VP-SPHINX-7BB
86.02
61.24
85.44
88.50
86.19
74.15
49.88
43.67
VP-SPHINX-13B
87.06
62.90
86.46
89.82
88.96
76.82
51.32
45.44
VP-LLaVA-8B
86.67
61.52
85,85
89.44
88.09
75.67
50.04
44.82
learning rate was adjusted to le 5. The imput images were processed using each model's unique dynamic resolution mechanism, and the maximum sequence length for the Large Language Model (LLM) was set to 3072. In all evaluation experiments, we will not continue to fine-tune on specific dataset but will instead adopt zero-shot testing approach.
4.2 REFERRING CLASSIFICATION
Object Classification. This task is defined as follows: the question targets a specific area within the image, requiring the model to identify the object in that designated region. Following (Yuan al., (024a), we employ two semantic relevance Indicators-Semaete
Similarity (SS) and Semantic Intersection over Union (Reaati(eeatoiighl et al., 2019) assess the model's Is classification performance the validation sets of the object-level LVISt et (019) and part-level PACO (Ramanathan et al. (2023) datasets. As shown in Table 1 our VP-SPHINX and VP-LLaVA significantly outperform offeeoffthhear methods in terms of SS and S-TOU by a notable margin.
We also conducted tests on traditional closed-set object classification tasks. To ensure the model outputs names of categories within the closed set, we followed the approach described in (You et al., 2023) and adapted our evaluation a binary-choice format. We posed questions such as, "Please identify the labels of each marked region in the image. Is the region (Class A) (Class B)?". The accuracy results presented in Table indicate that our VP-MLLMs demonstrate strong capability for accurately locating the Region of Interest (Rol) and identifying the corresponding object categories within that region.
Regional Optical Character Recognition. Optical character recognition (OCR) focuses on identifying text in images and is fundamental aspect of visual entity recognition. Following the approach in (Zhao) jtt al. 2023), we utilize the COCO-Text dataset Veit et al., (2166)) to assess the regional text recognition capabilities of our VP-MLLMs. Using the ground-truth bounding boxes provided in the dataset annotations, we prompt the model with requests such as, "Please provide the OCR results for the marked region in the image." The VP-MLLMs then analyze and respond with the textual content present in the specified regions.
Given that most visual prompt-based models do not support OCR, we compare our results with ChatSpot under identical zero-shot settings. As shown in the right column of T our models outperform ChatSpot by more than 10,, highlighting its promising capability in regional text recognition. Moreover, as shown in Fig. 5 our VP-MILMS not only recognize text in the specified regions but also exhibit enhanced under- standing and descriptive abilities. For instance, they can describe characteristics such as font type, color, and background. This underscores our models' exceptional pixel-level understanding capabilities.
7

---

Published as a conference paper ICLR 2025
Table 2: Region-level captioning performance on the validation set of RefCOCOg and performance com- parison on general MLLM benchmarks including VQA and OCR.
Method
Re1C0C0y22116
VQA 2017
MME 2024
POPE 2024.
SEED
TeuVQQ42019
DocVQA 2021
GPT-4V METEOR
CIDE
Generl VQA
OCR
Swen-VLL-74k
2024g
78.8
56.30
61.8
65.1
LLVV--Next-a
2021a
1603.7
64.6
59.7
SPHINX-X X-MXXXXX-33--
Shikra 7B 2023b
40.97
53.8
Shikra-133F2
1335 20236
13.16
77.4
59.2
Osprey-7B
77.54
16.6
108.3
Ferret-22
58.7
1o11 2020
81.8
1521.4
88.1
61.7
62.2
VP.SPHIXX
84.37
21.0
138.7
76.3
1377.0
88.6
71.4
59.3
58.1
VP-SPHINX-13H
88.19
23.9
1412.2
VP-LLaVA-SB
36.67
22.4
58.8
4.3 REFERRING REGION-LEVEL CAPTIONING
Brief Region Description. We provide quantitative comparisons for the region-level captioning task with using both mask- and box-based approaches. Specifically, we employ box visual prompt and text prompt, such "Please provide a brief description of each marked region in the image," prompt our VP-MLLM to concisely describe the content of the targeted region. Experiments were conducted on the validation sets of RefCOCOg. Following (Yuan et al., 2024a), we use the METEOR and CIDEr scores evaluate the semantic similarity between the generated captions and the ground truth. As the results shown in the left part of Table 2 our VP-MLLMs demonstrate superior performance compared to Kosmos-2 and Osprey.
Detailed Region Description. To evaluate the detailed region description capabilities, we leverage GPT-4 to measure the quality of responses for input referring regions. Following the approach used in Osprey (Yuan et al. 2024a), we sample 80 images from the RefCOCOg validation set (Yu et al. (016)) to generate detailed region captions using box visual prompts and text prompts like, "Please provide e detailed description of each marked region in the image." GPT-44 is then used to assess the captions generated by the MLLMs, with evaluation scores ranging from to 10 and calculate the ratio of the predicted score to that of GPT-4, expressed as percentage. The results shown in the first column of Table 2 indicate that VP-MILMS achieve the hest performance, with VP-SPHINX- -33 reaching an accuracy of 88. 18%,,, significantly outper- forming other region-basee based and mask-based methods. In comparison other models of similar size, our VP-LLa VA-8B also achieves a score of 86.67% 67%,, surpassing the current SOTA method Osprey-7B by 9.13%.
4.4 COMPREHENNSIE ASSESSMENT
To comprehensively evaluate the effectiveness of our VP-MLLMs, we first use LLaVA-Bench to compare with previous models and assess general image-level understanding capabilities. Since the questions in LLaVA-Bench and general MLLM benchmarks are based on the entire image, we use bounding boxes that cover nearly the full image visual prompts. The results are shown in Table Our VP-MLLMs demon- strate highly competitive performance on complex reasoning tasks and significantly outperform other visual prompting MILMMs in conversation and detailed captioning tasks. Additionally, we conducted evaluations on a series of modern MILM benchmarks. As shown in Table 2 our VP-MLLMs exhibit leading perfor- mance across image-level benchmarks such as MME, SEED-Bench, POPE, and OCRBench compared to existing visual prompting methods. Although their overall performance is slightly lower than that of the original MLLMs, we attribute this situation partly to our use of only a subset of open-source instruction data for training, which is in line with expectations.
Furthermore, we use Ferret-Bench and our proposed MDVP- -ennh to evaluate the pixel-level visual prompt- ing capabilities of our models. Experiments conducted on the Referring Description and Referring Rea- soning tasks within Ferret-Bench, with the results presented in Table 3 In both tasks, our models present competitive performance compared to the current SOTA model, Ferret-v2. For MDVP- Bench, combine
8

---

Published as a conference paper ICLR 2025
Table 3: Performance on the LLaVA Bench, Ferret Bench, and our MDVP Bench.
LLaVA Bench 30236 B0236
Ferret Bench 2023
MDVP Bench
Method
Conve- isation
Detail Description
Complex Reasoning
Referring Description
Referring Reasoning
Natural
Panel
Screenshot
Box
Point
Box
Poiot
Box
Point
Box
Point
MavA-778
202
85.4
68.3
92.1
41.4
31.7
SPHINX-X
2024
55.6
70.2
Kosmos-2
7025
71.7
63.4
74.9
51.8
33.7
Osprey-75
72.2
Ferret7DD
84.4
79.4
96.3
68.7
67.3
84,92
81.42
21.24
12:20
44.42
32.08
26.40
10.62
Ferrer1 155
2124
85.2
80.9
96.4
70.6
68.7
86.67
83.40
27.68
12,20
49.83
37.65
30.92
14.40
Ferce-v2-1
79.6
VP:SPHINX-11
81.1
X5.1
79.2
75.1
68.2
83.57
89.82
74.28
81.67
74.21
75. 24
76.72
50.4
VP-SPHINX-138
84.6
86.4
83.3
77.4
71.4
88.82
92.95
78.49
85,29
77.30
79.08
80.07
54.05
VP.LLaVA-8B
78.9
75.2
87.02
75.67
81.92
75.19
78.92
different tasks to provide a comprehensive assessment of the model's overall capabilities. As the results show, compared to Osprey and Ferret, our VP-MLLMs significantly outperformed them across multiple do- mains. This improvement can be attributed to training our MDVP-Instruct-Date which highlights the strong fine-grati une-ggained understanding and reasoning abilities of our VP-MLLMS.
Notably, we observed that box prompt scores were sometimes lower than point prompt scores. After review- ing the model's responses, we found that in tasks with fewer visual prompts per image, box-based prompts performed better, while in tasks involving more visual prompts, point-based prompts showed significantly stronger performance. This finding highlights the effectiveness of our visual prompt encoder in modeling different types of visual prompts and suggests potential areas for future research and enhancement.
Figiting Degesior The Blrks
What detala cas be sonr in :ogion I-d
Plenec aniyue celatieakip betwees 'Miak
Plane provide mo dosortiotii <Regicn ust Ragion to.
VP-PMIN
The Stapon baca white and slack for give, string skirly
Hott and Mark fandamental
paten,
storegin,
sapport, pigchlogiio wall-butng
<Reginn for Asmall bodaned dioure remors arax
VP-LLaVA
In Regten the cur with criting blok
VINLLA
< Mark 1s and Mark to aee counttii officcm spprrches. o Mal. Es piseilise sontal contiional will bemy while Mark haghiggna the importance ofplysicl supprting overall wolleoss
VILELAA
Ragicn Its There is siabll industrial postionno clese to agrizatunal nedss -Regicn modks stanth alene
while fu, atccttids old lona gace shasp
1200 PITTMAN RD
Ber
Ploae provade dentied descripize of Meggon Do.
User
proside detaiee doer -Regón to.
Uver
insepeat
aquay?
VE-5HIEX
This loss represntaa aboppina carl, which is commen synbel for excremnnn celine
VPP 4PHINN :Repion
bold,
SP-IINN
Total decklalies
nant feee, with white coleribar sande rod bf sign.
resppeded the modeal inve te cnmpany she delacing labiliio.
shopptgg opsicaly inliato vection
The mam cemeenu are porminetty dapaayed, expaoting thei signifeano,, posiily indlediig louse comteer or pot
agra era the not vaice owned skacchrdor.
ther cumentyy the carr's
depict salopping cat, vildly reoogrized yoinss sociin
<Region To: the teld, sesseiff ruarbers "290" dd white standin agains aod hevtgnend slgn
VPLLLAA
is Megian Dr, asoddaater expaty refers tnnwising
corpeny's moets dowbeldirs, opprorting their enke clain de company'ss valaz alle decte are settle.
zemo destte kay. The mumter shows that them is cxmmtty fen works ove.
vidrees indiorin shair
Figure 5: Qualitative examples generated by our VP-SPHINX and VP-LLaVA.
9

---

Published as a conference paper ICLR 2025
Table 4: Ablation study on different embedding format of visual prompts, training strategy, and dataset.
LVIS
PACO
COCO Text
MDVV-Bench
Method
Semantic Similarity
Semantic IOU
Accuracy
Semantic Similarity
Semantic IOU
Accuracy
Natural
Point
Box
Point
Box
SPHINX- 133 w/ Alpha Blending
56.89
33.73
46.84
40.53
21.14
21.48
41.19
64.58
SPHINX-ISS 13B w/ text
57.74
34.68
49.59
40.17
22.66
20.12
46.71
63.85
SPHINX--33 VPE (ours)
68.74
39.29
69.57
52.73
27.64
25,81
67.73
72.54
VP-SPHINX- 13B w/ One-Stuge
61.13
33.69
62.94
46.74
23.96
21.60
61.24
65.77
VP-SPHINX-130 w/ Two-Stage (ours)
68.74
39.29
69.57
52.73
27.64
25.81
67.73
72.54
VP-SPHINX-138 w/ all MDVP data
68. 74
39.29
69.57
52.73
27.64
25.81
67.73
72.54
w/ only Grounding QA data
67.67
37.24
67.98
50.93
25.14
22.90
52.59
54.64
w/ only GPT4v-related data
68.02
38.56
68.87
51.34
26.71
24.49
64.67
68.59
4.5 ABLATION STUDY
To evaluate the effectiveness of the key elements of our approach, we conduct the ablation experiments. Given the extensive amount of training data, our comparison was limited to the first 50k training iterations.
The Effectiveness of Visual Prompt Encoder. To assess the effectiveness of our visual prompt encoder (VPE), we explored two alternative methods for incorporating visual prompts into MLLMs: (1) overlaying the visual prompts on the original image via alpha blending with an alpha value 0.5; and (2) explicitly including the coordinates of the visual prompt in the instructional text prompt. As shown in Table 4 while these alternative methods enabled the MLLM to successfully identify the regions indicated by the visual prompts, their performance that our visual prompt encoder possesses superior capabilities for processing
not strong as that achieved with our visual prompt encoder. This suggests
prompts.
The Two-stage Training Strategy. To validate the effectiveness of our proposed two-stage training strat- egy. we conducted an additional experiment in which we bypassed the alignment phase (stage 1) and set both the visual prompt encoder and the LLM to trainable state for comprehensive model training. As shown in Table 4 the results demonstrate that, within the same training duration, the two-stage training approach significantly enhances visual prompting understanding. yielding improvements across all evaluated metrics.
The Importance of GPT-4N-Constructed Data. Since MDVP-Instruct-Datt is constructed from open- source grounding QA datasets and data constructed by GPT-4V, we further validate the effectiveness of GPT-4V-related dataset. Specifically, we divide MDVP-Instruct-Date into two parts: one contains only data from the Grounding QA datasets, while the other consists solely of GPT-4V-related data. We then perform fine-tuning on each part separately. The results, as shown in Table[4] indicate that for both classification tasks and complex reasoning and dialogue tasks, the GPP-4V-construce underscoring its effectiveness. Moreover, when we utilize the complete dataset, the model achieve the highest performance, highlighting the overall quality of our MDVP-Instruct-I
data provides significant improvements,
5 CONCLUSION
In summary, we introduce new framework, Draw-and- -and-undrsttadd, designed to equip existing MLLMs with robust visual prompting capabilities while preserving their original image-level perception. With our pro- posed visual prompt encoder, our VP-MLLMs can simultaneously support multiple types of visual prompts, including points, boxes, and free-form shapes, greatly enhancing user flexibility. Additionally, we curated the MDVP dataset, which comprises 1.2 million high-quality imate-point-text and image-region-tes. triplets for model training. along with the comprehensive and challenging MDVP-Bendh dataset for evaluation. Our superior performance in various visual prompting tasks demonstrates the efficacy and robustness of our VP- MLLMs. We believe our contributions provide a solid foundation for further exploration in the field of intelligent visual interaction systems.
10

---

Published as a conference paper ICLR 2025
6 ACKNOWLEDGMENTS
This project is funded in part by National Key R&D Program of China Project 20222D0161 100, by the Cen- tre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK, by NSFC-RGC Project NCCUHK49//24 Hongsheng Li is a PI of CPII under the InnoHK.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08771 2023.
Jean-Raptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances Neural Information Processing Systems, 35:23716-2333 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P Meyer, Yuning Chai, Dennis Park, and Yong Jae Lec. Making large multimodal models understand arbitrary visual prompts. arXiv preprint arXiv:2312.00788 2023.
Chi Chen, Ruoyu Qin, Fuwen Luo, Xiaoyue Mi, Peng Li. Maosong Sun, and Yang Liu. Position-enhanced visual instruction tuning for multimodal large language models. arXiv preprint arXiv:2308, 100008.13337 13437, 2023a.
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684 2024.
Hao Chen and Zhenwei Shi. A spatial-temporal attention-based method and a new dataset for remote sensing image change detection. Remote Sensing, 12(10):1662 2020.
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P. Xing. and Liang Lin. Geoqa: A
geometric question answering benchmark towards multimodal numerical reasoning, 2022. URL https:
//axxiv.crgaabs/////. http://aa.///20...1m 24111
14517
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal 11m's referential dialogue magic. ar preprint arXiv:2306. 15195, 2023b.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12799, 2023c.
Hiuyi Cheng, Peirong Zhang, Sihang Wu, Jiaxin Zhang, Qiyuan Zhu, Zecheng Xie, Jing Li, Kai Ding, and Lianwen Jin. M6doc: A large-scale multi-format, multi-type, multi-layout, multi-language, multi- annotation category dataset for modern document layout analysis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15138-15141 2023.
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.1093: 2024.
11

---

Published as a conference paper ICLR 2025
Chee Kheng Ch ng and Chee Seng Chan. Total-text: A comprehensive dataset for scene text detection and recognition. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pp. 935 942. IEEE, 2017.
Aakanksha Chowdhery. Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240)11-13,, 2023.
Marius Cordts, Mohamed Omran, Sehastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schicle. The cityscapes dataset for semantic urban scene understand- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint #CXiv:240.221783, 2024.
Sean Ford, Marco Cova, Christopher Kruegel, and Giovanni Vigna. Analyzing and detecting malicious flash advertisements. In 2009 Annual Computer Security Applications Conference, pp. 363-372. IEFE, 2009.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL httpttsss///ii..oo////a//tt////a 2ttps/23233333330
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and- language representation learning. Advances in Neural Information Processing Systems, 33: 6616-6628, 2020.
Peng Gao, Renrui Zhang. Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for family multi-modal large language models. arXiv preprint arXiv:2402.0599: 2024.
Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting seed of vision in large language wrw.
model, 2023. URL
08041
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5356-5364, 2019.
Tuomo Hiippala, Malihe Alikhani, Jonas Haverinen, Timo Kalliokoski, Evanfiya Logacheva, Serafina Orekhova, Aino Tuomainen, Matthew Stone, and John A. Bateman. Ai2d-rst: multimodal corpus of 1000 primary school science diagrams. Language Resources and Evaluation, 55(3):661 688, December 2020. ISSN 1574-0218. doi: 10. 10.00077/ Utthttp::/dxx..tmm doi wtw.:/.........11000 s 810579-020-095172
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvga: Understanding data visualizations via question answering, 2018. Utl/:ptcp.///aarx..ocr//////...1 08163
12

---

Published as conference paper ICLR 2025
Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez i Bigorda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Almazan, and Lluis Pere De Las Heras. Icdar 2013 robust reading competition. 2013 12th international conference on document analysis and recognition, pp. 1484-1493. IEEE, 2013.
Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. Icdar 2015 competition on robust reading. 2015 13th international conference on document analysis and recognition (ICDAR), pp. 1156-1160. IEEE, 2015.
Geewook Kim, Teakgyu Hong. Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun. Dongyoon Han, and Seunghyun Park. Ocr-free document understanding trans- former, 2022. BtLLitttss///axivv..o
2111. 211..156611
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yes Lo, et al. Segment anything. arXiv preprint arXiv:2304002643, 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
Kartik Kuckreja, Muhammad Sohail Danish, Muzammal Naseer, Abhijit Das, Salman Khan, and Fa- had Shahbaz Khan. Geochat: Grounded large vision-language model for remote sensing, 2023. URL
sttps://arxivv.ogg wtt.a//.rrgra/////////// 2211 15826
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Ka- mali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection scale. International Journal of 2020.
Computer Vision,
Feng Li. Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and recognize anything at any granularity, arXiv preprint Htl::2307.04761 2023a.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 11336-1 1344, 2020a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint urXiv:2301.12597 2023b.
Ke Li, Gang Wan, Gong Cheng, Liqiu Meng, and Junwei Han. Object detection in optical remote sensing images: A survey and a new benchmark. ISPRS Journal of Photogrammetry and Remote Sensing, 159: 296-307, January 2020b. ISSN 0924-2716. doi: 10.1001//ispsspps2209....1
URL http://dd/
doi ori.org/Ii0 10. 101611161100.1 1st.:rggg/.1011//...ssssssssrps 2019 20t..11.199.1.00 11.023
Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, and Ming Zhou. Docbank: A benchmark dataset for document layout analysis. arXiv preprint #xXiv22006.01033 2020c.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models, 2024. URL
https: org. 18814
13

---

Published as a conference paper ICLR 2025
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023a.
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b. URL
https://arxins org/ / 2304 08485
Haotian Liu, Chumyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llava-next:
Inprovereaasonigng reasoning, oct, and world knowledge, January 2024a. UtL/:ttps://laaaaa tihub.
io/blog/2024-011 01-30- ava- next/
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang. Bo Li, Songyang Zhang, Wangho Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024c. URL httpt::////// /ttp///aavvvvv/aa///aa//eaaaaaaa org/ /abp//b////a 2307 2307.06281111111
Yuliang Liu, Hao Chen, Chunhua Shen, Tong He, Lianwen Jin, and Liangwei Wang. Abenet: Real-time scene text spotting with adaptive bezier-curve network. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9809-9811 9818, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arX1v:1711.05101 2017.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for saiio-and-languual tasks. Advances in neural information processing systems, 32, 2019.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022. URL URL abs/2203.1022
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Doevqa: A dataset for vqa on document images, 2021. URL
00398
Nibal Nayef, Fei Yin, Imen Bizid, Hyunsoo Choi, Yuan Feng, Dimosthenis Karatzas, Zhenbo Luo, Umapada Pal, Christophe Rigaud, Joseph Chazalon, et al. Icdar2017 robust reading challenge on multi multi-lingual scene text detection and script identification-rre-lll. analysis and recognition (ICDAR), volume 1, pp. 1454-1459, IEEE, 2017.
2017 14th IAPR international conference on document
Nibal Nayef, Yash Patel, Michal Busta, Pinaki Nath Chowdbury, Dimosthenis Karatzas, Wafa Khlif, Jiri Matas, Umapada Pal, (ean-Christoppe Burie, Cheng-lin Liu, et al. lcdar2019 robust reading challenge on multi-lingual scene text detection and recognition -ropgittoon-mmmmm--2201199 In 2019 International conference on document analysis and recognition (ICDAR), pp. 1582-1587. IEEE, 2019.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.1488 2023.
Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S Nassar, and Peter Staar. Doclaynet: A large human-annotated dataset for document-layout segmentation. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 3743-3751, 2022.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentent models. In Proceedings of the IEEE international conference computer vision, pp. 2641 26449, 2015.
14

---

Published as conference paper ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang. Aaron Marquez, Rama Kovvuri, Abhishek Kadian, Amir Mousavi, Yiwen Song, Abhimanyu Dubey, and Dhruv Mahajan. PACO: Parts and attributes of common objects. In arXiv preprint #Xiv:2301.01799 2023.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multi- modal model. arXiv preprint arXiv:2311.03355 2023.
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Gen- eralized intersection over union: A metric and loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 658-666, 2019.
Kyungah Shim. Efficient id-based authenticated key agreement protocol based on weil pairing. Electronics Letters, 39(8):1, 2003.
Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about red circle? visual prompt engineering for vlms. arXiv preprint arXiv:2304.06712, 2023.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read, 2019. URL httts://aaxiv.org/////
Huan Su, Shuang Wei, Ming Yan, et al. Object detection and instance segmentation in remote sensing imagery based on precise mask r-cnn. In IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium, pp. 1454 1457. IEEE, 2019a.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual- linguisie representations. arXiv preprint
2019b.
Xian Sun, Peijin Wang, Zhiyuan Yan, Feng Xu, Ruiping Wang. Wenhui Diao, Jin Chen, Jihao Li, Yingchao Feng, Tao Xu, Martin Weinmann, Stefan Hinz, Cheng Wang, and Kun Fu. Fairlm: A benchmark dataset for fine-grained object recognition in high-resolution remote sensing imagery, 2021. URL https:
arxiv. org/ abs 2201/11331.. 05569
Jiashun Suo, Tianyi Wang, Xingzhou Zhang, Haiyang Chen, Wei Zhou, and Weisong Shi. Hit-uav: A high- altitude infrared thermal dataset for unmanned aerial vehicle-based object detection. Scientific Data, 10 (1), April 2023. ISSN 0052-4463 doi: 00013//41157700230000000 URL http: doi www.doo..org/// org/ 10. 1t.1008//41597-02-2006666
Milad Taleby Ahvanooey, Hassan Dana Mazrach, and Seyed Hashem Tabasi. An innovative technique for web text watermarking (aitw). Information Security Journal: A Global Perspective, 2914-6): 2016.
Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmre: Machine reading comprehension on docu- ment images, 2021. URL /t//:ttts/////oxw..oo///////./..
15

---

Published as a conference paper ICLR 2025
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions low dimensional domains. Advances in Neural Information Processing Systems, 33:7537- 7547, 2020.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and finc-tuned chat models. arXiv preprint arXiv:2307.0923 2023.
Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07140, 2016a.
Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images, 2016b. URL https://arxiv
abs/1601.07H
Sara Vicente, Joao Carreira, Lourdes Agapito, and Jorge Batista. Reconstructing pascal voc. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 41-48, 2014.
Jiaqi Wang, Pan Zhang, Tao Chu, Yuhang Cao. Yujie Zhou, Tong Wu, Bin Wang, Conghui He, and Dahua Lin. V3det: Vast vocabulary visual detection dataset. arXiv preprint arXiv:2304.03752 2023.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2024a.
Xudong Wang, Shufan Li, Konstantinos Kallidromitis, Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. Hicrarchical vocabulayy universal image segmentation. Advances in Neural Information Processing Systems, 36, 2024b.
Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection in aerial images, 2019. URL
htttps:/aaat wwtpaawwwwwww...///oa/aaa
10399
Bingo Xumo. Ucas-aod ffdp. http:s///tttt...comm/ioooooo///p FFDP 2023. Ac-
cessed: 2024-09-27.
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding gpt-4v, arXiv preprint arXiv:2310.1144: 2023.
Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun, Cpt: Colorful prompt tuning for pre-trained vision-language models. arXiv preprint arXiv:2109. 2021.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023.
Fei Yu. Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernic-vil: Knowledge enhanced lasion-language representations through scene graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 3208-32 0216, 2021.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara I. Berg. Modeling context in referring expressions. In Computer Fision-ECCy 2016: 14th European Conference, Amsterdam, The Netherlands, October 11- 11-14, 2016, Proceedings, Part II 14, pp. 69. Springer, 2016.
16

---

Published as a conference paper ICLR 2025
Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Os- prey: Pixel understanding with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28202-2822 222212 2024a.
Zhenghang Yuan, Lichao Mou, Yuansheng Hua, and Xiao Xiang Zhu. Rrsis: Referring remote sensing image segmentation, 2024b. UtLhttss::/arrt
org/ abs/ /200.. 08625
Liu Yuliang. Jin Lianwen, Zhang Shuaitao, and Zhang Sheng. Detecting curve text the wild: New dataset and new solution. arXiv preprint uxiv:1112.0021700 2017.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual common- sense reasoning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Yang Zhan, Zhitong Xiong, and Yuan Yuan. Rsvg: Exploring data and models for visual grounding on remote sensing data. IEEE Transactions on Geoscience and Remote Sensing, 61:1-13, 2023. ISSN 1558-0644. doi: 10.1 1w.1100//rrr.22233332220 8250471
URLhttp://dx.
DTR.0////TRR... 2023.
Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong- You Chen, Tsu-Jui Fu, William Yang Wang. Shih-Fu Chang, Zhe Gan, and Yinfei Yang. Ferret-v2: An improved baseline for referring and grounding with large language models, 2024. WRLhhttps:///rrt wwtp://rggg////o/a/o/o/ooo.ss///
abs ///b////444
07973
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint 0tti:::333.0060000 2023a.
Xiangrong Zhang, Tianyang Zhang, Guanchun Wang, Peng Zhu, Xu Tang, Xiuping Jia, and Licheng Jiao. Remote sensing object detection meets deep learning: meta-review of challenges and advances, 2023b.
org/ abs 2309 06751
Yuanlin Zhang, Yuan Yuan, Yachuang Feng, and Xiaoqiang Lu. Hierarchical and robust convolutional neural network for very high-resolution remote sensing object detection. IEEE Transactions on Geoscience and Remote Sensing, 57()):5535-5548, 2019.
Liang Zhao, En Yu, Zheng Ge, Jinrong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chunrui Han, et al. Chatspot: Bootstrapping multimodal Ilms via precise referring instruction tuning, arXiv preprint arXiv:2307.09474 2023.
Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset ever for document layout analysis. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1015- 1022. IEEE, 2019.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Se- mantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321, 2019.
Qiang Zhou, Chaohui Yu, Shaofeng Zhang, Sitong Wu, Zhibing Wang, and Fan Wang. Regionblip: unified multi-modal pre-training framework for holistic and regional comprehension. arXiv preprint arXiv:2308.02299 2023.
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi- detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7571-7580 2022.
17

---

Published as a conference paper ICLR 2025
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4995- 5004, 2016.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang. Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances Neural Information Processing Systems, 36, 2024.
18

---

Published as conference paper ICLR 2025
APPENDIX
Table 5: Statistics of Training Data Stage This data also serves as the source for the GPT-4V-construuteed dataset within Matt--nstructtDatta
Type
Raw Data
#Samples
COCO
Yu al. 2016
LVIS
Gupla al.
2019)
Visual Genome Krishna et
2017
OpenImages Kuznetsova al.
2020
Natural
Object365 Zhou
V3Det Wang et al. 2023
4.6M
ADE20k Zhou
2019
Cityscapes Cordis al.
2016
Pascal VOC Vicente el al. 2014
Flickr30k (P [Plummer al.
2001
Dochank (I. ct al. 2020c
PublayNet (Zhong al. 2019
Document layout
2.5M
DoclayNet PPitzman el al.
2022
M6Doc Cheng ei al. 2023
ICDAR13 Karatzas of al.
2013
ICDAR15 Karatzas ct al 2015
CTW1500 Yuliang el al 2017
MLT201 Nayel et al.
2017
OCR Spotting
550K
MLT2019 Naaef et 2019
totaltext (C ng Chan
2017
CurvedSynTextTS0k (Lil
Z020
DIOR (Li et al. 20206
DOTA Xia
2019
NWPU VHR-IU (Su
2019a
FARIM Sun el al. 2021
Remote Sense
420K
RSOD (Zhang al. 2022h
HRRSD Zhang cf al. 2019
UCAS (Numo 2023
HIT-UAV Suo el al. 2023
Android & Web
ATTW Taleby Ahvanooey et al.
2016
SeeClick (Cheng ot al.
2024
300K
Table 6: Statistics of Training Data in Stage 2.
Task
Raw Data
#Samples
Category Identification
Subset Stage-
1.5M
ReiCOCO Yu al 2016
RelCOCO+ Yu
2016
Brief Caption
RefCOCOg Yu al.
2016
Visual Genome
Krishna et
2017
1.6M
MDVP Instruct-
Detailed Caption
MDVP- Data
752K
Visual Genome (Krishna et
2017
RSOD Zhang al. 2023b
Relationship Analysis
DIOR (Li al. 20206
LEVIR Cenn & Sh1
2020
1.3M
MDVP-Instrur Instruc Cu Data
GRTT You et al. 2023
Flicker30K Plummer et 2015
Multi- Multi-Target Captioning
OpenPsgGCC GCG (Rasheed al
2023
RRSIS-D D Yuan et al 2024b
1.3
GcoChat (Kuckreja a ct al. 2023
VCR (Krishna et al. 2017
Visual7W Zhu et al. 2016
Referring Q&A and Reasoning
Osprey-724X. Yuan ct al 2024a
DIOR-RSV (Li al 2020b
2.7M
OPT-RSVG Zhan et al 2023
MDVP HDVVP-Instucl Dala
laionGPT4y (Chen el
2024
A12D HHipppla et all 2020
ChartQA Masry cl al.
2022
DocVQA (aateew al
2021
General Image Q&A and Reasoning
DVQA Kalle ct al. 2018
GeoQA (Chen ct al. 2022
LLaVA Instrucsttcc...mmm TSDK
2024b
shareGPT4y (Chen
2023e
4+M
VisualMRC
Tanaka
SynthDoG (Kim al.
MGM_Instretin
2024
In-house Dataset
A IMPLEMENTATION DETAILS
A.1 MODEL SETTINGS
To construct our VP-MLLMs, we start by inheriting the entire model structure from various pre-trained MLLMs. Specifically, for VP-SPHINX-X, utilize the Mixture of Visual Experts (Mo al.)
19

---

Published as a conference paper ICLR 2025
2024 as the base image encoder and LLaMA2-7B/13BB Touvron et al. (023) the large language model (LLM). For VP-LLaVA-88B we employ encoder and LLaMA3-8B (Dubey et al. 2024) as the LLM. We adopt each pre-trained MLLM's respective Any Resolution strategy to process input images, effectively capturing their details. This strategy primarily involves sub-image splitting, image padding, and optimal aspect ratio selection, ultimately generating a flattened list of image tokens. Subsequently, we introduce our proposed visual prompt encoder (VPE) to process visual prompt inputs, yielding visual prompt tokens for modeling. These image tokens, visual prompt tokens, and text tokens concatenated sequentially to form single input for the LLM. Notably, we prepend and append a learnable token to the visual prompt tokens to serve as start and end markers.
(336") (Radford et al. 2021) as the base image
A.2 TRAINING DATA
Our final fine-tuning data mixture comprises diverse range of datasets, encompassing not only a variety of image domains but also a wide array of task types. These tasks include pixel-level and region-level classification, brief captions, detailed descriptions, relationship analysis, and complex reasoning and Q&A.
Stage 1. Our pre-training tasks are similar to the image captioning tasks commonly used in mainstream MLLMs pre-training stage. However, they focus on learning image-level descriptions, our VP-MLLM targets pixel-level and region-level semantic classification. Specifically. we will first collect open-source datasets related to object detection, instance segmentation, and semantic segmentation. From their anno- tations, we can obtain ground truth pairs such as (bbox, label), (mask, label), and (pixel, label). Next, we will use bbox, mask (randomly sample a few points), and pixel (point) as visual prompts inputted into the VP-MLLM to train the model to respond with the corresponding category. During this stage, only the visual prompt encoder and the projection layer are trained. The structure our stage-1 pre-training data is as follows:
("from": "human", "value": "Please identify the labels of each marked point in the image."),
("from": "GPT", "value": "<Mark 1>; Label I'nss Mark 2>; Label 2in< Mark 3>; Label 3'm.......")
("from": "human", "value": "Please identify the labels of each marked region the image. "),
("from": "GPT", "value": "<Region >>: Label Ins Region 2>: Label 2'n= Region 3>: Label 3'mn......" ")
("from": "human", "value": "Please recognize the text of cach marked region in the image. "), Text 1'ns Region 2>; Text 2'n Region Text 3'n...... "3
("from": "GPT", "value": "<Region 1>:
All datasets used in stage 1 are open-source detection and segmentation datasets. A complete list of the datasets can be found in Table 5 Specifically, we collected data from five different image domains to en- hance data diversity: (1) Natural Images: containing over 10k real-world semantic categories; (2) Document Images: including layout classifications such as titles, abstracts, paragraphs, and images; (3) OCR in the Wild: mainly for recognizing text in natural scenes, such as billboards and signage: (4) Remote Sensing: for identifying different regions in remote sensing images, such as playgrounds, vehicles, and pedestrians; (5) Mobile and Web Interfaces: recognizing key elements in mobile and desktop user interfaces, such as icons, text, and search bars
Stage 2. In Stage 2, our primary focus on enhancing the VP-MLLMS' ability to accurately interpret user instructions and handle diverse pixel-level understanding tasks, including detailed captioning, inter- relationship analysis, complex reasoning and so on. T an overview of all the data used during Stage 2 fine-tuning. Figure 6a presents an example of converting object detection data into visual prompt training data.
Training Hyperparameters. Both Stage 1 and Stage 2 training conducted on 8 A100 GPUs. We show the training hyperparameters for both stage 1 iisioonlanguaae-visus prompt alignment pretraining and the stage instruction tuning in Table
20

---

Published as conference paper ICLR 2025
Annotations from Detection dataset
Input Image
Visual Prompt
Text Conversation
Category
[K.YW.H]
Qusonficttnn
seerity
dRggin Getcher dhagion
Catcher
[12,37,873301]
Snealer
Inter Relefinnns
99
@J Plecse andyaa relotieshhp Ai dlegiin IP drond bateboll cothcher etaglen The
Player
[10,32.82,292]
Baseball
(342,37..)
@) playrr sheport and eagest: tummmttes appooeme They wear differest jersepi and
Sneeker
[36,290,52.24]
(a) An example of converting open-source object detection data into visual prompt training data. The visual prompt training data consists of three parts: the image, the visual prompt, and the text prompts (question and answer). The image laken directly from the object detection dataset, while the visual prompt is the ground-truth bounding with slight perturbations. The text prompts can either be selected from fixed prompt templates (c.g., the red text Classification on the far right) or be enhanced by GPT-4V (c.g., the blue text of Inter- Rela--eetii and Conversations & Reasoning the far right).
Detail Description
Real--Realltionhhip Analysis
Complex Reasoning
Q:
Q:
Pleose provide detalled desceiption each nanked point image.
Pleaze andyyne the relationehip betheenn narked regions image.
What purpess object bost
A:
1
to-Regin digiin
dRlegion 10 safeeyy deviion designed be throwen
side block and white attiigg open auttcone..
Bic The hemnet skateboard and shirt are all part of The cuanssarcor's
3.
the center
face.
The helmet L- glon. 1-3 provides protcction. Lhg
carten equipments et iagg indicatt about the activther
occupying signifi cont space
intenio...
akalaboed
A:
allark 3r The dog hat sharp
abit Lefoeaion part of The casudd stire commonil by wom by pertiopants actinity
cRegion suggestnnnnn
facial Feotunee with dark fur
for fishing
ground the spaaa white
(b) Examples from the Stage training data. Each visual prompt is associated with response that focuses describing the specific region interest.
Figure 6: Training Data Preprocessing and Construction.
Training Settings
Stage 1
Stage
Batch Size
256
64
Training Epochs
1
Warmup Epochs
0.03
0.03
Learning Rate
4 x 10-5
1 x 10-5
LR schedule
cosine decay
cosine decay
Gradient Clipping
8
8
Weight Decay
O
Optimizer
AdamW W
Adam W
Table 7: Hyperparameters of VP-MLLMS.
21

---

Published as a conference paper ICLR 2025
B RELATED WORKS
Multimodal Large Language Models. Recently, the development of Large Language Models (LLMs) has marked significant milestone in the field of Natural Language Processing (NLP). L.MSs such as the GPT series (Achiam et al., 2023), PaLM (Chowdhery et al. 2023). and LLaMA (Touvron et al. 2023) have not only achieved remarkable results in text processing but have also laid the groundwork for multimodal learn- ing. Building on this progress, the emergence of Multimodal Large Language Models (MLLMs), including BLIP-2 (Li et al. 2023b). Flamingo (Alayrac et al., 2022), and LLaVA (Chen et al. 2024). has expanded the application scope of LLMs beyond text other modalities. Notably, fine linengrained understanding of vi- sion has emerged as a new focus. Works such as VisionLLM (Wang et al.) 2024a) employ language-guided tokenizers extract vision features at specific granularities, showeasing the potential in this direction.
Visual and Multi-Modal Prompting. Visual and multimodal prompting in deep learning (Kirillov et al., 2023 Yuan et al. 2024a; You et al./ 2023; Cai al., [2023 Zhao et al.) (203) is an emerging area of study. Techniques utilizing visual prompts (e.g.. boxes, masks) aim to enhance model performance on specific visual tasks. Key developments include SAM (Kirillov et al. 2023) and its enhanced versions. which support a broad range of prompts. Further advancements, such as SEEM (Zou et al. 2024). HIPIE (Wang et al. 2024b), and Semantic SAM (Li et al. (023a)), have improved semantic prediction. Nonetheless, for nuanced real-world applications, models require multidimensional semantic analysis, incorporating color and spatial information to fully comprehend and reason about visual scenes.
Recent advancements such as GPT4Rol (Zhang et al. 2023a), Kosmos-2 (Peng et al. 2023). Shikra (Chen et al. 2023b), Ferret (You et al. 2023), GLaMM (Rasheed et al. 2023), and ViP-LLa VA (Cai et al. 2023) have enhanced MLLMs capabilities in region-specific image comprehension. Innovations like Colorful Prompting Tuning (CPT) Yao et al. 2021) and RedCircle ReeCCircleShtterittskt (Shtedritski et al., 2023) leverage color cues to
improve model interpretative skills. Osprey (Yuan et al.) (2024)) further advances this by enabling interactive, precise visual understanding using natural language and mask-based instructions. However, these methods face challenges with flexibility and complex reasoning, particularly in simultaneous multi-target referencing and their reliance on pre-defined masks, which hinders broader applications.
C MORE EXPERIMENTS
C.I REGION-LEVEL, REASONING
To evaluate the reasoning capabilities of our VP-MLLMs, we utilized the Visual Commonsense Reason- ing (VCR) dataset (Zellers et al. 2019). challenging benchmark designed to assess a model's high-level cognitive and commonsense reasoning abilities within context. The VCR dataset comprises multiple-choice questions that require an understanding of the scene depicted in an image. Each question (Q) is accompanied by four possible answers (A), necessitating the model to not only identify the correct answer but also provide a rationale (R) supporting its selection. This process underscores the model's proficiency in interpreting and justifying visual elements within specific contexts, with accuracy serving as the evaluation metric. Follow- ing the evaluation approach employed in LLaVA- -iP (Liu et al. 2024a) and GPT4Rol (Zhang et al., 2023a). we fine-tuned our models using the training set of VCR. As shown in the comparison results in Tab. R our VP-SPHINX- 133 achieves the highest performance scores of 88.92%, 90.23%, and 80.65% across three distinct evaluation methods, respectively, showcasing its proficiency in visual commonsense reasoning.
D MORE DETAILS OF MDVP-INSTRUCT--AAN CONSTRUCTION
In this section, we outline our methodology for reconstructing open-source grounding datasets and our ap- plication of GPT-4V to produce instruction-based based data spanning variety of domains, leading to the devel-
22

---

Published as conference paper ICLR 2025
Table 8: Validation Accuracy on VCR dataset.
Model
+ (%)
QA + R (%)
Q - AR (%)
ViLBERT Lu al. 2019)
72.4
74.5
54.0
Unicoder- VL (L el al
2020a)
72.6
74.5
54.5
VLBERT-L Su al. 2019b
75.5
77.9
58.9
ERNIE-ViL-EE Yu ct al. 2021
78.52
83.37
65.81
VILLA-L (Gan et al. 20200
78.45
82.57
65.18
GPT4RoI-7H (Zhang et al. 2023a)
87.4
89.6
78.6
ViP- VA-Lse---B Ba (Cai 2023
87.66
89.80
78.93
VP-SPHINX-13B
88.92
90.23
80.65
VP-LLaVA-88
88.43
89.96
79.71
opment the MDVP-Instruct-Datta Fig depicts an example from our dynamic multi-target captioning dataset. Sec D.2 showcase the prompts used to generate data across these diverse domains with GPT-4V.
D.1 EXAMPLES OF DYNAMIC MULTI-TARGET CAPTIONING DATA
Using sample images from the Flickr30k dataset (Plummer al. 2015) illustrations, the Phrase Grounding task involves identifying the bounding boxes an image that correspond to various referring phrases based on given ground truth (GT) sentence. twist on this task, invert the inputs (captions) and outputs (bounding boxes): the model is provided with bounding boxes as visual prompts and is tasked with generating natural language descriptions as responses. This process illustrated in Fig.
Please provide summarized description based on all the marked regions in the image.
HEEM MUBILES
Multi-Target Captioning
Flicker30K
Multi- Margi-Target Captioning
Flicker30K
Two people are talking outside of the video game shop next door the mobile phone store.
professional soccer player dressed black and white run after the ball after his opponent kicks it in the wrong direction.
Figure 7: Examples of dynamic multi-target captioning data.
23

---

Published as conference paper ICLR 2025
D.2 DETAILED PROMPTS FOR GPT4V
We carefully designed prompts to facilitate data generation by GPT-4V, assigning it various roles and em- bedding category-specific information within these prompts. Table 9 outlines a universal prompt template applied across different domains. Meanwhile, Tables 10
12 1 and 1etaiii prompts tailored to specific
domains. An illustrative example of GPT-4V's response during data generation is provided in Table 15
MORE QUALITATIVE RESULTS
We provide additional visual examples to further illustrate the pixel-level visual prompting understanding capabilities of our VP-MLLMs across diverse range of visual prompts. Fig8ddisplay a variety of visual scenarios from different domains, such as natural scenes, mobile and web interfaces, optical character recog- nition (OCR) in natural settings, document layouts, and multi-panel images. In these complex situations, our model showcases its proficiency in accurately responding to queries, highlighting its robust and versatile capabilities.
24

---

Published as conference paper ICLR 2025
image
4
3
2
messages- ("role": "system", "'content" : In the image, I have marked each visual object with green point, and each is identified white numeric ID against black background.
The categories of visual objects are as follows:
<Mark 1>: dog
<Mark 2>: bed
<Mark 3> mattress
<Mark 4>
pillow
Your analysis will revolve around four main functions: <Role>
will now supply you with specific output templates for content corresponding to the four roles. Please adhere strictly to these templates when generating content, refraining from making any additional alterations, including the insertion of extra spaces or line breaks. Adhering to this guideline is extremely crucial. Format:
<F <Format>
Proceed with your analysis, keeping the language natural and clear.) }
Table 9: The public prompt template used to feed GPT-4V for data generation. Public prompts for different domains exhibit slight variations. For instance, in the screen-shot domain, "In the image" is modified "In the screen-shots The highlighted <Role> and <Format> red domain-specific.
25

---

Published as conference paper ICLR 2025
Role
<Role (Short Description) Provide succinct and clear description for each marked region, suring each description stands independently without reference or comparison others.
<Role (Detailed Description) For each marked region in the image, please provide as detailed a description as possible using natural language. Highlight the object's category, type, color, and additional attributes such as location, condition, and any relevant details. Envision yourself observing each region directly and convey your observations thoroughly and promptly as possible. minimum of 30 words is required in your description. Treat each mark as unique separate entity, requiring full description exclusive mark only.
Analysis) Delve into and analyze the relationships between the marked regions. In your analysis, reference specific areas using identifiers like <Mark 1>, <Mark 2>, etc. Elucidate the links and common features among these areas, which might encompass aspects such as their spa- tial arrangement, inherent qualities, underlying principles, resemblances, variances, contextual ties, or notable discrepancies. Should you find certain relational aspects insignificant or lacking in noteworthy content, omit them from your discussion. In instances where marked region stands apart without clear ties to others, high- light its distinctiveness and provide detailed description of this unique object <Role (Q&A and Conversation iinnss) > Dive deeper into the detailed content and intricacies of every marked regions, and interconnections among multiple marked regions. Employ identifiers such as <Mark 1>, <Mark 2>, etc., to specify cach area in your inquiries and responses. Assist in formulating question-answa pairs that focus either single target area multiple target arcas, aiming develop rich dialogue dataset (comprising least Q&A pairs). Ensure that the questions you craft inquire about one or more specified <Mark>s.
<Role Cinter-Relationse
Format
Role 1
<Mark 1>: Your Short Description
<Mark N>: Your Short Description
Role 2
<Mark 1>: Your Comprchensive Description
<Mark N>: Your Comprehensive Description
Role 3
<Mark Mark 2>... <Mark N>: Your Detailed Analysis
<Mark 1><M 2>... <Mark N>: Your Detailed Analysis
Role
{"question"; YYour created Question), "Answer": [Your created answer]}
("question" [Your created Question), " "Answer": [Your created answer]]
Table 10: The <Role> and <Format> prompt template used for natural images domain.
26

---

questions you craft inquire about one or Published as conference paper ICLR 2025
Role
<Role 1 (Detailed Analysis and Description) > First, you need to deeply understand the page content presented by the entire screenshot, and then thoroughly examine and explain the content and purpose of the highlighted Specify the area is interactive (such clickable button or search bar), and describe the result or action that occurs upon interaction. the highlighted area contains text, analyze whether it hyperlink, of the text and its intended function. To ensure precise and detailed descriptions for the marked regions, please observe the following guidelines. Avoid mentioning the green rectangle or numeric IDD of the marks placed significant. Treat each mark description exclusive to that mark only. minimum 30 words required in your description.
Provide comprehensive description of its features, layout and functionality.
the meaning
the image, they
unique and separate entity, requiring full
<Role 2 (Q&A and Conversations > Dive deeper into the detailed content and intricacies of every marked regions, and interconnections among multiple marked regions. Employ identifiers such as <Region 1>, <Region 2>, etc., specify each area in your inquiries and responses. Assist in formulating question- answer pairs that focus on either a single target area or multiple target areas, aiming to develop rich dialogue dataset (comprising at least Q&A pairs). Ensure more specified <Region>s.
Table 11: The <Role> prompt template used for screenshots domain. The <Format> can be referenced from natural images domain in Tab 10
Role
<Role (Detailed Region Description) > each marked region in the image, provide thorough description using natural language. First, referring to the categories visual objects provided you with, you need to tell me what the marked region is. Then, do your best to describe the content, characteristics, and function of the marked area. If the marked area text paragraph, you should first understand the content of the text, and then summarize the main idea. the marked is image, you need describe the content the image detail, as well as why this image included in document.
To ensure precise and detailed descriptions for the marked regions, please observe the following guidelines. Avoid mentioning the green rectangle or numeric ID of the marks I placed on the image. they are not significant. Treat each mark unique and separate entity, requiring full description exclusive that mark only.
<Role 2 (Q&A and Conversations) Dive deeper into the detailed content and intricacies of every marked regions, and interconnections among multiple marked regions. Employ identifiers such as <Region 1>, <Region 2>, etc., to specify each area in your inquiries and responses. Assist in formulating question-answer pairs that focus either single target dialogue dataset (comprising at least Q&A pairs). Ensure more specified <Rogion>s.
multiple target areas, aiming to develop rich questions you craft inquire about one or
Table 12: The <Role> prompt template used for document domain. The <Format> can be referenced from natural images domain in Tab 10
27

---

Published as conference paper ICLR 2025
Role
<Role 1 (Detailed Region Description) For each marked region in the image, I'd like you give detailed description, as if you examining in person. First refer the OCR results of the specified area provided to clarify text content. Then, delve into the characteristics of text, including the font style and its exact location within the image. I'd also appreciate insights into the background context of the region and an analysis of why the text is present-its intended purpose. Please ensure your explanation is clear and straighfforwar.
ensure precise and detailed descriptions for the marked regions, plcase observe the following guidelines. red polygan or numeric ID the marks placed Treat each mark as a unique and separate entity, requiring a full description exclusive to that mark only. <Role 2 (Q&A and Conversations
Avoid mentioning
image, they are not significant.
Dive deeper into the detailed content and intricacies of every marked regions, and interconnections among multiple marked regions. Employ identifiers such <Region 1>, <Region 2>, etc., to specify each area your inquiries and responses. Assist in formulating question-anae answer pairs that focus on either a single target area or multiple target areas, aiming to develop rich dialogue dataset (comprising at least 4 Q&A pairs). Ensure that the questions you craft inquire about one or more specified <Region>s,
Table 13: The <Role> prompt template used for OCR- OCR-spotting domain. The <Format> can be referenced from natural images domain Tab
Role
<Role (Detailed Region Description) > For each marked region the image, please provide as detailed description as possible using natural language. Carefully observe and analyze the content and details of each panel I have marked. Tell is, including the content of figure and the best you what the content and purpose of each marked area Envision yourself observing each region directly and convey your observations as thoroughly and promptly as possible. Treat each mark as a unique and separate entity, requiring a full description exclusive to that mark only. A minimum of 30 words is required in your description. <Role 2 (Inter-Relatin Relatinshiip Analysis) marked regions. In your analysis. reference specific Delve into and analyze the relationships between the using identifiers like ;Region 1t. ;Region 24, Elucidate the links common features among these areas, which might encompass aspects such as their spatial arrangement, reading order and contextual ties.
<Role 2 (Q&A and Conversations marked regions, and interconnections among multiple marked regions. Employ identifiers such <Region on 1>, <Region 2>, etc., to specify each in your inquiries and responses. Assist in formulating question- answer pairs focus on either a single target area or multiple target areas, aiming develop a rich dialogue dataset (comprising at least 4 Q&A pairs). Ensure that the questions you craft inquire about one or more specified RRegionss.
Dive deeper into the detailed content and intricacies of every
Table 14: The <Role> prompt template used for multi-panels images domain. The <Format> can be referenced from natural images domain in To
28

---

<Mark 2><Mark 3>: The bed <Mark 2> is made up with sheets that likely cover the mattress <Mark 3>, indicating that the mattress is part of the hed setup, essential for providing comfortable sleeping surface. <Mark 3> <Mark 4>: There is direct relationship between the mattress <Mark 3> and the pillow dif- Published as conference paper ICLR 2025
Response
<Role (Short Description)
<Mark 1>: Light brown dog sleeping peacefully bed.
<Mark 2>: A neatly arranged bed with light blue floral patterns.
<Mark 3>: Cream-colored mattress exposed at the edge of the bed.
<Mark 4>: Soft grey pillow partially tucked under sleeping dog's head.
<Role (Detailed Description)>
<Mark 1>: The image displays a light brown adult dog with closed eyes and relaxed ears, lying on its side,
comfortably asleep. The dog appears to be in a serene state of rest, with its head gently resting on a pillow, hinting at the trust and safety feels in its environment.
<Mark 2>: This mark points well-kept bed covered with light blue sheets that exhibit a delicate floral design. The sheets seem smooth and unwrinkled, suggesting bed was recently made well maintained, providing a cozy resting place.
<Mark 3>: An exposed cornrr a cream-colored mattress is visible here, revealing its thick, sturdy build. The mattress looks to be in good condition, with soft upper layer that likely contributes to comfortable sleeping experience.
<Mark 4>: Here, we see a grey pillow with smooth texture. appears to be a thickness that provides ample support for comfort. The pillow is partially tucked under the sleeping dog's head, indicating its use for rest.
<Role (Inter-Relationsi)
Analysis)
<Mark <>>Mkark 2>; These marks indicate relationship of proximity and usage. The dog <Mark 1> resting on the bed <Mark 2>, which implies the bed's purpose household as place relaxation sleep.
<Mark 1><Mark 4>: There is direct interaction here; the dog <Mark 1> haas its head on the pillow <Mark 4>, showcasing the pillow's function support for comfort and the dog's apparent enjoyment of this provision.
<Mark 4> other than they both contribute the overall comfort of the sleeping area. Each
ferent aspect of comfort and support.
<Role (Q&A and Conversations)> ("question": "What color is the dog at <Mark 1>?", "answer", "The dog <Mark 1> is light brown.") ("question": "Describe the design of the bed's sheets at <Mark 2>.", "answer": "The sheets <Mark 2> have a light blue color with floral design.") {" ("question": "What can be said about the condition the mattress at <Mark 3>?", "answer": "The mattress at <Mark 3> appears to be in good condition and sturdy."]
("question": "How does the pillow at <Mark 4> contribute to the dog's comfort?", "answer": "The pillow <Mark provides support and comfort for the dog's head while sleeps.")
Table 15: An example response from GPT-4V during data generation.
29

---

Published as a conference paper ICLR 2025
Please andlyne the relstionship between ofl marked points the image.
Please provide brief description of each marked region the
componentss reglons bakerry kitchen setting, with -Region 1- and <Region 3- being ottine worn by the individaal of (Region 2x, which angoyed in Food preparation.
The ee
Orange and white cat sitting on furniture.
Illnesses
Please provide detailed description <Mark l in the image
Plaae andlyze the relationsh behween all marked regions the image.
The mer ked regional collectively illustrate themes related to social and ersational learning, with a foous on maraging 'ADHD" in children.
person shown with bandaage wropped around the arn, indicating a recent injury or cilment.
Please provide detailed description of-ilegoon in the Image
Plecde provide detailed descr rRegion
image.
The
RRgion text
This appears search of settings menu, typically finding specific settings quickey.
Vhitps hestbbuy,oo
suggesting
hyparliek
Buy
Canada website. It typically interactive and would direct user to that webpage when clicked.
What plrncee do <Mark Is and <Wark together form and what longugg it?
What does the complete signboord suggest the lecation?
SEACREST
The complete textt *SEACREST BEACH suggests that the sigbboord identifying beach named which destination point of Interest visiters.
Together, <Mark 10 and <Mark form the phrose 'SELAMAT Datang' in Malay, which means 'Welcome in English.
Static
BEACH
Please provide detailed description of <Ragion the
Pleasee provide detailed description on of <Reeion Ia in the inage.
image.
This area likely represents shopping cart icon, which is commoniy used on e-commarce websites to indioate where userd can view items they have selected for purchase. If typically interactive, and clicking would take the to
Drrectinns
This region appears to be part of digital map interface) spacifically highlighting section of textt that provides directions. The text is likely part of a list of dinections noute that oon be followed. and presonted that incloctes is interactive. pessibly clickable, navigate the specified location selected.
review their chosen products, adjust quantities, proceed checkout
the
Please provide detailed description of each region in the image.
"What are. the stable phoses depicted <Region 1>?
*Regionn Iy: This region appears to be section- header that reads
The phoses depicted <Mark are ecclor---- BEC phuse (P(S)) and normal phase (N(S)) of paining gas with wide Fock's resonance.
"CONEOLIDA IDSSOLIIDAEDDD BALANCE SHEETS AT DECEMBER 31" This heoder indicates that the section below will conoolidated balance sheets of entity year ending December
the
current year.
Figure 8: Additional visual examples of our VP-SPHINX- 13B across multiple domains.
30


Reviews:
[{"review_id": "TLubhXZwhb", "reviewer": "Reviewer_SHy1", "summary": "The paper introduces a framework that enhances multimodal large language models (MLLMs) with robust visual prompting capabilities. It proposes a novel architecture, VP-MLLM, that integrates a vision encoder, a visual prompt encoder, and an LLM, allowing models to interpret diverse visual prompts like points, bounding boxes, and free-form shapes. Additionally, the authors present the MDVP-Instruct-Data, a large-scale, multi-domain dataset designed to support visual prompting and image-level perception. The experimental results, tested on MDVP-Bench, demonstrate that VP-MLLMs outperform existing methods in pixel-level understanding and multimodal interaction, enhancing MLLMs' capacity for detailed image analysis and spatial reasoning.", "questions": "1. Could the authors provide more information on the data used for alignment pre-training in stage 1? Specific details on the dataset types and the nature of the pre-training tasks would enhance reproducibility.\n2. How does VP-MLLM's performance vary with lower-quality or inconsistent visual prompt data? A discussion on the impact of prompt quality on model accuracy would clarify the robustness of the proposed framework.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  The paper introduces an innovative visual prompt encoder and adapts existing MLLMs for enhanced visual prompting, enabling user-friendly interactions that incorporate spatial and region-specific cues.\n2. The experimental results are comprehensive, covering a wide range of tasks and benchmarks. The performance on MDVP-Bench demonstrates the effectiveness of the proposed approach in terms of pixel-level understanding and multimodal interaction.\n3. The paper is generally well-organized, with a logical flow from the problem statement to the methodology and results. The figures and tables support understanding.", "weaknesses": "1. The two-stage training strategy is presented as a key component, but details on the alignment stage (stage 1) are sparse. Additional clarification regarding the pre-training tasks and specific data used in this phase would enhance reproducibility.\n2. While the paper presents ablation studies, there is limited insight into the specific impact of the visual prompt encoder's internal mechanisms. A more granular ablation of this component would clarify its effectiveness relative to simpler alternatives.\n3. The model’s reliance on the quality of visual prompt data, particularly the GPT-4V-constructed dataset, raises concerns about its robustness across datasets with varying annotation quality. An analysis of performance variation based on prompt quality would be beneficial.\n4. Although the framework is promising, the potential computational cost associated with handling multiple visual prompts and spatial references is not discussed. Assessing the model's performance in terms of processing time and scalability would address practical implementation concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sYXIMQsTQq", "reviewer": "Reviewer_Wihr", "summary": "The paper proposes a framework for enhancing Multimodal Large Language Models (MLLMs) by integrating visual prompting capabilities. The framework allows MLLMs to interact through visual and language-based prompts, such as points, bounding boxes, and free-form shapes, enabling users to guide the model’s focus to specific image regions. Key components include a Visual Prompting MLLM (VP-MLLM) architecture, which combines an image encoder, a visual prompt encoder, and a language model. It is supported by a large multi-domain dataset, MDVP-Instruct-Data, and evaluated on a benchmark, MDVP-Bench, designed to test visual prompt comprehension. The framework shows superior performance in multimodal interaction and fine-grained visual understanding tasks.", "questions": "How easily can the Draw-and-Understand framework be adapted to other MLLMs? Are there specific architectural features required in the MLLM for seamless integration?\n\nThe paper mentions that box prompts sometimes underperform compared to point prompts. Could you explain why this discrepancy occurs?\n\nCould you provide information about the hardware used for your experiments? Knowing the specifications would help in understanding the computational requirements for reproducing your results.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The integration of visual prompting with MLLMs addresses limitations in user interactivity and allows nuanced image region referencing, which enhances model usability. By enabling point-based, box-based, and free-form prompts, the framework gives users greater flexibility in interacting with models, thus expanding the application scope in real-world tasks. I like the idea of interaction between human and models.\n\nThe MDVP-Instruct-Data and MDVP-Bench provide a rich variety of images and tasks, which improve the model’s versatility across different domains and promote robust model evaluation.\n\nThe VP-MLLM framework is adaptable to existing MLLMs, enabling straightforward integration of visual prompts across different models with minimal disruption to their original capabilities", "weaknesses": "The paper provides limited information on key implementation specifics, particularly regarding model parameter settings and data preprocessing steps. This lack of detail may impact reproducibility, making it challenging for other researchers to achieve consistent results in similar settings.\n\nThe framework's reliance on pre-trained models may limit its performance based on the initial capabilities of the base MLLMs, potentially constraining generalization across drastically different datasets or new models.\n\nThe two-stage training process and reliance on large datasets are resource-intensive, requiring considerable computational power for alignment and fine-tuning, which may not be accessible to all practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jAboRERcXZ", "reviewer": "Reviewer_UN1Y", "summary": "The manuscript aims at improving the multi-modal LLMs’ capability of understanding user input and generating corresponding responses. \nFor this new task, the authors have curated a training dataset and evaluation benchmark based on open-source datasets. \nFor the model to be capable of handling the visual prompt input from the users, an architecture of VP-MLLM is proposed, which mainly includes an additional visual prompt encoder based on existing MLLM architecture.\nThe curated training data and the model architecture allows MLLM to generate responses based on point and box inputs. \nThe evaluation of VP-MLLM shows notable improvements on the capability of classifying and describing image contents based on specified image regions, compared to existing MLLMs. \nAblation studies demonstrate the effectiveness of the proposed visual prompt encoder and the training strategy.", "questions": "- How does different design choices affect the general capabilities of the MLLM in Table 4? \n- According to Sec 3, visual prompt encoder only supports points and bounding boxes. How does it accept free-form inputs as in Table 1?\n- Is there any relations between the <region 1> token and the first token given to the visual prompt encoder?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors have curated a training dataset as well as its corresponding evaluation benchmark for the visual prompt understanding task, which would be valuable to future research on the task. \n- A new architecture for performing the visual prompt understanding task is proposed, which is shown to be effective in the experiments. \n- Compared to existing approaches such as Ferret, the proposed VP-MLLM seems to be better at responding to the questions based on user inputs.", "weaknesses": "- It seems that the proposed approach have negative impacts on the general ability of MLLMs, as shown in Table 2. \n- Lack of comparison to important baselines. It seems that the VP-LLaVA-8B introduces limited improvement over ViP-LLaVA-Base-7B (in Table 7). But the comparison is only drawn on the VCR dataset. I would be interested to see more detailed comparisons between these two approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)