List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs
Authors: An Yan
Zhengyuan Yang
Junda Wu
Wanrong Zhu
Jianwei Yang
Linjie Li
Kevin Lin
Jianfeng Wang
Julian McAuley
Jianfeng Gao
Lijuan Wang
Venue: COLM (2024)
Keywords: multimodal LLMs, Synthetic Data, Learning Paradigm
TL;DR: A new data source and learning paradigm for training multimodal LLMs

Abstract: Set-of-Mark (SoM) Prompting unleashes the visual grounding capability of GPT-4V, by enabling the model to associate visual objects with tags inserted on the image. These tags, marked with alphanumerics, can be indexed via text tokens for easy reference. Despite the extraordinary performance from GPT-4V, we observe that other Multimodal Large Language Models (MLLMs) struggle to understand these visual tags. To promote the learning of SoM prompting for open-source models, we propose a new learning paradigm: list items one by one, which asks the model to enumerate and describe all visual tags placed on the image following the alphanumeric order of tags. By integrating our synthetic dataset with other visual instruction tuning datasets, we are able to equip existing MLLMs with the SoM prompting ability. Furthermore, we evaluate our finetuned SoM models on seven MLLM benchmarks. We find that this new dataset, even in a relatively small size (10k-30k images with tags), significantly enhances visual reasoning capabilities and reduces hallucinations for MLLMs. Perhaps surprisingly, these improvements persist even when the visual tags are omitted from input images during inference. This suggests the potential of ``list items one by one'' as a new paradigm for training MLLMs, which strengthens the object-text alignment through the use of visual tags in the training stage. Finally, we conduct analyses by probing trained models to understand the working mechanism of SoM. Our code and data are available at https://github.com/zzxslp/SoM-LLaVA.

Published as a conference paper at COLM 2024
List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs
An Yan Zhengyuan Yang Junda Wu Wanrong Zhu Z Jianwei Yang Linjie Li
Kevin Lin R Jianfeng Wang Julian McAuley Jianfeng Gao Lijuan Wang*
UC San Diego UC Santa Barbara Microsoff
juw06,, (mcculle)@@usse edu, wanrongzhu@ucsb.cm edu, [zhengyang, j jianwei. yang, keli, 1indsey. 1i, jianff, jfgao, juanw jfa.aaaaaaaaaanee/coooooo....... @emicrosoft.com
Abstract
Set-of-Mark (SoM) Prompting unleashes the visual grounding capability of GPT-4V, by enabling the model to associate visual objects with tags inserted on the image. These tags, marked with alphanumerics, can be indexed via text tokens for easy reference. Despite the extraordinary performance from GPT-4V, we observe that other Multimodal Large Language Models (MLLMs) struggle to understand these visual tags. To promote the learning of SoM prompting for open-source models, we propose new learning paradigm: "list items one by one," which asks the model to enumerate and describe all visual tags placed on the image following the alphanumeric order of tags. By integrating our synthetic dataset with other visual instruc- tion tuning datasets, we are able to equip existing MLLMs with the SoM prompting ability. Furthermore, we evaluate
finetuned SoM models on seven MLLM benchmarks. We find that this new dataset, even a relatively small size (10k-30k images with tags), significantly enhances vi- sual reasoning capabilities and reduces hallucinations for MLLMs. Perhaps surprisingly, these improvements persist even when the visual tags are omitted from input images during inference. This suggests the potential of "list items one by one" as a new paradigm for training MLLMs, which strengthens the object-text aligmment through the use of visual tags in the training stage. Finally, we conduct analyses by probing trained models to understand the working mechanism of SoM. Our code and data are available at https: //gt gi thub. com/zzxslppSoM--Laaa
Introduction
Recent advances in Multimodal Large Language Models (MLLMs) such as GPT-4V (OpenAI, 2023a) show strong performance in multimodal perception and reasoning, enabling various new capabilities (Yang et al., 2023b). Among these, Set-of-Mark Prompting (SoM) (Yang et al., 2023a) is an interesting new working mode that enhances the connection between visual objects and textual tokens via visual prompting, i.e., placing alphanumeric tags on input images. It provides natural interface for human-computer interaction, by linking visual locations to executable actions through visual tags, and enables various applications such as GUI navigation (Yan et al., 2023b) and robot interaction (Lin et al., 2023a). Furthermore, GPT- 4V with SoM (Yang et al., 2023a) can implicitly align visual objects with their corresponding tags. Such alignments (Li et al., 2020; Yang et al., 2021) allow MILMS to leverage index numbers to perform multi-hop visual reasoning (Yang et al., 2023a; Wei et al., 2022), thereby improving their abilities in multimodal understanding and reasoning tasks.
Despite the significant interest in SoM prompting and its broad applications, it remains unclear why GPT-4V can benefit from SoM prompting, We find that other MLLMs, includ- ing the ofte-of-thheart open-sourced models such as LLa LLaVA-v1.. (Liu et al., 2024), and commercial systems like Gemini (Team et al., 2023), struggle to understand SoM prompts. This gap prevents them from leveraging the effectiveness of SoM prompting. In this study,

---

Published as a conference paper at COLM 2024
Input: Image
Input: Image + SoM
B
Conversation
Conversation
items there near Marshal speaker?
What items are there near the Marshal speaker?
LLaVA-1.5
There laptop and near the Marshall speaker.
SoM-LLaVA
There is laptop tagged with number and a notebook tagged with number 8.
User
To move the speaker closer to the curtain while keeping laptop the position, which item should swap with?
User
To move the speaker closer to the curtain while keeping the laptop position, which item should You can swap it with the lamp tagged with number 9.
current
with?
SoM-LLaVA
LLaVA-1.5
You should swap the laptop with the cup
Figure Example conversations from LLaVA and SoM-LLaVA (LLaVA with SoM ability) to demonstrate the effectiveness of our paradigm. Left: Standard prompting on LLaVA-1.5, which fails to correctly answer the questions. Right: Set-of-Mark prompting on SoM-LLaVA. Simply placing tags on the input image can improve visual reasoning of Multimodal LLMs.
we aim to deepen the understanding of SoM, with goal of facilitating arbitrary MLLMs to benefit from it.
We break down SoM prompting into three core capabilities: (1) the ability to identify all tags and read the alphanumeric scene texts written on them; (2) the ability to recognize and pinpoint all objects in an image; (3) the ability to associate tags with corresponding objects the image. Despite possessing skills such as OCR and visual recognition to meet the first two capabilities, most ML.LMs still fail to fully understand SoM prompts. Therefore, we hypothesize that the crucial missing element the third capability, associating tags with objects, which requires deliberate training. We further validate that SoM-style data are sparse in common MLLM training sources, and it may necessary to create a specific dataset.
To facilitate such training, we introduce a new learning paradigm named "list items one by one". We show that by asking MLLMs to comprehensively list all tagged items following the alphanumeric order of visual tags, MLLMs can learn SoM prompting with a small number of itemmlisting samples. Specifically, we create tailored synthetic dataset, by tagging images with Semantic-SANN (Li et al., 2023c; Yang et al., 2023a), and prompting GPT-4V to generate paired text descriptions. With just 10k image-text pairs, MLLMs like LLaVA-1.5 (Liu al., 2023a) can reliably understand SoM tags. Based on this initial finding, we conduct studies to explore the effective recipes to help MLLMs best utilize SoM prompting.
We enhanced MLLMs with this "list items one by one" objective and assess their SoM performance from two aspects: model's ability to recognize and describe the SoM tags, and its ability to use SoM in improving multimodal reasoning Figuur 1). For the first aspect, we design the tag listing task, which requires MILMss list and describe all tags in the image, evaluated by listing accuracy. For the second aspect, we evaluate finetuned models on seven MLLM benchmarks, including GQA, POPE, MME, SEED-Bench, VABnech, MM-V Vet and MMBench, showcasing that MLLMs with SoM can significantly boost the multmodal understanding performance. Moreover, our model trained with SoM data outperforms the original MLLM, even without additional visual tags during inference. This demonstrates the potential of incorporating our proposed dataset and learning paradigm to boost general MLLMM training.
2

---

Published as a conference paper at COLM 2024
Finally, we revisit our original question regarding the working mechanism of SoM. The preliminary hypothesis is that the SoM capability may be related to OCR and the implicit association among text, tags, and objects. With our trained models, specifically SoM-LLaVA, we gain access to model features and attention maps for an in-depth analysis. We visualize the attention map to verify tag association. Compared with the original LLaVA model, SoM-LLaVA indeed learns better visual-tag-text associations, reflected in corresponding attention maps.
Our contributions are summarized as follows:
We present new training task and synthetic data source named "list items one by one", which effectively bootstraps MLLMs for the SoM visual prompting ability.
We evaluate our finetuned models on general MLLM benchmarks, and show im- proved performance even when SoM tags are removed from the input image.
We probe the working mechanism of SoM through the trained MLLMs, showcasing the implicit association between visual objects and text tokens when performing SoM prompting.
2 Related Work
Visual referring prompting. Other than text prompts, visual referring prompting (Yang et al., 2023b) is another effective approach when interacting with multimodal LLMs, where users directly draw on input images to specify their intent, such as drawing visual pointers or handwriting scene texts. Early studies show that vision-language models can understand visual pointers such as circles (Shtedritski et al., 2023) and dots (Mani et al., 2020). Recent studies (Yang et al., 2023b) show that more powerful multimodal LLMs (OpenAI, 2023a) can handle more complicated prompts such as arrows, boxes, circles, hand drawing, scene text, as well their combinations. Another major advancement is Set-of-Mark Prompting (SoM) (Yang et al., 2023a), where numbered tags can be placed on images to associate visual objects with text indexed. Its effective visual grounding capability (Kazemzadeh et al., 2014; Yu et al., 2016; Mao et al., 2016) enables various applications (Yan et al., 2023b; Zhang et al., 2023). In this work, we aim better understand SoM and extend its success from GPT-4V (OpenAI, 2023a) to other open-source multimodal LLMs.
Multimodal LLMs. Multimodal LLMs (Alayrac et al., 2022; Zhu et al., 2022; Open 2023a; Liu et al., 2023b; Li et al., 2023b; Xue et al., 2024) extend large language models (OpenAT, 2023b; Gao et al., 2023; Touvron et al., 2023) with visual perception capabilities. Recent studies (Chen et al., 2023) show the effectiveness of training open-source models on the GPT-4V generated detailed description data. Another thread of studies explore having multimoaal LLMs predicting object locations as bounding boxes (Wang et al., 2023b; Peng et al., 2023) or masks (Rasheed et al., 2023). In contrast to most prior studies that pair the images with different text instructions, our study explores a new direction of how visual prompts such as SoM can improve multimodal LLMs. Specifically, we show that the SoM visual tags provide fine-grained alignments between visual objects and text tokens, thereby improving various visual reasoning tasks, both with and without SoM prompting during inference.
3 Preliminary Examination
3.1 Visualizing SoM Prompting on LLavA
In this section, we first investigate the capacity of L.L in SoM, concerning its attention sensibility to the numeric IDs tagged on the objects and its answer to the SoM query. We show example task to list a series of objects tagged with numeric IDs in Figure 2, in which the attention map is extracted from LLaVA-1.5 based on the SoM query (e.g., "I have labeled bright numeric ID at the center for each visual object in the image. Please enumerate their names.") "). The top 20 image patches with the highest average attention weights across the user query tokens are highlighted in transparent red regions.
3

---

Published as a conference paper at COLM 2024
LLaVA-1.5 1.5 Attention Map of Tags
LLaVA-1.5 Attention Map of Tags
User: have labeled ID
center each visual object in the image
User: have laheled bright numeric in
at the center for each visual object the their
Please enumerate their
image, Please
LLaYA-1 AAV.A-1..
A-15:: 1.1:5: the inanee there is brows black perched a brunch surounded by leaves. The loaves surounnding the hird visible.
L vase with a flower in it.
2. white flower in the
3. vase an wooden
4. The vase is simingg on ladge.
5. The vase is sitting
ferce,
is sitting milling.
7. cn 8, c.. ... 10, U....
Two examples of SoM prompting in LLaVA-1.5. Left: Attention map extracted from VA-1.- the image of bird perching on branch, where objects are tagged. Attention map extracted from LLaVA-1.5 on the image of a vase placed on a table, where 7 objects are tagged. However, LLa VA-1.5 lists more than 7 object names that are repetitions of previous object names.
Figure 2:
Right:
Dataset
#Text
Text w/ Listing
Source of Text
CeaVA-Pretrain-Cee Sil.-PrrrinCCCMM-555555
595.4K
0
Raw CC3M image captions.
LL SANNA-Preraain-ICSS-665
558.1K
0
Captioned by BLIP.
LLaVA-v1.5-Mix66%
3356.2K
0.72%
Rule-based or generated by ShareGPT or CPT4-0314.
ShareGPT4V
102.0K
0.21%
Generated by GPT4-Vision.
CogVLM
333.5K
7.16%
Generated by MiniGPT4 or by GPT4-0314.
Table 1: Examined pretraining (1- and instruction-tuning (3-5) datasets in our preliminary study.
We can observe from the highly attended regions of LLaVA-1.5 that the numeric ID tags can be easily and correctly attended by LLaVA-1.5 along with their associated objects (e.g,, bird, vase, and branches). Such capacities in locating numeric ID tags may have been acquired by LLLVA-1.5 from its pretraining tasks in OCR and also benefited from the strong OCR abilities of the ViT feature encoder (Radford et al., 2021) adopted by LLaVA-v1.5. However, the response prompted by the user query in the first example Figure suggests that LLa cannot follow the SoM instruction to list all the items. Instead of providing the object descriptions corresponding to all the numeric ID tags, LLaVA-1.5 responds with general image caption, due large portion of image captioning samples in its pretraining stage. From the second example of Figure 2, we can also observe that although LLaVA-1.5 generates a list of tag IDs with object names, it cannot accurately associate the tags to corresponding objects, causing the model to hallucinate the descriptions of these objects.
3.2 Finding SoM Data in Existing Training Sources
We further look into the pretraining/ pretraiiing/nnstuuciinn-tss /instructioo-tuuinng (IT) dataset, aiming to inspect there are text contents with listings, images with SOM annotations. We examine the pretraining dataset of LLaVA-v1 and v1.5 (Liu et al., 2023b;a), and the IT dataset used by LLaVA-v1.5, ShareGPT4V (Chen et al., 2023), and Cog VLM (Wang al., 2023a).
Table 1 shows the source of text in each dataset and the percentage of text content with listing format. The text in the two pretraining datasets for LLaVA are image captions (either the raw caption or generated by BLIP (Dai et al., 2023)), and we did not find any text with listings in them using our parser. Aside from image captions, the IT dataset also contains instructions related to other visual tasks such as VQA. We noticed that the answers provided by GPT--(V) models sometimes construct the text in listing manner (e.g., list out possible reasons for a question, list out observed objects in the image, etc). More examples can be found in Appendix A.6. The instructionnfollowee highest percentage of text with listings (~7%). Through our interaction with these models, we also find CLgVLM is better at generating listing-style data than LLaVA-1.5.
dataset used by CogVLM has the
We add tags to MSCOCO-2017 images following the SoM (Yang et al., 2023a) format, and train a binary classifier with ViT/B-16 (Dosovitskiy et al., 2020). We use the classifiers to filter the images in the two LLaVA pretraining datasets, and take the top 2k images with the highest scores for each dataset. We then manually check the top 2k images, and found 12 images with tagging in CC3M-595K (~0.002%), and found 86 images with tagging in
4

---

Published as a conference paper at COLM 2024
LCS-558K (~0..15%) Figure 15 shows few images with tagging. Given that tagged images are sparse in those datasets and the SoM prompting performance of open-source MLLMs is unsatisfying, it may be worthwhile to design tailored dataset that empower open-source MLLMs with this emergent ability, similar to what GPT-4V is capable of.
4 Dataset Creation and Training
Motivated by the above analysis, in this section, we introduce the pipeline to create our dataset. First, in Section 4.1, we use Semantic-SAM to generate semantic visual prompts in the form of numeric tags for each image. We then discuss the learning paradigm of "list items one by one" in Section 4.2. Finally, we use visual prompted images to generate text data in Section 4.3.
4.1 Image Source and Visual Prompting Generation
There are various open-source image datasets available (Deng et al., 2009; Lin al., 2014; Schuhmann et al., 2022; Yan et al., 2023a). We use MS-COCO (Lin et al., 2014) as the image source to create our SoM dataset, since it contains comprehensive human annotations with bounding boxes, masks, and captions. It has also been widely used for visual instruction tuning (Liu et al., 2023b; Wang et al., 2023a; Chen et al., 2023), which could benefit controlled experiments as well as comparisons with previous work.
The first step is to create visual prompts by placing numeric tags on proper locations. Following SoM (Yang et al., 2023a), experiment with segmentation models including SEEM (Zou et al., 2023), Semantic-SAM (Li et al., 2023c), and SAM (Kirillov et al., 2023). Empirically, find that Semantic-SAM provides the annotation granularity that best fits COCO images, and thus use to create tagged images for our dataset.
4.2 A Learning Paraaigm:: List Items One by One
After obtaining the image data with semantic tags, the next question is how to design the instruction data to best distill the SoM visual prompting ability. A common approach (Liu et al., 2023b; Chen et al., 2023) in multimodal instruction-folloiing data creation is to de- irrestion-answeriigg
sign and collect ChatGPT/GPT-4 or alternative open-source models. Given an image and optional meta-
style samples. This is often done by prompting
data M2 such as captions, bounding boxes, various questions or instructions x are posed,
and the corresponding answers x!") from large models are collected.
However, such general question-answering data may not be the most effective in distilling the desired SoM prompting capability, due to the inadequate mention of objects in text. For SoM prompting, one core ability of interest is to associate numbered tags with visual objects in the image, thereby enabling effective referral of visual objects via text tokens. In general QA data, however, it is rare for multiple objects to be mentioned, even in an extended multi-turn conversation. To enhance tag association, approach: list items one by one, where the model is asked to comprehensively describe all tagged items within an image. Given an image IJ with N text tags on the image, we ask the
propose a simple and effective
model to enumerate all items in numerical order: (X1 X2 obj objr textual description of the j-th item, tagged by ID in the image.
XN obj where x! is the
Beyond promoting SoM learning, listing items one by one is also effective in general multi- modal LLM training: if model learns to list items in the images with specific order (in our case, the order is determined by the visual numeric tags), it gains a comprehensive and fine-grained understanding of images. This could directly benefit visual grounding and reasoning, which we verified through the standard multimodal QA and chat evaluation benchmarks.
Compared with existing visual instruction tuning datasets, such as LLaVA-665K (Liu et al., 2023a) and ShareGPT-4V (Chen et al., 2023), another difference is the implicit spatial infor-
5

---

Published as a conference paper at COLM 2024
80
Accuracy
Accuracy
list
mis6+
75
list 13B
nistor
GPT-4V
ocstet
70
65
10k
25k
50k
75k
100k
10k
25k
50k
75k
100k
Number listing data (k)
Number listing data (k)
(a) Ablation on model sizes with LLaVA-1.5 (b) Ablation on data sources with LIaVA-1.5-7B
Figure 3: Performance analysis on tag listing. Training samples of listing data grow from 10k to 100k. list++mi--665k is to mix listing data with 665k instruction tuning data from (Liu et al., 2023a). list+nonocr is to exclude the OCR and text data from the full 665k data, resulting in 563k samples. list+ocrtext to mix listing data with only OCR and text data from the full 665k data, resulting in 102k samples. Green-dashed line in Figure 3a is the zero-shot result from GPT--VV.
mation encoded by the visual tags in SoM prompting. Converting images into the language space inevitably loses information, especially spatiallocation. locations. For example, "a girl on the right" can only vaguely imply the posítion of the girl. However, with SoM visual prompting, we provide precise visual guidance on the image. Therefore, our data can be viewed as form of dense captioning with a new way of encoding spatial information.
4.3 Text Data Generation via GPT-4V
With the visual prompting enhanced images, the final step for dataset creation is to generate the corresponding text data. To automate this process, leverage GPT-4V (OpenAT, 2023a) to generate the listing data ( _np ...,, following the order of visual tags in the images. However, we find that simply prompting the model to list items in a zero-shot manner could lead to noisy and biased generation results, where the model may refer the tag to distant object that is easy to describe. (see examples in appendix A.4). To mitigate this problem, we seek two complementary solutions: (1) We modify the system message of GPT-4V to avoid assigning tags to distant objects. (2) We manually design few correct listing samples via human annotations, and use them as seed examples for in-context-learnnng to query GPT-4V. The details of our template is in Appendix.
In addition to listing, we also consider conversational data similar to LLaVA (Liu et al., 2023b), where GPT-4V is asked to generate multi-turn conversations between an Al assistant and person asking questions about the photo. Given tagged image I", we use GPT-4V to generate instruction-following data in the form of (Person:I' x(", Assistant: x(")
4.4 Model Training
We take the pretrained stage of LLaVA-1.5 (Liu et al., 2023a) as the base model, and continue finetuning by mixing instruction tuning data of L.a with our collected visual prompt- ing data. For SoM-listing, we create 40 task templates as human instructions (e.g.. "please enumerate object names in the tagged image"), and treat them as standard conversational data. We use the same training objective of next-token prediction to train general QA, SoM-QA and SoM-listing data. Specifically, we maximize the conditional log likelihood as follows:
log p(X,(X,,, log I Pe (x Xa,eir Xa<i), i=1
(1)
where O are the trainable model parameters, Xg.<i and XA.c are the instruction and answer tokens in all previous turns of conversations before the current prediction token x;. The input image is or 1s for LLaVA SoM data, respectively.

---

Published as a conference paper at COLM 2024
5 Experiments
5.1 Experimental Settings
Experiment overview. We validate the method effectiveness from two aspects. First, in Section 5.2, we benchmark the model's capabilities in understand and describing SoM visual prompting. We design the tag listing task on MS-COCO to test the SoM performance. Second, in Section 5.3 and 5.4, we evaluate if our dataset and model can benefit visual reasoning tasks, where we consider seven representative visual question answering and reasoning tasks detailed as follows.
MLLM benchmarks. We consider the following multimodal LLM benchmarks in Table 2 to validate SoM visual prompting's s benefit on general visual reasoning tasks. GQA (Hudson & Manning, 2019) focused on fine-grained compositional reasoning over real-world images. We compute match accuracy following (Liu et al., 2023a). POPE (Li et al., 2023e) is to evaluate object hallucination in multimodal LLMs. We follow POPE and report the F1 Score for the binary choice questions. MME (Fu et al., 2023) contains 2800 binary choice questions for perception and cognition evaluation. We report the overall perception score for the evaluated models. SEED-1 (Li et al., 2023a) contains 14K multiple choice questions on images. MMBench Liu al. (2023c) is another multi-choice benchmark to evaluate the multi-modal understanding capability of MLLMs. We report the multiple choice accuracy on SEED-I and MMBench dev set. LLaVA-W (LLaVA-Bench In-the-Wild) (Liu al., 2023b) and MM-Vet (Yu et al., 2023) are open-ended generation tasks, which compute the evaluation score by prompting a GPT-4 based evaluator (OpenAI, 2023b) with both the predicted and ground-truth reference answer. The score is then scaled to the range of 0 to 100.
We provide more implementation details in appendix A.1.
5.2 Evaluation on Tag Listing
First, we evaluate model performance on the tag listing task, aiming answer two research questions: (1) Do model sizes matter in terms of learning SoM ability? (2) How will different sets of extra training data impact the SoM performance? We design the listing data based on images with ground-truth mask annotations from MS-COCO, and enumerate each object with corresponding class name. An example list is "1, person, 2. cat, 3. dog.". We compute M list-wise accuracy, where for a caption with N items, the score is N with M items predicted correctly by the model. With human annotation of objects an image, we can automatically create abundant rule-based data (up to 100k) for studying model behaviors and perform quantitative evaluations.
For the first question, we find that larger LLM performs better for the listing task (see Fig- ure 3a), presumably benefiting from the stronger language prior to help learn SoM prompt- ing. For the second question, we decompose the 665k instruction data from LLaVA-1.5 (Liu et al., 2023a) into two parts. We find that both general caption-Qu -ationnQAAaaa data, as well as OCR-text data contribute to learning SoM ability when limited listing data are available (10k). The reason could be that OCR can help with identifying numeric tags, and general caption may help the model to recognize objects within an image, both of them are fundamental abilities required by SoM. In general, other visual instruction data may benefit learning SoM, especially when SoM data is scarce.
Overall, we observe that with only 10k data, we can outperform zero-shot GPT-4V in listing accuracy, whereas growing data size from 50k to 100k only slightly improves the listing performance. These findings suggest that collecting a small amount of data may be sufficient for learning SoM prompting.
5.3 Evaluation on MLLM Benchmarks
We then train LLaVA-1.5 marks. As shown in Table 2, we observe that our SoM-LLaVA-153 which is trained with a mixture of LLaVA visual instructions and our SoM data in order to learn SoM prompting,
collected dataset and perform evaluation on MLLM bench-
7

---

Published as a conference paper at COLM 2024
Method
LLM
Res.
Pre-Data
IT-Data
POPE
MME
SEED-I
LLaVA-W
MM-Vel
BLIP-2
Vicuna-13B
224
129M
85.3
1293.8
49.7
38.1
22.4
InstructBLIP
Vicuna-7B
224
129M
1.2M
58.8
60.9
26.2
InstructBLIP
Vicuna-13B
224
129M
1.2M
78.9
1212.8
58.2
25.6
Fuyu-8B
Fuyu-8B
600
74.1
728.6
21.4
V2
LLaMA2-7B
336
1328.4
35,2
mPLUG-Owl-2
LLMMA2-7B
448
348M
1450.2
64.1
36.2
Qwen-VL
Qwen-7B
448
1.48*
50M!
62.3
Qwen-7B
448
1.48T
50M
1487.5
65.4
SPHINX
MaMA2-7B
224
80.7
1476.1
73.5
36.0
LLaVA-1.5
Vicuna-13B
336
558K
665K
85.9
1531.3
68.2
70.7
35.4
VoM-LLaVA-155
Vicuna-13B
336
558K
695K
86.6
1563.1
69.6
75.3
35.9
SoM-LLaVA-155-T
Vicuna-13B
336
558K
695K
87.0
1572.8
69.5
73.3
37.2
Table 2: Performance comparison on popular MLLM benchmarks. Res., Pre-Data, IT- Data indicate input image resolution, the number of samples in pretraining and instruction tuning stage, respectively. *Includes in-house data that is not publicly accessible. Underlined numbers are the second best results in the column. SoM-LLaVA-1.5-I is the model with tagged images as input.
Method
LLM Size
GQA
POPE
MME
SEED-I
LLaVA-W
MM-Vet
MMBench
LLaVA-1.5
62.0
85.9
1464.0
64.8
63.4
30.5
65.4
SoM-LLaVA-155
62.7
86.5
1507.0
67.0
66.9
33.3
66.5
LLaVA-1.5
13B
63.3
85.9
1531.3
68.2
70.7
35.4
68.9
SoM-LLaVA-1.5
13B
63.8
86.6
1563.1
69.6
75.3
35.9
69.5
Table 3: Performance comparison on LLaVA models. We reproduced results of LLaVA-1.5
on MME and MMBench-dew with the official checkpoints and evaluation scripts.
also obtains superior performance on general MLLM tasks. Surprisingly, we find that even without tagged images, SoM-LLaVA still attains strong performance and substantial im- provement over the orignal LLaVA. This indicates the quality our data and the potential of introducing listing data into general MLLM training to improve visual understanding and reasoning, as well as reduce hallucinations. We conjecture the reason that the great per- formance of SoM-LLaVe
non-tagged images is that "listing items one by one" with visual prompting guides the model to learn fine-grained semantics for image features. Related case studies and visualizations are in appendix A.2. For the performance of open-vocabulary listing, we present examples in appendix A.3.
5.4 Additional Evaluation Results on LLaVA
We evaluate LLaVA and SoM-LLaVA models with 7B and 13B LLM backbones, to fur- ther demonstrate the effectiveness of adding SoM data into the instruction tuning stage. As shown in Table 3, we observed performance improvements across wide range of benchmarks testing multimodal understanding and reasoning capabilities. These gains are consistent on both model architectures.
5.5 Ablation Study on Mixture of Datasets
Finally, we perform ablation on different data mixture strategies in Table 4. We consider mixing our listing and QA data generated from Section 4.3 with LLaVA-665k (Liu al., 2023a), trained separately or together. Empirically, we find that mixing listing and QA data yields the best overall performance. of listing. Here we also notice that "listing item one by one" can in turn greatly improve the performance of OCR related task. The results on POPE indicates our data leads to lower hallucinations compared with ShareGPT-4V, which is dense caption dataset without visual prompting. Placing tags on the images can seamlessly encode spatial information into the data for MILMMs to learn fine-grained vision language alignment.
Section 5.2, we find OCR data can help the learning
8

---

Published as a conference paper at COLM 2024
POPE popular
MME
SEED-1
Data Composition
Data Size
random
adversarial
OCR
overall
overall
LLaVA-IT
665K
87.1
84.5
125.0
1531.3
68.2
LLaVA-IT Listing
665K
87.3
84.8
147.5
68.9
LLaVA-IT QA
695K
20k
87.5
86.4
84.7
110.0
1540.0
69.2
LLaVA-II Listing QA
695K 30k
87.8
86.7
85.2
140.0
1563.1
69.6
LLaVA-II ShareCPT-4V
695K 20k
87.1
86.0
84.3
110.0
1528.7
69.3
Table 4: Comparison for different data mixture strategies. LLaVA-IT is the mix665k data from (Liu et al., 2023a). Listing and QA is from SoM dataset with tagged image-text pairs. ShareGPT-4V is from (Chen et al., 2023) with the same MS-COCO images as our 2k QA data and detailed captions from GPT-4V. Results are evaluated on the 13B LLM.
6
Analysis
SoM-LLaVA-1.5
1. Laptop
2. Chair
3. Monitor
4. Desk Lamp
5. Printer
LLaVA-1.5
Laptop
Chair
3. Monitor
Desk Lamp
Printer
Figure 4: A comparative example of attention maps extracted from LLa and SoM- LL.,5,, where five objects (e.g,, laptop, chair, monitor, desk lamp, and printer) are tagged. We highlight the top-5 most attended image patches of the models on each object's numeric tags individually. SoM-LLaVA is better at attending to objects following numeric text and tags.
6.1 Probing Trained Models
We first analyze the tag-listing capacity of SoM-LLaVA-15 (13B) acquired through fine- tuning. In Figure 4, we show the attention maps on the five tagged objects, which are extracted from SoM-LIaVA-1. and LLLVA-1.5 respectively. The comparative example showcases that although both models can locate their model attention on the mentioned objects to some extent, the fine-tuned LAM-LLaVA-1.5 model can attend and focus on characteristic regions of the object, which can also be accurately guided by the numeric ID tags. For example, the comparative attention maps on the object "Laptop" tagged with number show that SoM-LLaVA-1.5 can clearly attend to the mentioned object with its main focus. In contrast, LLaVA-1.5 mistakenly attends to the monitor instead of the laptop, due to high similarity between these two objects.
In addition, we also observe that SoM-LLaVA-1.. can be efficiently guided by the numeric ID tags to focus on the specific object the user refers to, even with multiple similar objects within the image. For example, the attention map of SoM-LLaVA-1.5 on the "Chair" tagged with number 2 is mostly focusing on the chair on the left-hand side, instead of the similar chair the right-hand side. SoM prompting in SoM-LLaVA-1.5 with such the capacity to accurately locate the tagged object, enables more flexible and easier user-referring queries without complicated language descriptions. The attention maps also verify our early hypothesis regarding the implicit association among the text, tag, and object in SoM prompting.

---

Published as a conference paper at COLM 2024
Case Study
User: Whose pants' color the same as someone clae's white shirt?
LLaVA-1.5:
man wearing brown pants the same pants color woman wearing white shirt.
AA--15:
The lugged with number wearing that the sume color as the person tagged with number 9% white shirt.
0
20
GPT-4V:
The person labeled with the number is wearing white pants, and their color matches the white shirt the person labeled with the number 9,
Figure 5: An example comparison for LLaVA, SoM-LLaVA and GPT-4V.
Case Study
User:
many objects are type object tagged
LLaVA-1.5:
There objects of the type the ragged
16
SoM-LIaVA-1.5:
There three objects of the type the object tagged as 13, which an apple.
GPT-4V:
The object lagged as number is an apple. this image. there are three other apples, tagged with numbers total of four apples, including the one tagged as 13.
So, there
Figure 6: An example comparison for LLaVA, SoM-LLaVA and GPT-4V.
6.2 Visual Reasoning with SoM Prompting
In this section, we present two examples of different models reasoning over the tagged images, aiming to show the potential of visual reasoning over tags, with model that can understand SoM prompting.
In Figure 5, we examine a multi-step visual reasoning question (i.e., "Whose pants' color is the same as someone else's white shirt"), which requires the MLLM to first identify the mentioned objects (i.e,, all the pants in the image and the white shirt), then compare their visual features (i.e., the same white color) to answer the question. From Figure 5, we observe that VA. provides white shirt as a female. This error can be caused by the inferior object recognition capacity in VA-1.,, or the complexity of the multi-hop question. As for GPT-4V, it made a mistake by assigning tag-2 to the person on its right, leading to a wrong reasoning process and contrast, SoM-LLaVA-113 successfully identifies tags and 9 with the same color in those image regions, while recognizing the two objects as white pants and white shirt, respectively. We show another example Figure 6.
incorrect answer by falsely identifying the person wearing the
incorrect conclusion.
Conclusion
In this paper, we start with SoM prompting and propose new learning paradigm for multimodal LLM training. We show that ML..Ms can learn SoM prompting using a small set of synthetic data by listing items one by one. Moreover, we explore the broader impact and find our dataset can benefit general capabilities for MLLMs, where our enhanced model, SoM-LLaVA, consistently outperforms the original LLaVA model across seven multimodal benchmarks. Our dataset and models are released to facilitate vision and language research.
Overall, we hope this work could inspire future research exploring new learning paradigms and data recipe for MLLM training, as well as ways to create better synthetic datasets in the era of I.LMs.
10

---

Published as a conference paper at COLM 2024
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-237533 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint Sali::300.129666 2023.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: improving large multi-modal models with better captions. arv preprint ar Xiv:2311.12299. 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open- source chatbot impressing gpt-4 with 90%* * chatgpt quality. See https://oicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose Vision-language models with instruction tuning. arXiv preprint ar arXiv:2305.06500 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. ArXiv, abs/2010.11929, 2020. URL https:
//api. semamnicccccccholarr .are//aarrrr//////ee wttg//oorosI::2255555 225039882.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arviv preprint ar arXiv:2306.1339 2023.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. Xiv preprint 2tt::2304.10000 2023.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg, Referitgame: Refer- ring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint ar arXiv:2304.02643 2023.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal lims with generative comprehension. ar preprint arXio:2307.16125, 2023a.
Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, and Jianfeng Gao. Multimodal foundation models: From specialists to general-purpot assistants. arxiv preprint ar Xiv:2330. 10020, 2023b.
Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and recognize anything at any granularity. aixiv preprint arXit:2307.04777, 2023c.
11

---

Published as a conference paper at COLM 2024
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. ar Xio preprint wMXi:2231.11297, 2023d.
Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, 2020.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXto:2305.10355, 2023e.
Kevin Lin, Faisal Ahmed, Linjie Li, Chung-Ching Lin, Ehsan Azarnasab, Zhengyuan Yang, Jianfeng Wang, Lin Liang, Zicheng Liu, Yumao Lu, et al. Mm-vid: Advancing video understanding with gpt-4v (ision). ar Xiv preprint arXiv:2310.19773, 2023a.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint ar XrXt::2311.07575 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://llawa-vl.e gi thub. htt.:ooo/bb///44//0// -10--011--1-------00 nee-ltava-next.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: is your multi-modal model an all-around player? arXiv preprint ar Xrv:22307 006211, 2023c.
Arjun Mani, Nobline Yoo, Will Hinthorn, and Olga Russakovsky. Point and ask: Incorporat- ing pointing into visual question answering. arviv preprint xxX::2201..1668, 2020.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016.
OpenAI. Gpt-4v(ision) system card. 2023a. URL httpss:/ccc..opeaaa..oom com cowppapers//PPT System_Card. pdf.
OpenAL. Gpt-4 technical report, 2023b.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. Xiv preprint arXiv:2306. 14824, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning trans- ferable visual models from natural language supervision. arXiv preprint ar arXiv:2103.00022 2021.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, AbdeIrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming- Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. ar Xio preprint 23::::2311 033355, 2023.
12

---

Published as a conference paper at COLM 2024
Christoph Schuhmann, Romain Beaumont, Cade W Gordon, Ross Wightman, Theo Coombes, Aarush Katta, Clayton Mullis, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, et al. Laion-5b: An open large-scale dataset for training next genera- tion image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchnarks Track, 2022.
Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about red circle? visual prompt engineering for vlms. arv preprint ar Xio:2304.06712, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-aptiisee Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. ar preprint ar Xrx:22312.11805, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arx preprint arXiv:2307 0.99888,.. 2023.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. Xiu preprint ar arXiv:2311.03075 2023a.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint ar Xix:22055.11175 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint ar Xiv: 12220.1119933. 11903, 2022.
Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael $ Ryoo, Shrikant Kendre, Jieyu Zhang, Can Qin, Shu Zhang, Chia-Chih Chen, Yiming Yu, Juntao Tan, Tulika Manoj Awalgamkar, Shelby Heinecke, Huan Wang, Yejin Choi, Ludwig Schmidt, Zeyuan Chen, Silvio Savarese, Juan Carlos Niebles, Caiming Xiong, and Ran Xu. xgen-mm (blip-3): A family of open large multimodal models. ar arXiv preprint ar Xrio:24088.08872 2024.
An Yan, Zhankui He, Jiacheng Li, Tianyang Zhang, and Julian McAuley. Personalized showcases: Generating multi-modal explanations for recommendations. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2251-2255, 2023a.
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. Gpt-4v in wonderland: Large multi- modal models for zero-shot smartphone gui navigation. arXiv preprint ar 2110023 2023b.
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set- of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arX1::2300.11441, 2023a.
Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo. Tap: Text-aware pre-training for text-vqa and text- caption. In CVPR, pp. 8751-8761, 2021.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arxiv preprint ar Xii::2309.17422 2023b.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint ar arXio:300..14178 2023.
13

---

Published as a conference paper at COLM 2024
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In ECCV, 2016.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. Xrxiv preprint ar Xii::3308.02449, 2023.
Jiangning Zhang, Xuhai Chen, Zhucun Xue, Yabiao Wang, Chengjie Wang, and Yong Liu. Exploring grounding potential of vqa-oriented gpt-4v for zero-shot anomaly detection. arXiv preprint ar arXiv22311.02611 2023.
Wanrong Zhu, An Yan, Yujie Lu, Wenda Xu, Xin Eric Wang, Miguel Eckstein, and William Yang Wang. Visualize before you write: Imagination-guided open-ended text generation. Xiv preprint arXi::2210.03765, 2022.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Xiv preprint arXiv:2304.06 067118 2023.
A Appendix
A.1
Implementation details.
The LLaVA-1.5 model contains a CLIP-VIT-L-336px visual encoder (Radford et al., 2021) and Vicuna-7/133 1/3ua language model (Chiang et al., 2023), connected by an MLP projection layer. Our main experiments are conducted on 8X and 4X 80GB A100 GPUs for Ilava-13b and llava-7b models, with a batch size of 128 and 64, respectively. We train all models for epoch, following hyperparameter setting in (Liu et al., 2023a).
We collected 10k SoM-listing data and 20k SoM-QA data using GPT-4V turbo. For visual tagging, we use the level-2 granularity of Semantic SAM to annotate all images from MS- COCO, to learn fine-grained object-text alignment. During inference, we find that the existing MLLM benchmarks mostly consist of high-level questions about an image, and level-1 annotation with fewer tags works better.
We report results of following MLLMs on public benchmarks: BLIP-2 (Li et al., 2023d), InstructBLIP (Dai et al., 2023), Fuyu-8B LLLMA-Adapter-V V2 (Gao et al., 2023), mPLUG- Owl-2 (Ye et al., 2023), Qwen-VL (Bai et al., 2023), SPHINX (Lin et al., 2023b), and LLaVA- 1.5 (Liu et al., 2023a).
A.2 Comparison Results on Reasoning on Images without Tags
We additionally analyze how LLaVA-1.5 and SoM-LLaVA-115 perform differently when images with no tags are provided. In Figure 7 and Figure we can observe that the discrepancies between the attention maps extracted from the two models in both cases are relatively insignificant. Such observation suggests that LLaVA-1.5 has pre-trained with good multimodal cross-attention that enables the MLLM to capture the most characteristic visual features in the images. However, due to the lack of alignment between visual semantics and textual semantics, MLLMSs like LLL may not correctly associate textual information with relevant visual evidence, which further causes incorrect answers in visual reasoning. With SoM fine-tuning, we reinforce the MLLM's visual understanding of specific objects in the image by asking the model to list objects one by one. By bridging the objects' visual features and their semantic meanings, the MLLM can better refer to the visual objects and answer the questions with more accurate object descriptions.
A.3 List Items One by One with SoM-LLaVa and GPT-4V
We present the open vocabulary listing results with our SoM-LLaVA and GPT-4V. As shown in Figure 9 and 10, our model is able to generate accurate descriptions of each tagged object, which learned the implicit tag-object association on images.
htttps://www. adept. ai wwtppt//uboboooouu-/eeeee
14

---

Published as a conference paper at COLM 2024
LLaVA-1.5 Attention Map of Query
SoM-LLaVA-1.5 Attention Map of Query
User: How many sheep there the image?
User: How many sheep are thece in the image?
LLaVA-I.S 1.5: There are sheep the
LLaVA-I-S two sheep in image.
Figure 7: Attention map and visual question-answering comparative results from LLaVA-1.5 and SoM-LLaVA-1.5.
LLaVA-1 1.5 Attention Map Query
SoM- BoM-LLaVA-1.3. Attention Map of Query
User:What color the bird lest the red bird?
User: colar bird left the red hird?
LLaVA-L. The bird left the red bird gray.
LLLVV--1.S The left to the rod bird yellow.
Figure 8: Attention map and visual question-answering comparative results from LLaVA-1.5 and SoM-LLaVA-1.5
Listing Comparison
Eis all the tagged objects in the image.
Liet all the tagged objects in the image.
SuM-LLaVA,
GPT-4V:
lying
cat lying
carpot.
2. cat standing and looking at the Ielevisium
2. Another eat standing on its hind legs,
appearing internet with
displaying image.
compunmeents.
4. small selevision or mmnittr on stand.
4. A small picoure frame on a table to the left of the TV stand
shelving
|mpty shelves, within television stand.
wooden table stand.
6. A woeden table the right, purtially visible.
7. The sereen of the selevasion displaying
7. television displaying image, possibly
from video
show.
bolding
small relovision moniter.
TV stand, holding the picture frome.
Figure 9: Open vocabulary listing with our model (SoM-LLaVA-1.5) and GPT-4V.
Listing Comparison
User:
Userrt
List the tagged objects the image.
List all the teggeed objects in the image.
SoM-l I A-VA-1..
GPT-4V:
are togged objects image.
the image, we see following lugged objects:
banana).
lying borizuntailyy
GUINNES
another bununal.
Soreground.
is on bottle of Dragon Stout beer,
edge of weodes surfhee
bottle of Fareign Extra Guinness
Dmagon
beer,
hottle Guinness Foreign Extm Stout the center.
is on a bottle of Dragon Stout beer.
Another bottle Deagon Stout the right.
Figure 10: Open vocabulary listing with our model (SoM-LLaVA-1.5) and GPT-4V.
A.4 GPT-4V listings with Different Prompting Methods
We present the listing results from GPT-4V with different prompting methods, as shown in Table 5 and Table 6. 2-shot in-context learning leads to more accurate listings.
15

---

Published as a conference paper at COLM 2024
Listing example from GPT-4V, woman by the water.
GPT-4 zero-shot
TTag 1 is on the water of a lake or river. -aa 2 is on boat. Tag 3 is on a swimming platform or dock in the water. -Tag 4 is on the land or shore. TTag 5 is informational sign notice board. Tag is on tree. Tag 7 is on pink parasol or umbrella. Tag is bathing suit. Tag 9 placed where the blurred face is, describing item.
GPT-4V improved sys-msg
Here are the items tagged in the image: 1. Blue water, possibly lake river. 2. The lower part of white, vertical pole, likely part of the structure tagged with 5. 3. Green foliage trees in the distance across the water. The edge of large white sign on a poll, with some text visible but not readable. lifeguard stand or chair with blue accents and a sign attached. Blue sky. 7. Blue sky with clouds. 8. The midsection of a person wearing a floral swimsuit. pink umbrella, held by the person whose face is blurred. The face the person holding the umbrella is blurred for privacy reasons.
white
GPT-4V 2-shot learning
tagged items the image: The bottom edge body water, likely lake or river given the surrounding context. 2. Lower arm and hand a person holding an umbrella. The person's hand is specifically at the center of this tag. 3. Green foliage in the distance, on the other side of the water, which could be trees on the far shore. 4. The tag appears to be the lower part of the torso of person, specifically around the waist area. 5. A white structure with blue markings, which appears to be part of lifeguard stand beach-related signage. 6. A sign, partially obscured by the tag, with written text. 7. Clear blue sky. 8. The tag centered on the midsection of person wearing floral swimsuit. pink parasol or umbrella, which the person is holding.
Here's a
Table 5: Examples of GPT-4V listings with zero-shot, improved system message, and 2-shot in-context learning.
A.5 SoM Granularity Analysis
We present examples of visual tagging with semantic-SAM and different granularity, as shown in Figures to 14.
A.6 SoM Data in Existing Training Sources
Tables 7 to 9 shows a few examples that consist of listing in the text content.
16

---

Published as a conference paper at COLM 2024
SoM Granularity
Level 1
Level 2
Level 3
Figure 11: SoM tagging granularity analysis with level-1, level-2 and level-3 as coarse to fine.
SoM Granularity
Level 1
Level 2
Level -3 3
Figure 12: SoM tagging granularity analysis with level-1, level-2 and level-3 as coarse to fine.
SoM Granularity
Level
Level 2
Level
Figure 13: SoM tagging granularity analysis with level-1, level-2 and level-3 as coarse to fine.
SoM Granularity
Level
Level 2
Level 3

---

Published as a conference paper at COLM 2024
Listing example from GPT-4V, flower and
GPT-4 zero-shot
1. White vase 2. Small white flowers (species not clearly identifiable due to focus) 3. Green foliage plant (specific species unclear) 4. Green plant (specific species unclear) 5. Plant stems (specific species unclear) Green leaves (specific species unclear) 7. Green leaves (specific species unclear) 8. Green foliage plant (specific species unclear) Green foliage or plant in the background (specific species unclear due focus)
GPT-4V improved sys-msg
1. A white vase. 2. Part of shrub bush, possibly with tiny white flowers, in the background. Greenery that looks like lawn ground cover, it's blurred and behind ledge or step with tag number 8. 4. Some plants which they could be grass or small plants. Flowers, specifically white and possibly some type of hydrangea, 6. Green foliage a tree or large bush in the background. 7. The pedestal or foot of the vase from tag number 1. 8. This tag seems to be positioned on the edge of a railing or surface with a white finish. 9. More blurred greenery, possibly bushes or trees in the background.
blurred,
GPT-4V 2-shot learning
1. A white vase with a textured design. 2. Tag 2 is hovering above some sort vegetation, likely bush or shrub. 3. The tag is pointing to an area that appears to be part of well-manicured lawn or grassy area. 4. Green grass is suggested by this tag. although it is not entirely clear. 5. This appears to be pointing to flowers in the bouquet, possibly white Hydrangeas. 6. Appears be pointing shadow cast on the lawn of flowers within the vase; they are likely white hydrangeas or another type of similar- looking flower. Is tagging the flat surface which appears to ledge or balustrade, likely made out of painted wood based on the context. 9, A blurred area, suggestive of more green foliage or shrubbery in the background.
plant
shaded area of grass. 7. A cluster
Table 6: Examples of GPT-4V listings with zero-shot, improved system message, and 2-shot in-context learning.
Scitcl
Figure 15: Discovered images with tagging annotations in LLaVA-PPetrrin-LCC-555K.
18

---

Published as a conference paper at COLM 2024
Image
Text
Question:
What are some potential benefits or reasons people might gather at the beach to fly kites or engage in kiteboarding?
Answer:
There are several reasons or benefits for people to gather at
beach to fly kites engage in kiteboarding:
1. Socializing and bonding: Kite flying and kiteboarding
events can bring people together, fostering sense of com- munity and social interaction among the participants. These activities allow people of all ages and backgrounds estab- lish connections and develop new friendships.
Outdoor recreation: Engaging in kite flying or kiteboard- ing can be fun way to spend time outdoors and enjoy the fresh air and natural scenery. Beaches often offer wide-open spaces and consistent winds, which are ideal conditions for these activities.
3. Physical exercise: Both kite flying and kiteboarding can provide form of physical exercise, improving the partic- ipants' overall fitness and well-being. Kite flying can be low-intensity workout, strengthening the arms and shoul- ders, while kiteboarding is more physically demanding sport that can enhance balance, core strength, and cardiovas- cular endurance.
4. Stress relief: Participating in kite flying or kiteboarding can help reduce stress and promote relaxation. These activi- ties can serve form of mindfulness and mental exercise, allowing the participants to focus solely on the task at hand, temporarily setting aside everyday concerns and responsi- bilities.
5. Skill development: Flying kites and kiteboarding involve the development and refinement of various skills, such as coordination, dexterity, and control. Mastering these skills can provide a sense accomplishment and personal growth. th. In summary, people may gather at beach to fly kites or engage in kiteboarding for various reasons, including so- cializing, outdoor recreation, exercise, stress relief, and skill development. These activities can offer range of physical, mental, and social benefits, making them popular and enjoy- able pastimes for those who take part.
Table An example from CogVLLM-SFT-3111, with the answer text generated by GPT4-0314 and contains listing.
19

---

Published as a conference paper at COLM 2024
Image
Text
Question:
Can you provide some background information about coal- powered locomotives?
Answer:
Coal-powered locomotives, also known as steam locomo- tives, were the primary mode of railway transportation from the early 19th century until the mid-20th century. These lo- comotives used steam power generated by burning coal to operate their engines, which provided the force necessary for the train to move along the tracks.
A coal-powered locomotive was usually comprised of sev- eral key components:
1. A firebox or combustion chamber where coal was burned to create heat.
A boiler that contained water, heated by the firebox, to produce steam.
Pistons and cylinders that converted the steam pressure into mechanical motion, allowing the locomotive's wheels turn.
4. A coal car tender, which carried the coal fuel for the locomotive, usually located just behind the engine.
During the first half of the 20th century, coal-powered loco- motives were steadily replaced by diesel and electric trains, which proved to be much more efficient and cleaner. Al- though steam locomotives are largely obsolete today, many preserved in museums operated for tourist attractions to provide a glimpse into the history of railway transporta- tion.
The image of an old coal-powered locomotive and coal car on old tracks in a city setting offers a snapshot of a bygone era, showcasing the marvels of early railway technology and providing us with window into the past.
Table 8: An example from CogVLM-SFT-311K, with the answer text generated by GPT4-0314 and contains listing.
20

---

Published as a conference paper at COLM 2024
Image
Text
Question:
Can you talk about the role of footwork, movement, and jumping in tennis?
Answer:
Footwork, movement, and jumping are crucial components of good tennis game, both in terms of enhancing overall performance and reducing the risk of injuries.
1. Footwork: Good footwork allows tennis player to move efficiently and quickly around the court, enabling them to reach the ball and return it effectively. Proper footwork involves maintaining a balanced stance, taking small adjust- ment steps, and employing various movement patterns such as the shuffle, crossover, and side-step. Mastering footwork not only improves speed and agility but also ensures the player is in the right position to execute their strokes with optimal power and accuracy.
2. Movement: Movement is an essential aspect of tennis, as players must continually readjust their position respond to their opponent's s shots. Efficient movement on the court requires a combination of speed, agility, flexibility, and en- durance. Anticipation plays significant role in effective movement, as players must predict the trajectory of the ball and adjust their position accordingly. Top players often pos- sess the ability to change direction quickly and cover the court effectively, enabling them to counter their opponent's strategies and make challenging shots.
3. Jumping: Jumping a valuable skill in tennis, particularly when executing specific shots such as the overhead smash, serve or aggressive baseline shots. well-executed jump allows the player to generate more power, reach for higher balls, and create sharper angles their shots. Jumping while hitting the ball gives the player a slight height advantage, which enables them to strike the ball at a higher contact point, resulting in more aggressive and effective shot. However, it is essential to practice proper techniques and ensure a safe landing to avoid potential injuries.
In conclusion, footwork, movement, and jumping are vital components of tennis performance. Mastering these skills makes player more effective and efficient on the court, enhancing their ability respond various situations and execute wide range of shots. Proper training and practice in these areas can significantly elevate a player's game and contribute to overall success in the sport.
Table An example from CogVL.M-SFT-3111, with the answer text generated by GPT4-0314 and contains listing.
21


Reviews:
[{"review_id": "lYSyE5U8g3", "reviewer": "Reviewer_HXE7", "summary": "Set-of-Mark (SoM) prompting (placing numbered tags on images before feeding them to the models) has been shown to improve visual reasoning on GPT-4V.\nThis paper creates a dataset helping MLLMs to acquire the SoM visual prompting ability, and shows that MLLMs other than GPT-4V (such as LLaVA-1.5) can also benefit from this technique. The SoM models are evaluated on 5 benchmarks and show improvement even without the tags. The authors also investigated the mechanism of SoM.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for the appreciation of our work! Below are the responses to your questions.\n\n**Q1** Effectiveness on other types or sizes of VLMs \n\n**A1** Thanks for your suggestion. The reason we choose LLaVA is, it is a fully open-sourced model (both data and model weights for both pretraining and SFT stages are released), so it makes the full diagnosis possible. And its model size and training recipe is what we can afford given the limited budget.\n\nHere we deliver additional results from another recently open-sourced M-LLM (xgen-mm-phi3-mini-base on HuggingFace, with a different architecture and pretrain recipe). We use two datasets: LLaVA-665K and SoM-LLaVA-695k to finetune the same base model, and evaluate their performance: \n\nThe performance comparison is shown below:\n| Model       | POPE | MME  | SEED-I | LLaVA-W | MM-Vet |\n|-------------|------|------|--------|---------|--------|\n| xgen-mm     | 86.4 | 1425 | 69.4   | 68.4    | 37.8   |\n| SoM-xgen-mm | 87.2 | 1437 | 71.2   | 72.0    | 39.9   |\n\nWe also tested our dataset with LLaVA-1.5-7B with a smaller LM (vicuna-7B), as shown below:\n| Model       | POPE | MME  | SEED-I | LLaVA-W | MM-Vet |\n|-------------|------|------|--------|---------|--------|\n| LLaVA-1.5-7B          | 85.9    | 1510 |  64.8 | 63.4  | 30.5      |\n| SoM-LLaVA-1.5-7B | 86.5    | 1525 |  67.0 | 66.9  | 33.3      |\n\n**Q2** Additional overhead for visual instruction tuning. \n\n**A2** Our dataset is relatively small (30K in total), adding less than 5% training time for finetuning. Finetuning an already instruction-tuned model with SoM data is possible, though it may lead to forgetting of previous data. We follow the common practice to mix the training data.\n\n**Q3** listing data from 75k to 100k \n\n**A3** We generate the listing with coco annotations. The object classes are limited (80 classes) hence there could be some overfitting issue. Also, the annotation itself can be noisy, so a matching score of around 85% may be hard to improve. We will clarify this in the revision.\n\n**Q4** Adding tags to small objects\n\n**A4** We follow the tagging method in the SoM paper [1] where we slightly move the tag down from the center by a few pixels, so small objects are still visible.\n\n**Q5** Training time\n\n**A5** It takes around 20-22 hours to train LLaVA-1.5-13B on 8 A100s (80GB). \n\n**Q6** Typo and formatting. \n\n**A6** Thanks for your careful review, we will revise the typos.\n\n[1] Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Interesting work, with three-fold contributions: probing SoM's mechanism, proposing an approach adding the SoM capability for other MLLMs with a dedicated dataset. The work's insight can be beneficial for the research community.\n\nThe paper is clearly written, well structured, and easy to follow. The appendix also contains useful details.\n\nThe authors stated that the dataset and models will be released.", "reasons_to_reject": "The approach's effectiveness is only demonstrated with LLaVA-1.5. Probing other types or sizes of VLMs can be useful to better understand its applicability. \n\nThis adds additional overhead and needs to retrain the instruction tuning stage. It seems unclear if fine-tuning an already instruction-tuned model using the SoM data would work.", "questions_to_authors": "In Figure 3 (a), increasing the listing data from 75k to 100k seems to lead to a slightly lower accuracy for both the 7B and 13B models. Do the authors have an idea of why?\n\nAdding tags may obstruct very small objects. Can the technique still be helpful in this scenario?\n\nCan the authors indicate how long the training was for both models?\n\nThe LaTeX style seems different from the official version.\n\nTypo:\n- Paragraph 1: \"can implicitly aligning [...]\" -> \"can implicitly align [...]\"\n- 4.3, \"GPT-4V is asked to generate mulit-turn\" -> \"multi-turn\"\n\nMinor:\n- Figure 2: it's a bit hard to read the tags behind the attention maps, they can be emphasized. Also, some red patches seem to be going a bit off the image.", "extra_scores": {}}, {"review_id": "WU1zyvwlJB", "reviewer": "Reviewer_YAe1", "summary": "The paper proposes a new dataset for Set-of-Mark (SoM) prompting for MLLMs. SoM allows MLLMs to associate objects with text by assigning numbers to different objects. Adding the proposed dataset to existing visual instruction tuning datasets improves the performance on five visual reasoning tasks. The SoM dataset is created by modifying an existing dataset (MS-COCO) that has human annotations corresponding to bounding boxes, and captions. An existing model (SAM) is used to segment the images prior to adding numbers on top of each segment then use GPT4-v to generate the textual descriptions. The authors fine-tune the pertained LLaVA-1.5 model using LLAVA-1.5 fine-tuning data in addition to the newly created dataset and show improvements on five visual reasoning tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the appreciation of our work! Below are the responses to your questions. \n\n**Q1** The proposed approach and dataset are only applied to one MLLM. It would be good to try other MLLMs to see if the results generalize.\n\n**A1** Thanks for your suggestion. We deliver additional results from two other MLLMs. \n\nFirst, we used another recently open-sourced M-LLM with pretrained weights available (xgen-mm-phi3-mini-base-r-v1 on HuggingFace, with a different model architecture and pretrain data compared with LLaVA). We compare with two visual instruction tuning recipes as in our paper: LLaVA-665K and SoM-LLaVA-695k to finetune the same base model, and evaluate their performance on the five benchmarks, using raw images without tags during testing. The performance comparison is shown below:\n\n| Model       | POPE | MME  | SEED-I | LLaVA-W | MM-Vet |\n|-------------|------|------|--------|---------|--------|\n| xgen-mm     | 86.4 | 1425 | 69.4   | 68.4    | 37.8   |\n| SoM-xgen-mm | 87.2 | 1437 | 71.2   | 72.0    | 39.9   |\n\nWe also tested our dataset with LLaVA-1.5-7B which has a smaller language model (vicuna-7B) compared to the one we reported in Table.2. The performance comparison is shown below:\n\n| Model       | POPE | MME  | SEED-I | LLaVA-W | MM-Vet |\n|-------------|------|------|--------|---------|--------|\n| LLaVA-1.5-7B          | 85.9    | 1510 |  64.8 | 63.4  | 30.5      |\n| SoM-LLaVA-1.5-7B | 86.5    | 1525 |  67.0 | 66.9  | 33.3      |\n\n\n**Q2** It would be good to evaluate the in-context learning (without fine-tuning) performance using exemplars from the new dataset.\n\n**A2** Thanks, that is an interesting direction to explore, with the recent releases of open-sourced interleaved M-LLMs that are capable of performing in-context learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting approach & well though-out analysis on using SoM data to improve the  visual reasoning performance of MLLMs.\n- Fine-tuning an MLLM using the proposed dataset improves the performance on five visual reasoning tasks.", "reasons_to_reject": "- The proposed approach and dataset are only applied to one MLLM. It would be good to try other MLLMs to see if the results generalize.\n- It would be good to evaluate the in-context learning (without fine-tuning) performance of using exemplars from the new dataset.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2hp60kgmCT", "reviewer": "Reviewer_nfpB", "summary": "This paper presents a simple and compelling idea for enriching the spatial and fine-grained reasoning of multimodal large language models (MLLMs): rather than solely condition language generation on an image, the key idea is to augment an image with a “set of marks” (alphanumeric tags overlaid on the image) that are associated with key locations or reference points. As an example, given an image of a crowd at a concert, a plausible set of marks might include a different tag on each of the members of the crowd, a mark on the stage, and a mark on each of the musicians and their corresponding instruments.\n\nNote that this idea of “marking” is an alternative input scheme that type checks with existing MLLMs that take in arbitrary images and generate text; at inference time, one can still provide untagged images, or they can manually tag an image with marks, and have the MLLM generate outputs referring to individual marked objects.\n\nThe bulk of the paper 1) discusses the failures of existing MLLMs such as LLaVa-1.5 when it comes to reasoning over “marked images”, 2) curating a small “marked” dataset of just 10-30K examples of images with labeled outputs just consisting of “List items one by one: <tag 1>: <description> ...”, and 3) evaluating MLLMs trained with this additional data on a spectrum of evaluation tasks. \n\nIn all, the proposed “marking” scheme is simple, scalable, and leads to MLLMs that outperform non-marked variants **in the absence of marking data during evaluation**, especially on tasks requiring fine-grained reasoning. The paper also presents case studies verifying that MLLMs trained with marking can integrate such marks for complex reasoning over full scenes, beyond the capabilities in the training data.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the appreciation of our work! Below are the responses to your questions.\n\n**Q1** More evaluation benchmarks \n\n**A1** Thanks for the suggestion. We will test on more benchmarks before the final version.\n\n**Q2** Results on 7B parameter model\n\n**A2** Thanks for pointing it out. We are limited by compute resources when running ablations. Results from LLaVA-1.5-7B are here:\n\n| Model       | POPE | MME  | SEED-I | LLaVA-W | MM-Vet |\n|-------------|------|------|--------|---------|--------|\n| LLaVA-1.5-7B          | 85.9    | 1510 |  64.8 | 63.4  | 30.5      |\n| SoM-LLaVA-1.5-7B | 86.5    | 1525 |  67.0 | 66.9  | 33.3      |\n\nAdditionally, we also present results from another recent M-LLM (xgen-mm-phi3 on HuggingFace, with a different architecture and pretrain recipe). We follow the setting on LLaVA to finetune the base model, with or without SoM data:\n\n| Model       | POPE | MME  | SEED-I | LLaVA-W | MM-Vet |\n|-------------|------|------|--------|---------|--------|\n| xgen-mm     | 86.4 | 1425 | 69.4   | 68.4    | 37.8   |\n| SoM-xgen-mm | 87.2 | 1437 | 71.2   | 72.0    | 39.9   |\n\nWe will add these results to show the performance gain of different models with our SoM data\n\n**Q3** Do we need GPT-4V to produce the descriptions for each of the tagged objects?\n\n**A3** Thanks for initiating an interesting discussion. For MS-COCO we do have abundant annotations to create synthetic data. We considered this method in the early stage of our project, but after checking samples from VG, we find that the captions are not really “open-vocabulary” as what GPT-4V is capable of. For example, regional captions such as “man wearing in a hat”, “tree near the water” missing fine-grained attributes about the color of the hat, type of the tree, man’s posture or action, etc.\n\nA similar idea that may work for all image sources could be extracting local patches using object detectors, and use GPT-4V or other caption models to annotate the patches. In our work, we go with a simple and scalable method that relies on two foundation models, semantic-SAM and GPT-4V, to generate such data. \n\nFor fair comparison with LLaVA-150k and share-GPT4v data, we use the same images from COCO to make sure no extra information are introduced in the images, so we can faithfully verify this new learning paradigm (list items one by one) is indeed effective. In the future, one can scale up the data to different image sources, using our pipeline or the alternatives discussed above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a simple and scalable method for enriching arbitrary MLLMs to perform better at fine-grained reasoning tasks, and provides a new “input modality” for controllable interaction with MLLMs for more complex tasks. The work does a great job in identifying the limitations of existing MLLMs and MLLM datasets, and present a scalable approach for generating such “marked” data from existing image datasets (one that scales far beyond the data generated for this work). \n\nThe evaluations show that MLLMs trained with “marked” data not only can use such “marks” for complex reasoning, but also show that just training with this marked data enriches the abilities of MLLMs to perform fine-grained and spatial reasoning in the **absence of marks during evaluation** as well.", "reasons_to_reject": "While the existing evaluations are compelling, it would make this work even stronger to explicitly evaluate the trained MLLMs on evaluations that provide calibrated/absolute metrics on spatial and compositional reasoning (e.g., GQA question answering accuracy, RefCOCO bounding box prediction, etc.). I recommend that the authors look into strengthening the existing evaluations for the final version of the paper.", "questions_to_authors": "Is there a reason that the results don’t include a 7B parameter model? Table 2 only shows a 13B variant of the mark-augmented models.\n\nDo we need GPT-4V to produce the descriptions for each of the tagged objects? If we’re using MSCOCO images to seed our image dataset anyway, can we just use the object labels from that dataset (and possibly other datasets like Visual Genome that have individual region/object captions)?", "extra_scores": {}}, {"review_id": "YJwhLlROBJ", "reviewer": "Reviewer_L6Ef", "summary": "This paper presents a new data source to better learn multimodal LLMs by collecting listing and QA data with set-of-mark (SoM) prompting from GPT4V. A new learning paradigm is also designed to enhance the visual understanding ability of MLLMs. Results show that the new dataset can improve the performance of MLLMs on various benchmarks and enable SoM prompting ability of MLLMs.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for the appreciation of our work! Below are the responses to your questions.\n\n**Q1** improvements on POPE are not very significant.\n\n**A1** The results are significant if we compare with the performance from other models (average F1 score), for example, the recently released phi-3-vision model has a score of 85.8, LLaVA-next-LLaMa-3 has a score of 87.0. Most models fall into the 83 to 87 range, even with more advanced models and data. The performance of LLaVA-1.5 is already decently high on POPE, hence it may be hard to improve further. Our SoM data can boost LLaVA-1.5 from 85.9 to 87.0, which shows its efficiency. We also present results from LLaVA-1.5-7B, where the performance improvements are consistently observed (slightly higher on some tasks, such as SEED-I and MM-Vet, than the 13B model)\n\n| Model       | POPE | MME  | SEED-I | LLaVA-W | MM-Vet |\n|-------------|------|------|--------|---------|--------|\n| LLaVA-1.5-7B          | 85.9    | 1510 |  64.8 | 63.4  | 30.5      |\n| SoM-LLaVA-1.5-7B | 86.5    | 1525 |  67.0 | 66.9  | 33.3      |\n\n**Q2** How to ensure the quality of the dataset. \n\n**A2** Thanks for the question. We explore two approaches, one is to modify system prompts, the other is to design high-quality demonstrations for few-shot prompting. We have an example in Table.4 in Appendix. We conduct qualitative comparison between our model and GPT-4V in Figure.9 and 10, showing our model can generate better listings than GPT-4V in a zero-shot setting. Overall, how to control the quality of any synthetic datasets is an interesting direction that requires further investigation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Overall I think this is an interesting paper with some new insights and sufficient experimental study. The proposed method to collect SoM prompting data to improve MLLMs is well-motivated and interesting. Enabling the SoM prompting ability of MLLMs may be an effective method to close the gap between open-sourced MLLMs and GPT4V.\n\n- Extensive experiments are conducted to show the effectiveness of the proposed method.", "reasons_to_reject": "- The improvements look marginal on some of the benchmarks. In the abstract, it is mentioned that the method can \" reduce hallucinations in Multimodal LLMs\" but the improvements on POPE are not very significant. Can you provide an analysis of this phenomenon? \n\n- How to ensure the quality of the dataset? GPT4V may produce wrong results. Would this affect the final performance of the models trained with the dataset?", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wq0yTOTY2I", "reviewer": "Reviewer_v9mL", "summary": "This paper studies the Set-of-Mark capability of existing VLMs other than GPT4-V, and proposes a valid solution to construct training data for getting this capability.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and the appreciation of it in terms of strengths. Below are our responses.\n\n**Q1** Contribution of the work.\n\n**A1** We would like to clarify our contributions here. First of all, data is perhaps one of the most critical aspects for today’s foundational models.\nThough we start with discussing SoM prompting and our work is motivated by it, we find “listing items one by one” is also effective for general MLLM training: if a model learns to list items with a specific order (in our case, it is determined by the visual numeric tags), it gains a fine-grained understanding of images.\nVerified by our experiments, we believe this is an important implication and opens up an interesting discussion on how to better align visual and language.\n\n**Q2** Details about the approach\n\n**A2** Thanks for the suggestion. Our data creation pipeline is simple, we add numeric tags to the images using semantic-SAM (section 4.1), and generate listing-style captions with GPT-4V (section 4.3). Data examples can be found in Figure.9-10 and Table.4 in appendix, we will add constructed examples to the main paper. We will also add a figure for readers to better understand the pipeline. \n\n**Q3** Comparison with direct referring\n\n**A3** In LLaVA-665k dataset we used, there is a category of visual referring data, ref-coco and visual genome, with 134k data. Hence our results in table.2 show that SoM data complements this type of data. Most visual referring LLMs [1,2] put coordinates of 1-2 objects into text, while we add visual tags on images to encode spatial information. We will cite more papers along this line, for example, [2, 3, 4]. Let us know if there are any specific papers to be included.\n\n**Q4** Details about the binary classifier\n\n**A4** We use the ViT from [5]. We use all raw images from coco train-2017, and construct tagged images by randomly adding 1-10 numeric tags (numbers chosen from 0-99) to the images. We built a balanced dataset with 236K images in total. We train the binary classifier for 5 epochs with learning rate 5e-5 and batch size 128. We will add details to the paper. \n\n[1] Improved Baselines with Visual Instruction Tuning\n\n[2] VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks\n\n[3] Ferret: Refer and Ground Anything Anywhere at Any Granularity\n\n[4] Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic\n\n[5] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Interesting problem of studying the Set-of-Mark capability of the VLM, the study from the paper indicates all VLMs can get similar capability if the model is trained specifically. \n\n2. Promising but not the major contribution of the paper, the authors show that adding the training data designed for learning SoM capabilities can improve the VLM performance in general.", "reasons_to_reject": "1. The contribution of the work is limited. It is yet another paper which constructs a training data, train the model, and get the capability. \n\n2. The paper is not ready to publish yet. Many details about the approach are missing, I tried to read the section 4 multiple times but still cannot get the full picture of the construction process. More illustrations of the approach and the constructed examples will be necessary for reader to understand the proposed approach. Maybe move the examples from Appendix 1.6 into the main content. Instead, section 3 should go to Appendix, conjunction with moving the primary investigation results / conclusion goes to the introduction. \n\n\n3. One key motivation is that the Set-of-Mark capability is important for visual referring prompting. As pointed in the related work, there is another important type of the referring technology which put user inputs (point, bounding boxes, sketch) directly into the text input. A comparison of Set-of-Mask approach and direct referring approach will make the paper more motivated. Also, the author should discuss and cite more papers here, there are many recent work on this direction.", "questions_to_authors": "In section 3.2 it is unclear what is the binary classifier, and other details about the classifier.", "extra_scores": {}}, {"review_id": "byAYTDdIsk", "reviewer": "Reviewer_gxtp", "summary": "The paper introduces a novel dataset and learning paradigm aimed at enhancing Multimodal Large Language Models (MLLMs) by teaching them to list items in images one by one. This method is designed to improve the association of visual objects with corresponding textual descriptions through a process called Set-of-Mark (SoM) Prompting. By tagging images with numeric identifiers and prompting models to list these tags, the study demonstrates significant improvements in visual reasoning and a reduction in hallucinations across several MLLM benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the appreciation of our work! Below are the response to your question. \n\n**Q1** The application area of this work and expanding this methodology to foundational models. \n\n**A1** Thanks for your suggestion. In our experiments, we have shown that our proposed dataset and learning paradigm is useful not only for empowering MLLMs with SoM prompting, but also for enhancing general visual reasoning and reducing hallucinations on popular benchmarks. It can be added as a complementary data source to existing visual instruction tuning datasets (see our additional results provided to other reviewers) for any visual LLM training. \n\nNevertheless, it would certainly be interesting to explore how this methodology can apply to other types of foundational models, e.g., enhancing image/video generation for diffusion models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Clear Motivation: The paper clearly articulates the motivation to enhance Multimodal Large Language Models (MLLMs) with the ability to list items, which could improve how these models handle visual objects in combination with textual descriptions.\n* Innovative Approach: Introducing a novel dataset and a \"list items one by one\" learning paradigm is innovative. This method allows models to better integrate and process visual and textual information.\n* Strong Results: The paper reports substantial benefits in terms of reduced hallucinations and improved visual reasoning across several MLLM benchmarks.", "reasons_to_reject": "The research primarily focuses on t specific aspect of multimodal learning, which might be seen as a limited application area. Expanding this methodology to foundational models could make the work more impactful.", "questions_to_authors": "n/a", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper examines whether multimodal large language models (MLLM) can map objects "marked" in the image to the words in a prompt. The authors create a dataset based on MSCOCO by automatically finding the objects, tagging them with numbers ("marking them"), and adding captions using a LLM. They also discuss how a recent model fail at describing the marked aspects of an image, and show that the model performance can be improved when trained on the collected dataset. The reviewers find the paper clear, and the proposed approach interesting and well executed. I agree with the reviewers but encourage the authors to evaluate their method on commonly-used fine-grained benchmarks such as GQA (as Reviewer nfpB points out) as well as typical retrieval and QA benchmark to see if the performance on generic tasks changes/degrades with the additional training.

[At least one review was discounted during the decision process due to quality]
Decision: Accept