List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs

Authors: an  yan
zhengyuan  yang
junda  wu
wanrong  zhu
jianwei  yang
linjie  li
kevin  lin
jianfeng  wang
julian  mcauley
jianfeng  gao
lijuan  wang
u c san
diego ♡ uc
santa  barbara
♠  microsoft
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-20 22:58:50

Abstract: Set-of-Mark (SoM) Prompting unleashes the visual grounding capability of GPT-4V, by enabling the model to associate visual objects with tags inserted on the image. These tags, marked with alphanumerics, can be indexed via text tokens for easy reference. Despite the extraordinary performance from GPT-4V, we observe that other Multimodal Large Language Models (MLLMs) struggle to understand these visual tags. To promote the learning of SoM prompting for open-source models, we propose a new learning paradigm: list items one by one, which asks the model to enumerate and describe all visual tags placed on the image following the alphanumeric order of tags. By integrating our synthetic dataset with other visual instruction tuning datasets, we are able to equip existing MLLMs with the SoM prompting ability. Furthermore, we evaluate our finetuned SoM models on seven MLLM benchmarks. We find that this new dataset, even in a relatively small size (10k-30k images with tags), significantly enhances visual reasoning capabilities and reduces hallucinations for MLLMs. Perhaps surprisingly, these improvements persist even when the visual tags are omitted from input images during inference. This suggests the potential of ``list items one by one'' as a new paradigm for training MLLMs, which strengthens the object-text alignment through the use of visual tags in the training stage. Finally, we conduct analyses by probing trained models to understand the working mechanism of SoM. Our code and data are available at https://github.com/zzxslp/SoM-LLaVA.

Main paper: 
 introduction: 
 Recent advances in Multimodal Large Language Models (MLLMs) such as GPT-4V (OpenAI, 2023a) show strong performance in multimodal perception and reasoning, enabling various new capabilities (Yang et al., 2023b) . Among these, Set-of-Mark Prompting (SoM) (Yang et al., 2023a) is an interesting new working mode that enhances the connection between visual objects and textual tokens via visual prompting, i.e., placing alphanumeric tags on input images. It provides a natural interface for human-computer interaction, by linking visual locations to executable actions through visual tags, and enables various applications such as GUI navigation (Yan et al., 2023b) and robot interaction (Lin et al., 2023a) . Furthermore, GPT-4V with SoM (Yang et al., 2023a) can implicitly align visual objects with their corresponding tags. Such alignments (Li et al., 2020; Yang et al., 2021) allow MLLMs to leverage index numbers to perform multi-hop visual reasoning (Yang et al., 2023a; Wei et al., 2022) , thereby improving their abilities in multimodal understanding and reasoning tasks.Despite the significant interest in SoM prompting and its broad applications, it remains unclear why GPT-4V can benefit from SoM prompting, We find that other MLLMs, including the state-of-the-art open-sourced models such as LLaVA-v1.5 (Liu et al., 2024) , and commercial systems like Gemini (Team et al., 2023) , struggle to understand SoM prompts. This gap prevents them from leveraging the effectiveness of SoM prompting. In this study,To move the speaker closer to the curtain while keeping the laptop in the current position, which item should I swap it with?LLaVA-1.5 You should swap the laptop with the cup .a cupWhat items are there near the Marshal speaker?SoM-LLaVA There is a laptop tagged with number 7 and a notebook tagged with number 8.To move the speaker closer to the curtain while keeping the laptop in the current position, which item should I swap it with?SoM-LLaVA You can swap it with the lamp tagged with number 9. we aim to deepen the understanding of SoM, with a goal of facilitating arbitrary MLLMs to benefit from it.We break down SoM prompting into three core capabilities: (1) the ability to identify all tags and read the alphanumeric scene texts written on them;(2) the ability to recognize and pinpoint all objects in an image;(3) the ability to associate tags with corresponding objects in the image. Despite possessing skills such as OCR and visual recognition to meet the first two capabilities, most MLLMs still fail to fully understand SoM prompts. Therefore, we hypothesize that the crucial missing element is the third capability, associating tags with objects, which requires deliberate training. We further validate that SoM-style data are sparse in common MLLM training sources, and it may be necessary to create a specific dataset.To facilitate such training, we introduce a new learning paradigm named "list items one by one". We show that by asking MLLMs to comprehensively list all tagged items following the alphanumeric order of visual tags, MLLMs can learn SoM prompting with a small number of item-listing samples. Specifically, we create a tailored synthetic dataset, by tagging images with Semantic-SAM (Li et al., 2023c; Yang et al., 2023a) , and prompting GPT-4V to generate paired text descriptions. With just 10k image-text pairs, MLLMs like LLaVA-1.5 (Liu et al., 2023a) can reliably understand SoM tags. Based on this initial finding, we conduct studies to explore the effective recipes to help MLLMs best utilize SoM prompting.We enhanced MLLMs with this "list items one by one" objective and assess their SoM performance from two aspects: model's ability to recognize and describe the SoM tags, and its ability to use SoM in improving multimodal reasoning ( Figure 1 ). For the first aspect, we design the tag listing task, which requires MLLMs to list and describe all tags in the image, evaluated by listing accuracy. For the second aspect, we evaluate finetuned models on seven MLLM benchmarks, including GQA, POPE, MME, SEED-Bench, LLaVA-Bench, MM-Vet and MMBench, showcasing that MLLMs with SoM can significantly boost the multmodal understanding performance. Moreover, our model trained with SoM data outperforms the original MLLM, even without additional visual tags during inference. This demonstrates the potential of incorporating our proposed dataset and learning paradigm to boost general MLLM training.Finally, we revisit our original question regarding the working mechanism of SoM. The preliminary hypothesis is that the SoM capability may be related to OCR and the implicit association among text, tags, and objects. With our trained models, specifically SoM-LLaVA, we gain access to model features and attention maps for an in-depth analysis. We visualize the attention map to verify tag association. Compared with the original LLaVA model, SoM-LLaVA indeed learns better visual-tag-text associations, reflected in corresponding attention maps.Our contributions are summarized as follows:• We present a new training task and synthetic data source named "list items one by one", which effectively bootstraps MLLMs for the SoM visual prompting ability. • We evaluate our finetuned models on general MLLM benchmarks, and show improved performance even when SoM tags are removed from the input image. • We probe the working mechanism of SoM through the trained MLLMs, showcasing the implicit association between visual objects and text tokens when performing SoM prompting. 
related work: 
 Visual referring prompting. Other than text prompts, visual referring prompting (Yang et al., 2023b) is another effective approach when interacting with multimodal LLMs, where users directly draw on input images to specify their intent, such as drawing visual pointers or handwriting scene texts. Early studies show that vision-language models can understand visual pointers such as circles (Shtedritski et al., 2023) and dots (Mani et al., 2020) . Recent studies (Yang et al., 2023b) show that more powerful multimodal LLMs (OpenAI, 2023a) can handle more complicated prompts such as arrows, boxes, circles, hand drawing, scene text, as well as their combinations. Another major advancement is Set-of-Mark Prompting (SoM) (Yang et al., 2023a) , where numbered tags can be placed on images to associate visual objects with text indexed. Its effective visual grounding capability (Kazemzadeh et al., 2014; Yu et al., 2016; Mao et al., 2016) enables various applications (Yan et al., 2023b; Zhang et al., 2023) . In this work, we aim to better understand SoM and extend its success from GPT-4V (OpenAI, 2023a) to other open-source multimodal LLMs.Multimodal LLMs. Multimodal LLMs (Alayrac et al., 2022; Zhu et al., 2022; OpenAI, 2023a; Liu et al., 2023b; Li et al., 2023b; Xue et al., 2024) extend large language models (OpenAI, 2023b; Gao et al., 2023; Touvron et al., 2023) with visual perception capabilities. Recent studies (Chen et al., 2023) show the effectiveness of training open-source models on the GPT-4V generated detailed description data. Another thread of studies explore having multimodal LLMs predicting object locations as bounding boxes (Wang et al., 2023b; Peng et al., 2023) or masks (Rasheed et al., 2023) . In contrast to most prior studies that pair the images with different text instructions, our study explores a new direction of how visual prompts such as SoM can improve multimodal LLMs. Specifically, we show that the SoM visual tags provide fine-grained alignments between visual objects and text tokens, thereby improving various visual reasoning tasks, both with and without SoM prompting during inference. 
visualizing som prompting on llava: 
 In this section, we first investigate the capacity of LLaVA-1.5 in SoM, concerning its attention sensibility to the numeric IDs tagged on the objects and its answer to the SoM query. We show an example task to list a series of objects tagged with numeric IDs in Figure 2 , in which the attention map is extracted from LLaVA-1.5 based on the SoM query (e.g., "I have labeled a bright numeric ID at the center for each visual object in the image. Please enumerate their names."). The top 20 image patches with the highest average attention weights across the user query tokens are highlighted in transparent red regions. We can observe from the highly attended regions of LLaVA-1.5 that the numeric ID tags can be easily and correctly attended by LLaVA-1.5 along with their associated objects (e.g., bird, vase, and branches). Such capacities in locating numeric ID tags may have been acquired by LLaVA-1.5 from its pretraining tasks in OCR and also benefited from the strong OCR abilities of the ViT feature encoder (Radford et al., 2021) adopted by LLaVA-v1.5. However, the response prompted by the user query in the first example of Figure 2 suggests that LLaVA-1.5 cannot follow the SoM instruction to list all the items. Instead of providing the object descriptions corresponding to all the numeric ID tags, LLaVA-1.5 responds with a general image caption, due to a large portion of image captioning samples in its pretraining stage. From the second example of Figure 2 , we can also observe that although LLaVA-1.5 generates a list of tag IDs with object names, it cannot accurately associate the tags to corresponding objects, causing the model to hallucinate the descriptions of these objects.We further look into the pretraining/instruction-tuning (IT) dataset, aiming to inspect if there are text contents with listings, or images with SOM annotations. We examine the pretraining dataset of LLaVA-v1 and v1.5 (Liu et al., 2023b; a) , and the IT dataset used by LLaVA-v1.5, ShareGPT4V (Chen et al., 2023) , and CogVLM (Wang et al., 2023a) .Table 1 shows the source of text in each dataset and the percentage of text content with a listing format. The text in the two pretraining datasets for LLaVA are image captions (either the raw caption or generated by BLIP (Dai et al., 2023) ), and we did not find any text with listings in them using our parser. Aside from image captions, the IT dataset also contains instructions related to other visual tasks such as VQA. We noticed that the answers provided by GPT-4(V) models sometimes construct the text in a listing manner (e.g., list out possible reasons for a question, list out observed objects in the image, etc). More examples can be found in Appendix A.6. The instruction-following dataset used by CogVLM has the highest percentage of text with listings (∼7%). Through our interaction with these models, we also find CogVLM is better at generating listing-style data than LLaVA-1.5.We add tags to MSCOCO-2017 images following the SoM (Yang et al., 2023a) format, and train a binary classifier with ViT/B-16 (Dosovitskiy et al., 2020) . We use the classifiers to filter the images in the two LLaVA pretraining datasets, and take the top 2k images with the highest scores for each dataset. We then manually check the top 2k images, and found 12 images with tagging in CC3M-595K (∼0.002%), and found 86 images with tagging in LCS-558K (∼0.015%). Figure 15 shows a few images with tagging. Given that tagged images are sparse in those datasets and the SoM prompting performance of open-source MLLMs is unsatisfying, it may be worthwhile to design a tailored dataset that empower open-source MLLMs with this emergent ability, similar to what GPT-4V is capable of. 
dataset creation and training: 
 Motivated by the above analysis, in this section, we introduce the pipeline to create our dataset. First, in Section 4.1, we use semantic-SAM to generate semantic visual prompts in the form of numeric tags for each image. We then discuss the learning paradigm of "list items one by one" in Section 4.2. Finally, we use visual prompted images to generate text data in Section 4.3.There are various open-source image datasets available (Deng et al., 2009; Lin et al., 2014; Schuhmann et al., 2022; Yan et al., 2023a) . We use MS-COCO (Lin et al., 2014) as the image source to create our SoM dataset, since it contains comprehensive human annotations with bounding boxes, masks, and captions. It has also been widely used for visual instruction tuning (Liu et al., 2023b; Wang et al., 2023a; Chen et al., 2023) , which could benefit controlled experiments as well as comparisons with previous work.The first step is to create visual prompts by placing numeric tags on proper locations. Following SoM (Yang et al., 2023a) , we experiment with segmentation models including SEEM (Zou et al., 2023) , Semantic-SAM (Li et al., 2023c) , and SAM (Kirillov et al., 2023) . Empirically, we find that Semantic-SAM provides the annotation granularity that best fits COCO images, and thus use it to create tagged images for our dataset.After obtaining the image data with semantic tags, the next question is how to design the instruction data to best distill the SoM visual prompting ability. A common approach (Liu et al., 2023b; Chen et al., 2023) in multimodal instruction-following data creation is to design and collect "question-answering" style samples. This is often done by prompting ChatGPT/GPT-4 or alternative open-source models. Given an image I and optional metadata M I such as captions, bounding boxes, various questions or instructions X (i)Q are posed, and the corresponding answers X (i)A from large models are collected. However, such general question-answering data may not be the most effective in distilling the desired SoM prompting capability, due to the inadequate mention of objects in text. For SoM prompting, one core ability of interest is to associate numbered tags with visual objects in the image, thereby enabling effective referral of visual objects via text tokens. In a general QA data, however, it is rare for multiple objects to be mentioned, even in an extended multi-turn conversation. To enhance tag association, we propose a simple and effective approach: list items one by one, where the model is asked to comprehensively describe all tagged items within an image. Given an image I T with N text tags on the image, we ask the model to enumerate all items in numerical order:{X 1 obj , X 2 obj , • • • , X N obj }, where X j obj is the textual description of the j-th item, tagged by ID j in the image.Beyond promoting SoM learning, listing items one by one is also effective in general multimodal LLM training: if a model learns to list items in the images with a specific order (in our case, the order is determined by the visual numeric tags), it gains a comprehensive and fine-grained understanding of images. This could directly benefit visual grounding and reasoning, which we verified through the standard multimodal QA and chat evaluation benchmarks.Compared with existing visual instruction tuning datasets, such as LLaVA-665K (Liu et al., 2023a) and ShareGPT-4V (Chen et al., 2023) , another difference is the implicit spatial infor- Figure 3 : Performance analysis on tag listing. Training samples of listing data grow from 10k to 100k. list+mix-665k is to mix listing data with 665k instruction tuning data from (Liu et al., 2023a) . list+nonocr is to exclude the OCR and text data from the full 665k data, resulting in 563k samples. list+ocrtext is to mix listing data with only OCR and text data from the full 665k data, resulting in 102k samples. Green-dashed line in Figure 3a is the zero-shot result from GPT-4V. mation encoded by the visual tags in SoM prompting. Converting images into the language space inevitably loses information, especially spatial locations. For example, "a girl on the right" can only vaguely imply the position of the girl. However, with SoM visual prompting, we provide precise visual guidance on the image. Therefore, our data can be viewed as a form of dense captioning with a new way of encoding spatial information.With the visual prompting enhanced images, the final step for dataset creation is to generate the corresponding text data. To automate this process, we leverage GPT-4V (OpenAI, 2023a) to generate the listing data {X 1 obj , X 2 obj , • • • , X N obj }, following the order of visual tags in the images. However, we find that simply prompting the model to list items in a zero-shot manner could lead to noisy and biased generation results, where the model may refer the tag to a distant object that is easy to describe. (see examples in appendix A.4). To mitigate this problem, we seek two complementary solutions: (1) We modify the system message of GPT-4V to avoid assigning tags to distant objects. (2) We manually design a few correct listing samples via human annotations, and use them as seed examples for in-context-learning to query GPT-4V. The details of our template is in Appendix.In addition to listing, we also consider conversational data similar to LLaVA (Liu et al., 2023b) , where GPT-4V is asked to generate multi-turn conversations between an AI assistant and a person asking questions about the photo. Given a tagged image I T , we use GPT-4V to generate instruction-following data in the form of {Person:I T X (i) Q , Assistant: X (i) A }.We take the pretrained stage of LLaVA-1.5 (Liu et al., 2023a) as the base model, and continue finetuning by mixing instruction tuning data of LLaVA-1.5 with our collected visual prompting data. For SoM-listing, we create 40 task templates as human instructions (e.g., "please enumerate object names in the tagged image"), and treat them as standard conversational data. We use the same training objective of next-token prediction to train general QA, SoM-QA and SoM-listing data. Specifically, we maximize the conditional log likelihood as follows:EQUATIONwhere Θ are the trainable model parameters, X Q,<i and X A,<i are the instruction and answer tokens in all previous turns of conversations before the current prediction token x i . The input image is I or I T for LLaVA or SoM data, respectively. 
experimental settings: 
 Experiment overview. We validate the method effectiveness from two aspects. First, in Section 5.2, we benchmark the model's capabilities in understand and describing SoM visual prompting. We design the tag listing task on MS-COCO to test the SoM performance. Second, in Section 5.3 and 5.4, we evaluate if our dataset and model can benefit visual reasoning tasks, where we consider seven representative visual question answering and reasoning tasks detailed as follows.MLLM benchmarks. We consider the following multimodal LLM benchmarks in Table 2 to validate SoM visual prompting's benefit on general visual reasoning tasks. GQA (Hudson & Manning, 2019) focused on fine-grained compositional reasoning over real-world images.We compute match accuracy following (Liu et al., 2023a) . POPE (Li et al., 2023e) is to evaluate object hallucination in multimodal LLMs. We follow POPE and report the F1 Score for the binary choice questions. MME (Fu et al., 2023) contains 2800 binary choice questions for perception and cognition evaluation. We report the overall perception score for the evaluated models. SEED-I (Li et al., 2023a) contains 14K multiple choice questions on images. MMBench Liu et al. ( 2023c) is another multi-choice benchmark to evaluate the multi-modal understanding capability of MLLMs. We report the multiple choice accuracy on SEED-I and MMBench dev set. LLaVA-W (LLaVA-Bench In-the-Wild) (Liu et al., 2023b) and MM-Vet (Yu et al., 2023) are open-ended generation tasks, which compute the evaluation score by prompting a GPT-4 based evaluator (OpenAI, 2023b) with both the predicted and ground-truth reference answer. The score is then scaled to the range of 0 to 100.We provide more implementation details in appendix A.1.First, we evaluate model performance on the tag listing task, aiming to answer two research questions:(1) Do model sizes matter in terms of learning SoM ability? (2) How will different sets of extra training data impact the SoM performance? We design the listing data based on images with ground-truth mask annotations from MS-COCO, and enumerate each object with corresponding class name. An example list is "1. person, 2. cat, 3. dog.". We compute list-wise accuracy, where for a caption with N items, the score is M N with M items predicted correctly by the model. With human annotation of objects in an image, we can automatically create abundant rule-based data (up to 100k) for studying model behaviors and perform quantitative evaluations.For the first question, we find that larger LLM performs better for the listing task (see Figure 3a ), presumably benefiting from the stronger language prior to help learn SoM prompting. For the second question, we decompose the 665k instruction data from LLaVA-1.5 (Liu et al., 2023a) into two parts. We find that both general caption-QA data, as well as OCR-text data contribute to learning SoM ability when limited listing data are available (10k). The reason could be that OCR can help with identifying numeric tags, and general caption may help the model to recognize objects within an image, both of them are fundamental abilities required by SoM. In general, other visual instruction data may benefit learning SoM, especially when SoM data is scarce.Overall, we observe that with only 10k data, we can outperform zero-shot GPT-4V in listing accuracy, whereas growing data size from 50k to 100k only slightly improves the listing performance. These findings suggest that collecting a small amount of data may be sufficient for learning SoM prompting.We then train LLaVA-1.5 on our collected dataset and perform evaluation on MLLM benchmarks. As shown in Table 2 , we observe that our SoM-LLaVA-1.5, which is trained with a mixture of LLaVA visual instructions and our SoM data in order to learn SoM prompting, Table 3 : Performance comparison on LLaVA models. We reproduced results of LLaVA-1.5 on MME and MMBench-dev with the official checkpoints and evaluation scripts. also obtains superior performance on general MLLM tasks. Surprisingly, we find that even without tagged images, SoM-LLaVA still attains strong performance and substantial improvement over the orignal LLaVA. This indicates the quality of our data and the potential of introducing listing data into general MLLM training to improve visual understanding and reasoning, as well as reduce hallucinations. We conjecture the reason that the great performance of SoM-LLaVA on non-tagged images is that "listing items one by one" with visual prompting guides the model to learn fine-grained semantics for image features. Related case studies and visualizations are in appendix A.2. For the performance of open-vocabulary listing, we present examples in appendix A.3.We evaluate LLaVA and SoM-LLaVA models with 7B and 13B LLM backbones, to further demonstrate the effectiveness of adding SoM data into the instruction tuning stage.As shown in Table 3 , we observed performance improvements across a wide range of benchmarks testing multimodal understanding and reasoning capabilities. These gains are consistent on both model architectures.Finally, we perform ablation on different data mixture strategies in Table 4 . We consider mixing our listing and QA data generated from Section 4.3 with LLaVA-665k (Liu et al., 2023a) , trained separately or together. Empirically, we find that mixing listing and QA data yields the best overall performance. In Section 5.2, we find OCR data can help the learning of listing. Here we also notice that "listing item one by one" can in turn greatly improve the performance of OCR related task. The results on POPE indicates our data leads to lower hallucinations compared with ShareGPT-4V, which is a dense caption dataset without visual prompting. Placing tags on the images can seamlessly encode spatial information into the data for MLLMs to learn fine-grained vision language alignment. (Liu et al., 2023a) . Listing and QA is from our SoM dataset with tagged image-text pairs. ShareGPT-4V is from (Chen et al., 2023) with the same MS-COCO images as our 2k QA data and detailed captions from GPT-4V. Results are evaluated on the 13B LLM. 
analysis: 
 SoM-LLaVA-1.5 LLaVA-1.5Figure 4 : A comparative example of attention maps extracted from LLaVA-1.5 and SoM-LLaVA-1.5, where five objects (e.g., laptop, chair, monitor, desk lamp, and printer) are tagged. We highlight the top-5 most attended image patches of the models on each object's numeric tags individually. SoM-LLaVA is better at attending to objects following numeric text and tags.We first analyze the tag-listing capacity of SoM-LLaVA-1.5 (13B) acquired through finetuning. In Figure 4 , we show the attention maps on the five tagged objects, which are extracted from SoM-LLaVA-1.5 and LLaVA-1.5 respectively. The comparative example showcases that although both models can locate their model attention on the mentioned objects to some extent, the fine-tuned SoM-LLaVA-1.5 model can attend to and focus on characteristic regions of the object, which can also be accurately guided by the numeric ID tags. For example, the comparative attention maps on the object "Laptop" tagged with number 1 show that SoM-LLaVA-1.5 can clearly attend to the mentioned object with its main focus. In contrast, LLaVA-1.5 mistakenly attends to the monitor instead of the laptop, due to high similarity between these two objects.In addition, we also observe that SoM-LLaVA-1.5 can be efficiently guided by the numeric ID tags to focus on the specific object the user refers to, even with multiple similar objects within the image. For example, the attention map of SoM-LLaVA-1.5 on the "Chair" tagged with a number 2 is mostly focusing on the chair on the left-hand side, instead of the similar chair on the right-hand side. SoM prompting in SoM-LLaVA-1.5 with such the capacity to accurately locate the tagged object, enables more flexible and easier user-referring queries without complicated language descriptions. The attention maps also verify our early hypothesis regarding the implicit association among the text, tag, and object in SoM prompting.User:Whose pants' color is the same as someone else's white shirt?LLaVA-1.5:The man wearing brown pants has the same pants color as the woman wearing a white shirt.SoM-LLaVA-1.5:The person tagged with number 1 is wearing pants that are the same color as the person tagged with number 9's white shirt.The person labeled with the number 2 is wearing white pants, and their color matches the white shirt of the person labeled with the number 9. User: How many objects are the same type of the object tagged as 13?There are two objects of the same type as the one tagged as 13.There are three objects of the same type as the object tagged as 13, which is an apple.The object tagged as number 13 is an apple. In this image, there are three other apples, tagged with the numbers 3, 6, and 9. So, there are a total of four apples, including the one tagged as 13. In Figure 5 , we examine a multi-step visual reasoning question (i.e., "Whose pants' color is the same as someone else's white shirt"), which requires the MLLM to first identify the mentioned objects (i.e., all the pants in the image and the white shirt), then compare their visual features (i.e., the same white color) to answer the question. From Figure 5 , we observe that LLaVA-1.5 provides an incorrect answer by falsely identifying the person wearing the white shirt as a female. This error can be caused by the inferior object recognition capacity in LLaVA-1.5, or the complexity of the multi-hop question. As for GPT-4V, it made a mistake by assigning tag-2 to the person on its right, leading to a wrong reasoning process and incorrect conclusion. In contrast, SoM-LLaVA-1.5 successfully identifies tags 1 and 9 with the same color in those image regions, while recognizing the two objects as white pants and white shirt, respectively. We show another example in Figure 6 . 
conclusion: 
 In this paper, we start with SoM prompting and propose a new learning paradigm for multimodal LLM training. We show that MLLMs can learn SoM prompting using a small set of synthetic data by listing items one by one. Moreover, we explore the broader impact and find our dataset can benefit general capabilities for MLLMs, where our enhanced model, SoM-LLaVA, consistently outperforms the original LLaVA model across seven multimodal benchmarks. Our dataset and models are released to facilitate vision and language research.Overall, we hope this work could inspire future research on exploring new learning paradigms and data recipe for MLLM training, as well as ways to create better synthetic datasets in the era of LLMs. 
 
 Appendix: Level -1 Level -2 Level -3Figure 11 : SoM tagging granularity analysis with level-1, level-2 and level-3 as coarse to fine.Level -1 Level -2 Level -3Figure 12 : SoM tagging granularity analysis with level-1, level-2 and level-3 as coarse to fine.Level -1 Level -2 Level -3Figure 13 : SoM tagging granularity analysis with level-1, level-2 and level-3 as coarse to fine.Level -1 Level -2 Level -3Question:What are some potential benefits or reasons people might gather at the beach to fly kites or engage in kiteboarding?There are several reasons or benefits for people to gather at a beach to fly kites or engage in kiteboarding:1. Socializing and bonding: Kite flying and kiteboarding events can bring people together, fostering a sense of community and social interaction among the participants. These activities allow people of all ages and backgrounds to establish connections and develop new friendships. In summary, people may gather at a beach to fly kites or engage in kiteboarding for various reasons, including socializing, outdoor recreation, exercise, stress relief, and skill development. These activities can offer a range of physical, mental, and social benefits, making them popular and enjoyable pastimes for those who take part. Table 7 : An example from CogVLM-SFT-311K, with the answer text generated by GPT4-0314 and contains listing.Question:Can you talk about the role of footwork, movement, and jumping in tennis?Answer: Footwork, movement, and jumping are crucial components of a good tennis game, both in terms of enhancing overall performance and reducing the risk of injuries. 1. Footwork: Good footwork allows a tennis player to move efficiently and quickly around the court, enabling them to reach the ball and return it effectively. Proper footwork involves maintaining a balanced stance, taking small adjustment steps, and employing various movement patterns such as the shuffle, crossover, and side-step. Mastering footwork not only improves speed and agility but also ensures the player is in the right position to execute their strokes with optimal power and accuracy. 2. Movement: Movement is an essential aspect of tennis, as players must continually readjust their position to respond to their opponent's shots. Efficient movement on the court requires a combination of speed, agility, flexibility, and endurance. Anticipation plays a significant role in effective movement, as players must predict the trajectory of the ball and adjust their position accordingly. Top players often possess the ability to change direction quickly and cover the court effectively, enabling them to counter their opponent's strategies and make challenging shots. 3. Jumping: Jumping is a valuable skill in tennis, particularly when executing specific shots such as the overhead smash, serve or aggressive baseline shots. A well-executed jump allows the player to generate more power, reach for higher balls, and create sharper angles in their shots. Jumping while hitting the ball gives the player a slight height advantage, which enables them to strike the ball at a higher contact point, resulting in a more aggressive and effective shot. However, it is essential to practice proper techniques and ensure a safe landing to avoid potential injuries.In conclusion, footwork, movement, and jumping are vital components of tennis performance. Mastering these skills makes a player more effective and efficient on the court, enhancing their ability to respond to various situations and execute a wide range of shots. Proper training and practice in these areas can significantly elevate a player's game and contribute to overall success in the sport. https://www.adept.ai/blog/fuyu-8b 


Decision: This paper examines whether multimodal large language models (MLLM) can map objects "marked" in the image to the words in a prompt. The authors create a dataset based on MSCOCO by automatically finding the objects, tagging them with numbers ("marking them"), and adding captions using a LLM. They also discuss how a recent model fail at describing the marked aspects of an image, and show that the model performance can be improved when trained on the collected dataset. The reviewers find the paper clear, and the proposed approach interesting and well executed. I agree with the reviewers but encourage the authors to evaluate their method on commonly-used fine-grained benchmarks such as GQA (as Reviewer nfpB points out) as well as typical retrieval and QA benchmark to see if the performance on generic tasks changes/degrades with the additional training.

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: Set-of-Mark (SoM) prompting (placing numbered tags on images before feeding them to the models) has been shown to improve visual reasoning on GPT-4V.
This paper creates a dataset helping MLLMs to acquire the SoM visual prompting ability, and shows that MLLMs other than GPT-4V (such as LLaVA-1.5) can also benefit from this technique. The SoM models are evaluated on 5 benchmarks and show improvement even without the tags. The authors also investigated the mechanism of SoM.
strength_weakness: strength_weakness: The approach's effectiveness is only demonstrated with LLaVA-1.5. Probing other types or sizes of VLMs can be useful to better understand its applicability. 

This adds additional overhead and needs to retrain the instruction tuning stage. It seems unclear if fine-tuning an already instruction-tuned model using the SoM data would work.
questions: questions: In Figure 3 (a), increasing the listing data from 75k to 100k seems to lead to a slightly lower accuracy for both the 7B and 13B models. Do the authors have an idea of why?

Adding tags may obstruct very small objects. Can the technique still be helpful in this scenario?

Can the authors indicate how long the training was for both models?

The LaTeX style seems different from the official version.

Typo:
- Paragraph 1: "can implicitly aligning [...]" -> "can implicitly align [...]"
- 4.3, "GPT-4V is asked to generate mulit-turn" -> "multi-turn"

Minor:
- Figure 2: it's a bit hard to read the tags behind the attention maps, they can be emphasized. Also, some red patches seem to be going a bit off the image.
paper_summary: paper_summary: The paper proposes a new dataset for Set-of-Mark (SoM) prompting for MLLMs. SoM allows MLLMs to associate objects with text by assigning numbers to different objects. Adding the proposed dataset to existing visual instruction tuning datasets improves the performance on five visual reasoning tasks. The SoM dataset is created by modifying an existing dataset (MS-COCO) that has human annotations corresponding to bounding boxes, and captions. An existing model (SAM) is used to segment the images prior to adding numbers on top of each segment then use GPT4-v to generate the textual descriptions. The authors fine-tune the pertained LLaVA-1.5 model using LLAVA-1.5 fine-tuning data in addition to the newly created dataset and show improvements on five visual reasoning tasks.
strength_weakness: strength_weakness: - The proposed approach and dataset are only applied to one MLLM. It would be good to try other MLLMs to see if the results generalize.
- It would be good to evaluate the in-context learning (without fine-tuning) performance of using exemplars from the new dataset.
paper_summary: paper_summary: This paper presents a simple and compelling idea for enriching the spatial and fine-grained reasoning of multimodal large language models (MLLMs): rather than solely condition language generation on an image, the key idea is to augment an image with a “set of marks” (alphanumeric tags overlaid on the image) that are associated with key locations or reference points. As an example, given an image of a crowd at a concert, a plausible set of marks might include a different tag on each of the members of the crowd, a mark on the stage, and a mark on each of the musicians and their corresponding instruments.

Note that this idea of “marking” is an alternative input scheme that type checks with existing MLLMs that take in arbitrary images and generate text; at inference time, one can still provide untagged images, or they can manually tag an image with marks, and have the MLLM generate outputs referring to individual marked objects.

The bulk of the paper 1) discusses the failures of existing MLLMs such as LLaVa-1.5 when it comes to reasoning over “marked images”, 2) curating a small “marked” dataset of just 10-30K examples of images with labeled outputs just consisting of “List items one by one: <tag 1>: <description> ...”, and 3) evaluating MLLMs trained with this additional data on a spectrum of evaluation tasks. 

In all, the proposed “marking” scheme is simple, scalable, and leads to MLLMs that outperform non-marked variants **in the absence of marking data during evaluation**, especially on tasks requiring fine-grained reasoning. The paper also presents case studies verifying that MLLMs trained with marking can integrate such marks for complex reasoning over full scenes, beyond the capabilities in the training data.
strength_weakness: strength_weakness: While the existing evaluations are compelling, it would make this work even stronger to explicitly evaluate the trained MLLMs on evaluations that provide calibrated/absolute metrics on spatial and compositional reasoning (e.g., GQA question answering accuracy, RefCOCO bounding box prediction, etc.). I recommend that the authors look into strengthening the existing evaluations for the final version of the paper.
questions: questions: Is there a reason that the results don’t include a 7B parameter model? Table 2 only shows a 13B variant of the mark-augmented models.

Do we need GPT-4V to produce the descriptions for each of the tagged objects? If we’re using MSCOCO images to seed our image dataset anyway, can we just use the object labels from that dataset (and possibly other datasets like Visual Genome that have individual region/object captions)?
paper_summary: paper_summary: This paper presents a new data source to better learn multimodal LLMs by collecting listing and QA data with set-of-mark (SoM) prompting from GPT4V. A new learning paradigm is also designed to enhance the visual understanding ability of MLLMs. Results show that the new dataset can improve the performance of MLLMs on various benchmarks and enable SoM prompting ability of MLLMs.
strength_weakness: strength_weakness: - The improvements look marginal on some of the benchmarks. In the abstract, it is mentioned that the method can " reduce hallucinations in Multimodal LLMs" but the improvements on POPE are not very significant. Can you provide an analysis of this phenomenon? 

- How to ensure the quality of the dataset? GPT4V may produce wrong results. Would this affect the final performance of the models trained with the dataset?
paper_summary: paper_summary: This paper studies the Set-of-Mark capability of existing VLMs other than GPT4-V, and proposes a valid solution to construct training data for getting this capability.
strength_weakness: strength_weakness: 1. The contribution of the work is limited. It is yet another paper which constructs a training data, train the model, and get the capability. 

2. The paper is not ready to publish yet. Many details about the approach are missing, I tried to read the section 4 multiple times but still cannot get the full picture of the construction process. More illustrations of the approach and the constructed examples will be necessary for reader to understand the proposed approach. Maybe move the examples from Appendix 1.6 into the main content. Instead, section 3 should go to Appendix, conjunction with moving the primary investigation results / conclusion goes to the introduction. 


3. One key motivation is that the Set-of-Mark capability is important for visual referring prompting. As pointed in the related work, there is another important type of the referring technology which put user inputs (point, bounding boxes, sketch) directly into the text input. A comparison of Set-of-Mask approach and direct referring approach will make the paper more motivated. Also, the author should discuss and cite more papers here, there are many recent work on this direction.
questions: questions: In section 3.2 it is unclear what is the binary classifier, and other details about the classifier.
paper_summary: paper_summary: The paper introduces a novel dataset and learning paradigm aimed at enhancing Multimodal Large Language Models (MLLMs) by teaching them to list items in images one by one. This method is designed to improve the association of visual objects with corresponding textual descriptions through a process called Set-of-Mark (SoM) Prompting. By tagging images with numeric identifiers and prompting models to list these tags, the study demonstrates significant improvements in visual reasoning and a reduction in hallucinations across several MLLM benchmarks.
strength_weakness: strength_weakness: The research primarily focuses on t specific aspect of multimodal learning, which might be seen as a limited application area. Expanding this methodology to foundational models could make the work more impactful.
questions: questions: n/a

Comments: title: Author follow-up
Author response
Author follow-up
Author response
Reponse to the authors
Author Response
Post-Rebuttal Response
comment: Dear reviewer, 

Thank you again for reviewing our paper and providing useful feedback! Since the author-reviewer discussion is coming to an end, please let us know if there are any unresolved concerns. Moreover, we would be happy to hear your thoughts on our replies, and if you could consider increasing the score after reading other reviews and rebuttals.

Thanks,
Authors
Thank you so much for the support of our work!
Dear reviewer, 

Thank you again for reviewing our paper and providing useful feedback! Since the author-reviewer discussion is coming to an end, please let us know if there are any unresolved concerns. Moreover, we would be happy to hear your thoughts on our replies, especially the additional results on llava-1.5-7b and xgen-mm, and if you could consider increasing the score given our responses above.

Thanks,
Authors
Thanks for your response. After reading other reviews as well as the rebuttal, I think the proposed SoM dataset and the results presented in the paper would be beneficial for future research. Therefore, I would upgrade my rating to "accept".
Thank you so much for the appreciation and support of our work, and the prompt response!
Thanks to the authors for the details and additional results on other open-sourced base models, showing consistent improvement. I believe that adding the details and results provided can make the paper clearer.

I increased my score as the answers dealt with my main concerns.
Thank you so much for the appreciation and support of our work! 

Here are the results on GQA ( it takes longer than expected to add GQA results with some bug fixes). We will continue exploring additional benchmarks to show the effectiveness of our method. 

| Model             | GQA  |
|-------------------|------|
| LLaVA-1.5-7B      | 62.0 |
| SoM-LLaVA-1.5-7B  | 62.7 |
| LLaVA-1.5-13B     | 63.3 |
| SoM-LLaVA-1.5-13B | 63.8 |
Thank you for answering my questions and taking the time to add the additional 7B results; I thought this paper was very strong to begin with, and that opinion has not changed with the rebuttal - I will definitely be pushing for acceptance at this time! 

(Would definitely be great to run out the additional benchmarks for the final version though; just as a way to make the paper even stronger).