Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
Authors: name: Jianwei Yang; name: Hao Zhang; name: Feng Li; name: Xueyan Zou; name: Chunyuan Li; name: Jianfeng Gao
Published: 2023-10-17 17:51:31

Abstract: We present Set-of-Mark (SoM), a new visual prompting method, to unleash the
visual grounding abilities of large multimodal models (LMMs), such as GPT-4V.
As illustrated in Fig. 1 (right), we employ off-the-shelf interactive
segmentation models, such as SAM, to partition an image into regions at
different levels of granularity, and overlay these regions with a set of marks
e.g., alphanumerics, masks, boxes. Using the marked image as input, GPT-4V can
answer the questions that require visual grounding. We perform a comprehensive
empirical study to validate the effectiveness of SoM on a wide range of
fine-grained vision and multimodal tasks. For example, our experiments show
that GPT-4V with SoM outperforms the state-of-the-art fully-finetuned referring
segmentation model on RefCOCOg in a zero-shot setting.