GLaMM: Pixel Grounding Large Multimodal Model
Authors: name: Hanoona Rasheed; name: Muhammad Maaz; name: Sahal Shaji; name: Abdelrahman Shaker; name: Salman Khan; name: Hisham Cholakkal; name: Rao M. Anwer; name: Erix Xing; name: Ming-Hsuan Yang; name: Fahad S. Khan
Published: 2023-11-06 18:59:57

Abstract: Large Multimodal Models (LMMs) extend Large Language Models to the vision
domain. Initial efforts towards LMMs used holistic images and text prompts to
generate ungrounded textual responses. Very recently, region-level LMMs have
been used to generate visually grounded responses. However, they are limited to
only referring a single object category at a time, require users to specify the
regions in inputs, or cannot offer dense pixel-wise object grounding. In this
work, we present Grounding LMM (GLaMM), the first model that can generate
natural language responses seamlessly intertwined with corresponding object
segmentation masks. GLaMM not only grounds objects appearing in the
conversations but is flexible enough to accept both textual and optional visual
prompts (region of interest) as input. This empowers users to interact with the
model at various levels of granularity, both in textual and visual domains. Due
to the lack of standard benchmarks for the novel setting of generating visually
grounded detailed conversations, we introduce a comprehensive evaluation
protocol with our curated grounded conversations. Our proposed Grounded
Conversation Generation (GCG) task requires densely grounded concepts in
natural scenes at a large-scale. To this end, we propose a densely annotated
Grounding-anything Dataset (GranD) using our proposed automated annotation
pipeline that encompasses 7.5M unique concepts grounded in a total of 810M
regions available with segmentation masks. Besides GCG, GLaMM also performs
effectively on several downstream tasks e.g., referring expression
segmentation, image and region-level captioning and vision-language
conversations. Project Page: https://mbzuai-oryx.github.io/groundingLMM.