Visual In-Context Prompting
Authors: name: Feng Li; name: Qing Jiang; name: Hao Zhang; name: Tianhe Ren; name: Shilong Liu; name: Xueyan Zou; name: Huaizhe Xu; name: Hongyang Li; name: Chunyuan Li; name: Jianwei Yang; name: Lei Zhang; name: Jianfeng Gao
Published: 2023-11-22 18:59:48

Abstract: In-context prompting in large language models (LLMs) has become a prevalent
approach to improve zero-shot capabilities, but this idea is less explored in
the vision domain. Existing visual prompting methods focus on referring
segmentation to segment the most relevant object, falling short of addressing
many generic vision tasks like open-set segmentation and detection. In this
paper, we introduce a universal visual in-context prompting framework for both
tasks. In particular, we build on top of an encoder-decoder architecture, and
develop a versatile prompt encoder to support a variety of prompts like
strokes, boxes, and points. We further enhance it to take an arbitrary number
of reference image segments as the context. Our extensive explorations show
that the proposed visual in-context prompting elicits extraordinary referring
and generic segmentation capabilities to refer and detect, yielding competitive
performance to close-set in-domain datasets and showing promising results on
many open-set segmentation datasets. By joint training on COCO and SA-1B, our
model achieves 57.7 PQ on COCO and 23.2 PQ on ADE20K. Code will be
available at https://github.com/UX-Decoder/DINOv.