GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
Authors: name: Shilong Zhang; name: Peize Sun; name: Shoufa Chen; name: Min Xiao; name: Wenqi Shao; name: Wenwei Zhang; name: Kai Chen; name: Ping Luo
Published: 2023-07-07 13:43:44

Abstract: Instruction tuning large language model (LLM) on image-text pairs has
achieved unprecedented vision-language multimodal abilities. However, their
vision-language alignments are only built on image-level, the lack of
region-level alignment limits their advancements to fine-grained multimodal
understanding. In this paper, we propose instruction tuning on
region-of-interest. The key design is to reformulate the bounding box as the
format of spatial instruction. The interleaved sequences of visual features
extracted by the spatial instruction and the language embedding are input to
LLM, and trained on the transformed region-text data in instruction tuning
format. Our region-level vision-language model, termed as GPT4RoI, brings brand
new conversational and interactive experience beyond image-level understanding.
(1) Controllability: Users can interact with our model by both language and
spatial instructions to flexibly adjust the detail level of the question. (2)
Capacities: Our model supports not only single-region spatial instruction but
also multi-region. This unlocks more region-level multimodal capacities such as
detailed region caption and complex region reasoning. (3) Composition: Any
off-the-shelf object detector can be a spatial instruction provider so as to
mine informative object attributes from our model, like color, shape, material,
action, relation to other objects, etc. The code, data, and demo can be found
at https://github.com/jshilong/GPT4RoI.