Omni-RGPT: Unifying Image and Video Region-level Understanding via Token
  Marks
Authors: name: Miran Heo; name: Min-Hung Chen; name: De-An Huang; name: Sifei Liu; name: Subhashree Radhakrishnan; name: Seon Joo Kim; name: Yu-Chiang Frank Wang; name: Ryo Hachiuma
Published: 2025-01-14 18:58:04

Abstract: We present Omni-RGPT, a multimodal large language model designed to
facilitate region-level comprehension for both images and videos. To achieve
consistent region representation across spatio-temporal dimensions, we
introduce Token Mark, a set of tokens highlighting the target regions within
the visual feature space. These tokens are directly embedded into spatial
regions using region prompts (e.g., boxes or masks) and simultaneously
incorporated into the text prompt to specify the target, establishing a direct
connection between visual and text tokens. To further support robust video
understanding without requiring tracklets, we introduce an auxiliary task that
guides Token Mark by leveraging the consistency of the tokens, enabling stable
region interpretation across the video. Additionally, we introduce a
large-scale region-level video instruction dataset (RegVID-300k). Omni-RGPT
achieves state-of-the-art results on image and video-based commonsense
reasoning benchmarks while showing strong performance in captioning and
referring expression comprehension tasks.