VIRTUE: Visual-Interactive Text-Image Universal Embedder
Authors: name: Wei-Yao Wang, name: Kazuya Tateishi, name: Qiyu Wu, name: Shusuke Takahashi, name: Yuki Mitsufuji
Organization: Sony

Multimodal representation learning models have demonstrated successful
operation across complex tasks, and the integration of vision-language models
(VLMs) has further enabled embedding models with instruction-following
capabilities. However, existing embedding models lack visual-interactive
capabilities to specify regions of interest from users (e.g., point, bounding
box, mask), which have been explored in generative models to broaden their
human-interactive applicability. Equipping embedding models with visual
interactions not only would unlock new applications with localized grounding of
user intent, which remains unexplored, but also enable the models to learn
entity-level information within images to complement their global
representations for conventional embedding tasks. In this paper, we propose a
novel Visual-InteRactive Text-Image Universal Embedder (VIRTUE) that extends
the capabilities of the segmentation model and the vision-language model to the
realm of representation learning. In VIRTUE, the segmentation model can process
visual prompts that pinpoint specific regions within an image, thereby enabling
the embedder to handle complex and ambiguous scenarios more precisely. To
evaluate the visual-interaction ability of VIRTUE, we introduce a large-scale
Segmentation-and-Scene Caption Retrieval (SCaR) benchmark comprising 1M samples
that aims to retrieve the text caption by jointly considering the entity with a
specific object and image scene. VIRTUE consistently achieves a
state-of-the-art performance with significant improvements across 36 universal
MMEB (3.1%-8.5%) and five visual-interactive SCaR (15.2%-20.3%) tasks.