ABC: Achieving Better Control of Multimodal Embeddings using VLMs
Authors: name: Benjamin Schneider, name: Florian Kerschbaum, name: Wenhu Chen

Visual embedding models excel at zero-shot tasks like visual retrieval and
classification. However, these models cannot be used for tasks that contain
ambiguity or require user instruction. These tasks necessitate a multimodal
embedding model, which outputs embeddings that combine visual and natural
language input. Existing CLIP-based approaches embed images and text
independently, and fuse the result. We find that this results in weak
interactions between modalities, and poor user control over the representation.
We introduce ABC, an open-source multimodal embedding model that uses a
vision-language model backbone to deeply integrate image features with natural
language instructions. ABC achieves bestfor-size performance on MSCOCO
image-to-text retrieval and is the top performing model on classification and
VQA tasks in the Massive Multimodal Embedding Benchmark. With a strongly
unified vision-language representation, ABC can use natural language to solve
subtle and potentially ambiguous visual retrieval problems. To evaluate this
capability, we design CtrlBench, a benchmark that requires interleaving textual
instructions with image content for correct retrieval. ABC advances the state
of multimodal embeddings by offering high-quality representations and flexible
natural language control. Our model and datasets are available at our project
page.