OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces
Authors: name: Zehan Wang; name: Ziang Zhang; name: Hang Zhang; name: Luping Liu; name: Rongjie Huang; name: Xize Cheng; name: Hengshuang Zhao; name: Zhou Zhao
Published: 2024-07-16 16:24:31

Abstract: Recently, human-computer interaction with various modalities has shown
promising applications, like GPT-4o and Gemini. Given the foundational role of
multimodal joint representation in understanding and generation pipelines,
high-quality omni joint representations would be a step toward co-processing
more diverse multimodal information. In this work, we present OmniBind,
large-scale multimodal joint representation models ranging in scale from 7
billion to 30 billion parameters, which support 3D, audio, image, and language
inputs. Due to the scarcity of data pairs across all modalities, instead of
training large models from scratch, we propose remapping and binding the spaces
of various pre-trained specialist models together. This approach enables
"scaling up" by indirectly increasing the model parameters and the amount of
seen data. To effectively integrate various spaces, we dynamically assign
weights to different spaces by learning routers with two objectives:
cross-modal overall alignment and language representation decoupling. Notably,
since binding and routing spaces both only require lightweight networks,
OmniBind is extremely training-efficient. Learning the largest 30B model
requires merely unpaired unimodal data and approximately 3 days on a single
8-4090 node. Extensive experiments demonstrate the versatility and superiority
of OmniBind as an omni representation model, highlighting its great potential
for diverse applications, such as any-query and composable multimodal
understanding.