SEA: Supervised Embedding Alignment for Token-Level Visual-Textual
  Integration in MLLMs
Authors: name: Yuanyang Yin; name: Yaqi Zhao; name: Yajie Zhang; name: Ke Lin; name: Jiahao Wang; name: Xin Tao; name: Pengfei Wan; name: Di Zhang; name: Baoqun Yin; name: Wentao Zhang
Published: 2024-08-21 17:58:02

Abstract: Multimodal Large Language Models (MLLMs) have recently demonstrated
remarkable perceptual and reasoning abilities, typically comprising a Vision
Encoder, an Adapter, and a Large Language Model (LLM). The adapter serves as
the critical bridge between the visual and language components. However,
training adapters with image-level supervision often results in significant
misalignment, undermining the LLMs' capabilities and limiting the potential of
Multimodal LLMs. To address this, we introduce Supervised Embedding Alignment
(SEA), a token-level alignment method that leverages vision-language
pre-trained models, such as CLIP, to align visual tokens with the LLM's
embedding space through contrastive learning. This approach ensures a more
coherent integration of visual and language representations, enhancing the
performance and interpretability of multimodal LLMs while preserving their
inherent capabilities. Extensive experiments show that SEA effectively improves
MLLMs, particularly for smaller models, without adding extra data or inference
computation. SEA also lays the groundwork for developing more general and
adaptable solutions to enhance multimodal systems.