FUSION: Fully Integration of Vision-Language Representations for Deep
  Cross-Modal Understanding
Authors: name: Zheng Liu; name: Mengjie Liu; name: Jingzhou Chen; name: Jingwei Xu; name: Bin Cui; name: Conghui He; name: Wentao Zhang
Published: 2025-04-14 06:33:29

Abstract: We introduce FUSION, a family of multimodal large language models (MLLMs)
with a fully vision-language alignment and integration paradigm. Unlike
existing methods that primarily rely on late-stage modality interaction during
LLM decoding, our approach achieves deep, dynamic integration throughout the
entire processing pipeline. To this end, we propose Text-Guided Unified Vision
Encoding, incorporating textual information in vision encoding to achieve
pixel-level integration. We further design Context-Aware Recursive Alignment
Decoding that recursively aggregates visual features conditioned on textual
context during decoding, enabling fine-grained, question-level semantic
integration. To guide feature mapping and mitigate modality discrepancies, we
develop Dual-Supervised Semantic Mapping Loss. Additionally, we construct a
Synthesized Language-Driven Question-Answer (QA) dataset through a new data
synthesis method, prioritizing high-quality QA pairs to optimize text-guided
feature integration. Building on these foundations, we train FUSION at two
scales-3B, 8B-and demonstrate that our full-modality integration approach
significantly outperforms existing methods with only 630 vision tokens.
Notably, FUSION 3B surpasses Cambrian-1 8B and Florence-VL 8B on most
benchmarks. FUSION 3B continues to outperform Cambrian-1 8B even when limited
to 300 vision tokens. Our ablation studies show that FUSION outperforms
LLaVA-NeXT on over half of the benchmarks under same configuration without
dynamic resolution, highlighting the effectiveness of our approach. We release
our code, model weights, and dataset. https://github.com/starriver030515/FUSION