Florence-VL: Enhancing Vision-Language Models with Generative Vision
  Encoder and Depth-Breadth Fusion
Authors: name: Jiuhai Chen; name: Jianwei Yang; name: Haiping Wu; name: Dianqi Li; name: Jianfeng Gao; name: Tianyi Zhou; name: Bin Xiao
Published: 2024-12-05 18:50:39

Abstract: We present Florence-VL, a new family of multimodal large language models
(MLLMs) with enriched visual representations produced by Florence-2, a
generative vision foundation model. Unlike the widely used CLIP-style vision
transformer trained by contrastive learning, Florence-2 can capture different
levels and aspects of visual features, which are more versatile to be adapted
to diverse downstream tasks. We propose a novel feature-fusion architecture and
an innovative training recipe that effectively integrates Florence-2's visual
features into pretrained LLMs, such as Phi 3.5 and LLama 3. In particular, we
propose "depth-breath fusion (DBFusion)" to fuse the visual features extracted
from different depths and under multiple prompts. Our model training is
composed of end-to-end pretraining of the whole model followed by finetuning of
the projection layer and the LLM, on a carefully designed recipe of diverse
open-source datasets that include high-quality image captions and
instruction-tuning pairs. Our quantitative analysis and visualization of
Florence-VL's visual features show its advantages over popular vision encoders
on vision-language alignment, where the enriched depth and breath play
important roles. Florence-VL achieves significant improvements over existing
state-of-the-art MLLMs across various multi-modal and vision-centric benchmarks
covering general VQA, perception, hallucination, OCR, Chart,
knowledge-intensive understanding, etc. To facilitate future research, our
models and the complete training recipe are open-sourced.
https://github.com/JiuhaiChen/Florence-VL