VLsI: Verbalized Layers-to-Interactions from Large to Small Vision
  Language Models
Authors: name: Byung-Kwan Lee; name: Ryo Hachiuma; name: Yu-Chiang Frank Wang; name: Yong Man Ro; name: Yueh-Hua Wu
Published: 2024-12-02 18:58:25

Abstract: The recent surge in high-quality visual instruction tuning samples from
closed-source vision-language models (VLMs) such as GPT-4V has accelerated the
release of open-source VLMs across various model sizes. However, scaling VLMs
to improve performance using larger models brings significant computational
challenges, especially for deployment on resource-constrained devices like
mobile platforms and robots. To address this, we propose VLsI: Verbalized
Layers-to-Interactions, a new VLM family in 2B and 7B model sizes, which
prioritizes efficiency without compromising accuracy. VLsI leverages a unique,
layer-wise distillation process, introducing intermediate "verbalizers" that
map features from each layer to natural language space, allowing smaller VLMs
to flexibly align with the reasoning processes of larger VLMs. This approach
mitigates the training instability often encountered in output imitation and
goes beyond typical final-layer tuning by aligning the small VLMs' layer-wise
progression with that of the large ones. We validate VLsI across ten
challenging vision-language benchmarks, achieving notable performance gains
(11.0% for 2B and 17.4% for 7B) over GPT-4V without the need for model scaling,
merging, or architectural changes.