FastVLM: Efficient Vision Encoding for Vision Language Models
Authors: name: Pavan Kumar Anasosalu Vasu; name: Fartash Faghri; name: Chun-Liang Li; name: Cem Koc; name: Nate True; name: Albert Antony; name: Gokul Santhanam; name: James Gabriel; name: Peter Grasch; name: Oncel Tuzel; name: Hadi Pouransari
Published: 2024-12-17 20:09:55

Abstract: Scaling the input image resolution is essential for enhancing the performance
of Vision Language Models (VLMs), particularly in text-rich image understanding
tasks. However, popular visual encoders such as ViTs become inefficient at high
resolutions due to the large number of tokens and high encoding latency caused
by stacked self-attention layers. At different operational resolutions, the
vision encoder of a VLM can be optimized along two axes: reducing encoding
latency and minimizing the number of visual tokens passed to the LLM, thereby
lowering overall latency. Based on a comprehensive efficiency analysis of the
interplay between image resolution, vision latency, token count, and LLM size,
we introduce FastVLM, a model that achieves an optimized trade-off between
latency, model size and accuracy. FastVLM incorporates FastViTHD, a novel
hybrid vision encoder designed to output fewer tokens and significantly reduce
encoding time for high-resolution images. Unlike previous methods, FastVLM
achieves the optimal balance between visual token count and image resolution
solely by scaling the input image, eliminating the need for additional token
pruning and simplifying the model design. In the LLaVA-1.5 setup, FastVLM
achieves 3.2times improvement in time-to-first-token (TTFT) while
maintaining similar performance on VLM benchmarks compared to prior works.
Compared to LLaVa-OneVision at the highest resolution (1152times1152),
FastVLM achieves comparable performance on key benchmarks like SeedBench and
MMMU, using the same 0.5B LLM, but with 85times faster TTFT and a vision
encoder that is 3.4times smaller.