Feather the Throttle: Revisiting Visual Token Pruning for
  Vision-Language Model Acceleration
Authors: name: Mark Endo, name: Xiaohan Wang, name: Serena Yeung-Levy

Recent works on accelerating Vision-Language Models show that strong
performance can be maintained across a variety of vision-language tasks despite
highly compressing visual information. In this work, we examine the popular
acceleration approach of early pruning of visual tokens inside the language
model and find that its strong performance across many tasks is not due to an
exceptional ability to compress visual information, but rather the benchmarks'
limited ability to assess fine-grained visual capabilities. Namely, we
demonstrate a core issue with the acceleration approach where most tokens
towards the top of the image are pruned away. Yet, this issue is only reflected
in performance for a small subset of tasks such as localization. For the other
evaluated tasks, strong performance is maintained with the flawed pruning
strategy. Noting the limited visual capabilities of the studied acceleration
technique, we propose FEATHER (Fast and Effective Acceleration wiTH Ensemble
cRiteria), a straightforward approach that (1) resolves the identified issue
with early-layer pruning, (2) incorporates uniform sampling to ensure coverage
across all image regions, and (3) applies pruning in two stages to allow the
criteria to become more effective at a later layer while still achieving
significant speedup through early-layer pruning. With comparable computational
savings, we find that FEATHER has more than 5times performance improvement
on the vision-centric localization benchmarks compared to the original
acceleration approach.