MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs
Authors: name: Sixun Dong, name: Juhua Hu, name: Mian Zhang, name: Ming Yin, name: Yanjie Fu, name: Qi Qian

Vision-Language Models (VLMs) demonstrate impressive performance in
understanding visual content with language instruction by converting visual
input to vision tokens. However, redundancy in vision tokens results in the
degenerated inference efficiency of VLMs. While many algorithms have been
proposed to reduce the number of vision tokens, most of them apply only
unimodal information (i.e., vision/text) for pruning and ignore the inherent
multimodal property of vision-language tasks. Moreover, it lacks a generic
criterion that can be applied to different modalities. To mitigate this
limitation, in this work, we propose to leverage both vision and text tokens to
select informative vision tokens by the criterion of coverage. We first
formulate the subset selection problem as a maximum coverage problem.
Afterward, a subset of vision tokens is optimized to cover the text tokens and
the original set of vision tokens, simultaneously. Finally, a VLM agent can be
adopted to further improve the quality of text tokens for guiding vision
pruning. The proposed method MMTok is extensively evaluated on benchmark
datasets with different VLMs. The comparison illustrates that vision and text
information are complementary, and combining multimodal information can surpass
the unimodal baseline with a clear margin. Moreover, under the maximum coverage
criterion on the POPE dataset, our method achieves a 1.87x speedup while
maintaining 98.7% of the original performance on LLaVA-NeXT-13B. Furthermore,
with only four vision tokens, it still preserves 87.7% of the original
performance on LLaVA-1.5-7B. These results highlight the effectiveness of
coverage in token selection.