VeLAR: Vision-oriEnted Language-Attentive token Reduction for multimodal large language models

Authors: Yizheng Sun, Yanze Xin, Hao Li, Chenghua Lin, Riza Batista-Navarro
Primary area: foundation or frontier models, including LLMs
Keywords: multi-modal large language models, token reduction, model aceleration, foundation models, vision-language learning, instruction tuning

Abstract:
Multi-modal large language models (MLLMs) have made significant strides by integrating visual and textual modalities. However, architectures that pass all vision tokens to the language model (LLM), such as LLaVA, incur high computational costs due to the large number of vision tokens. While approaches using Q-formers as vision-language connectors reduce computational overhead by generating fewer vision tokens, they often experience performance degradation. In this paper, we propose a progressive token reduction method, called VeLAR, that retains the performance of LLaVA-based MLLMs while substantially reducing computational load. We introduce a lightweight cross-attention decision module where vision tokens attend to language tokens. This module is inserted into various layers of the LLM to compute a relevance score for each vision token, dynamically determining whether to prune it. During training, we apply a targeting pruning ratio with Gumbel-Softmax activation to maintain differentiability in the pruning process by attention masking, while in inference, the pruning ratio can be flexibly adjusted to consider different computational trade-offs without re-training. By progressively pruning redundant vision tokens throughout the LLM backbone, our method can reduce 87.5% vision tokens by the final layer and achieve up to a 42% decrease in FLOPs. Across 12 multi-modal benchmarks, the average performance loss is less than 1%, with superior performance observed in 7 of them.

Decision: withdrawn
Reviewer scores: 0

Reviews and discussion:

Metareview: