Rethinking Token Reduction in MLLMs: Towards a Unified Paradigm for
  Training-Free Acceleration
Authors: name: Yuhang Han; name: Xuyang Liu; name: Pengxiang Ding; name: Donglin Wang; name: Honggang Chen; name: Qingsen Yan; name: Siteng Huang
Published: 2024-11-26 18:53:51

Abstract: To accelerate the inference of heavy Multimodal Large Language Models
(MLLMs), this study rethinks the current landscape of training-free token
reduction research. We regret to find that the critical components of existing
methods are tightly intertwined, with their interconnections and effects
remaining unclear for comparison, transfer, and expansion. Therefore, we
propose a unified ''filter-correlate-compress'' paradigm that decomposes the
token reduction into three distinct stages within a pipeline, maintaining
consistent design objectives and elements while allowing for unique
implementations. We additionally demystify the popular works and subsume them
into our paradigm to showcase its universality. Finally, we offer a suite of
methods grounded in the paradigm, striking a balance between speed and accuracy
throughout different phases of the inference. Experimental results across 10
benchmarks indicate that our methods can achieve up to an 82.4% reduction in
FLOPs with a minimal impact on performance, simultaneously surpassing
state-of-the-art training-free methods. Our project page is at
https://ficoco-accelerate.github.io/.