Are We Using the Right Benchmark: An Evaluation Framework for Visual
  Token Compression Methods
Authors: name: Chenfei Liao, name: Wensong Wang, name: Zichen Wen, name: Xu Zheng, name: Yiyu Wang, name: Haocong He, name: Yuanhuiyi Lyu, name: Lutao Jiang, name: Xin Zou, name: Yuqian Fu, name: Bin Ren, name: Linfeng Zhang, name: Xuming Hu

Recent endeavors to accelerate inference in Multimodal Large Language Models
(MLLMs) have primarily focused on visual token compression. The effectiveness
of these methods is typically assessed by measuring the accuracy drop on
established benchmarks, comparing model performance before and after
compression. However, these benchmarks are originally designed to assess the
perception and reasoning capabilities of MLLMs, rather than to evaluate
compression techniques. As a result, directly applying them to visual token
compression introduces a task mismatch. Strikingly, our investigation reveals
that simple image downsampling consistently outperforms many advanced
compression methods across multiple widely used benchmarks. Through extensive
experiments, we make the following observations: (i) Current benchmarks are
noisy for the visual token compression task. (ii) Down-sampling is able to
serve as a data filter to evaluate the difficulty of samples in the visual
token compression task. Motivated by these findings, we introduce VTC-Bench, an
evaluation framework that incorporates a data filtering mechanism to denoise
existing benchmarks, thereby enabling fairer and more accurate assessment of
visual token compression methods. All data and code are available at
https://github.com/Chenfei-Liao/VTC-Bench.