Can Visual Input Be Compressed? A Visual Token Compression Benchmark for
  Large Multimodal Models
Authors: name: Tianfan Peng; name: Yuntao Du; name: Pengzhou Ji; name: Shijie Dong; name: Kailin Jiang; name: Mingchuan Ma; name: Yijun Tian; name: Jinhe Bi; name: Qian Li; name: Wei Du; name: Feng Xiao; name: Lizhen Cui
Published: 2025-11-04 15:17:06

Abstract: Large multimodal models (LMMs) often suffer from severe inference
inefficiency due to the large number of visual tokens introduced by image
encoders. While recent token compression methods, such as pruning and merging,
have shown promise in reducing redundancy, their evaluation remains fragmented
and inconsistent. In this work, we present UniPruneBench, a unified and
extensible benchmark for visual token pruning in multimodal LLMs. UniPruneBench
provides standardized protocols across six ability dimensions and ten datasets,
covering ten representative compression algorithms and three families of LMMs
(LLaVA-v1.5, Intern-VL3, and Qwen2.5-VL). Beyond task accuracy, it incorporates
system-level metrics such as runtime and prefilling latency to provide a
holistic view. Our experiments uncover several key findings: (1) random pruning
is a surprisingly strong baseline, (2) no single method consistently
outperforms others across scenarios, (3) pruning sensitivity varies
significantly across tasks, with OCR being most vulnerable, and (4) pruning
ratio is the dominant factor governing performance degradation. We believe
UniPruneBench will serve as a reliable foundation for future research on
efficient multimodal modeling.