CoreMatching: A Co-adaptive Sparse Inference Framework with Token and
  Neuron Pruning for Comprehensive Acceleration of Vision-Language Models
Authors: name: Qinsi Wang; name: Hancheng Ye; name: Ming-Yu Chung; name: Yudong Liu; name: Yueqian Lin; name: Martin Kuo; name: Mingyuan Ma; name: Jianyi Zhang; name: Yiran Chen
Published: 2025-05-25 17:16:34

Abstract: Vision-Language Models (VLMs) excel across diverse tasks but suffer from high
inference costs in time and memory. Token sparsity mitigates inefficiencies in
token usage, while neuron sparsity reduces high-dimensional computations, both
offering promising solutions to enhance efficiency. Recently, these two
sparsity paradigms have evolved largely in parallel, fostering the prevailing
assumption that they function independently. However, a fundamental yet
underexplored question remains: Do they truly operate in isolation, or is there
a deeper underlying interplay that has yet to be uncovered? In this paper, we
conduct the first comprehensive investigation into this question. By
introducing and analyzing the matching mechanism between Core Neurons and Core
Tokens, we found that key neurons and tokens for inference mutually influence
and reinforce each other. Building on this insight, we propose CoreMatching, a
co-adaptive sparse inference framework, which leverages the synergy between
token and neuron sparsity to enhance inference efficiency. Through theoretical
analysis and efficiency evaluations, we demonstrate that the proposed method
surpasses state-of-the-art baselines on ten image understanding tasks and three
hardware devices. Notably, on the NVIDIA Titan Xp, it achieved 5x FLOPs
reduction and a 10x overall speedup. Code is released at
https://github.com/wangqinsi1/2025-ICML-CoreMatching/tree/main.