SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs
Authors: name: Jiahui Wang, name: Zuyan Liu, name: Yongming Rao, name: Jiwen Lu

Multimodal Large Language Models (MLLMs) are commonly derived by extending
pre-trained Large Language Models (LLMs) with visual capabilities. In this
work, we investigate how MLLMs process visual inputs by analyzing their
attention mechanisms. We reveal a surprising sparsity phenomenon: only a small
subset (approximately less than 5%) of attention heads in LLMs actively
contribute to visual understanding, termed visual heads. To identify these
heads efficiently, we design a training-free framework that quantifies
head-level visual relevance through targeted response analysis. Building on
this discovery, we introduce SparseMM, a KV-Cache optimization strategy that
allocates asymmetric computation budgets to heads in LLMs based on their visual
scores, leveraging the sparity of visual heads for accelerating the inference
of MLLMs. Compared with prior KV-Cache acceleration methods that ignore the
particularity of visual, SparseMM prioritizes stress and retaining visual
semantics during decoding. Extensive evaluations across mainstream multimodal
benchmarks demonstrate that SparseMM achieves superior accuracy-efficiency
trade-offs. Notably, SparseMM delivers 1.38x real-time acceleration and 52%
memory reduction during generation while maintaining performance parity on
efficiency test. Our project is open sourced at
https://github.com/CR400AF-A/SparseMM.