Token Reduction Should Go Beyond Efficiency in Generative Models -- From
  Vision, Language to Multimodality
Authors: name: Zhenglun Kong; name: Yize Li; name: Fanhu Zeng; name: Lei Xin; name: Shvat Messica; name: Xue Lin; name: Pu Zhao; name: Manolis Kellis; name: Hao Tang; name: Marinka Zitnik
Published: 2025-05-23 11:30:30

Abstract: In Transformer architectures, tokens\textemdash discrete units derived from
raw data\textemdash are formed by segmenting inputs into fixed-length chunks.
Each token is then mapped to an embedding, enabling parallel attention
computations while preserving the input's essential information. Due to the
quadratic computational complexity of transformer self-attention mechanisms,
token reduction has primarily been used as an efficiency strategy. This is
especially true in single vision and language domains, where it helps balance
computational costs, memory usage, and inference latency. Despite these
advances, this paper argues that token reduction should transcend its
traditional efficiency-oriented role in the era of large generative models.
Instead, we position it as a fundamental principle in generative modeling,
critically influencing both model architecture and broader applications.
Specifically, we contend that across vision, language, and multimodal systems,
token reduction can: (i) facilitate deeper multimodal integration and
alignment, (ii) mitigate "overthinking" and hallucinations, (iii) maintain
coherence over long inputs, and (iv) enhance training stability, etc. We
reframe token reduction as more than an efficiency measure. By doing so, we
outline promising future directions, including algorithm design, reinforcement
learning-guided token reduction, token optimization for in-context learning,
and broader ML and scientific domains. We highlight its potential to drive new
model architectures and learning strategies that improve robustness, increase
interpretability, and better align with the objectives of generative modeling.