Title: TokenSculpt: Pruning with Min-Max Spatio-Temporal Duplication for Video Grounding
Primary area: foundation or frontier models, including LLMs
Keywords: Token Pruning, Spatial Grounding, Temporal Grounding, MLLM Acceleration

Abstract: Visual token pruning is essential for reducing computational overhead in multimodal large language models (MLLMs), especially for videos where visual tokens outnumber text ones. Existing pruning methods, typically based on attention or similarity, barely consider the spatiotemporal structure of videos and may incorrectly merge low-similarity or irrelevant tokens, leading to information loss. We propose TokenSculpt, a structure-aware pruning approach designed for video inputs. It aggregates tokens based on similarity while explicitly avoiding low-similarity merges, and applies a bipartite matching strategy to uniformly sample tokens across spatial and temporal dimensions. This design helps preserve the structural integrity of video representations. Additionally, TokenSculpt is compatible with Flash-Attention, enabling efficient integration into modern MLLMs. Experiments across multiple video-language tasks show that TokenSculpt consistently outperforms prior methods. It achieves an average improvement of 2.9% over baselines. While particularly effective in redundant video settings, it also performs well across a range of scenarios. Our approach provides an efficient and scalable solution for video token pruning and improves performance in grounding and related video-language understanding tasks.

TL;DR: SOTA token pruning method for grounding tasks.