Title: COMPACT: Common-token Optimized Model Pruning Across Channels and Tokens
Primary area: foundation or frontier models, including LLMs
Keywords: Pruning, Efficient ML

Abstract: Making large language models (LLMs) more efficient in memory, latency, and serving cost is crucial for edge deployment, interactive applications, and sustainable inference at scale. Pruning is a promising technique, but existing pruning methods are limited: width pruning often breaks the standard transformer layout, requiring custom inference code, while depth pruning can cause abrupt accuracy drops. Also, while many pruning approaches are effective against LLMs, they struggle to maintain performance on small language models (SLMs). In this work, we propose COMPACT, which jointly (i) prunes rare vocabulary to shrink embedding/LM head layers and (ii) prunes FFN intermediate channels using common-token–weighted activations, aligning importance with the post-pruning token distribution. COMPACT inherits strengths of both depth and width pruning, such as: deployment-friendliness (keeps a standard transformer architecture), scale-adaptivity (trade off vocab. vs. FFN pruning), competitive pruning times, and strong memory savings alongside throughput gains. Experiments across Qwen, LLaMA, and Gemma families (0.5B–70B) show state-of-the-art downstream performance, with substantial reductions in parameters, GPU memory, and latency.

TL;DR: We present COMPACT: a training-free, deployment-friendly pruning method that jointly removes rare vocabulary and prunes FFN intermediate channels using common-token–weighted activations.