SEAP: Training-free Sparse Expert Activation Pruning Unlock the
  Brainpower of Large Language Models
Authors: name: Xun Liang, name: Hanyu Wang, name: Huayi Lai, name: Simin Niu, name: Shichao Song, name: Jiawei Yang, name: Jihao Zhao, name: Feiyu Xiong, name: Bo Tang, name: Zhiyu Li

Large Language Models have achieved remarkable success across various natural
language processing tasks, yet their high computational cost during inference
remains a major bottleneck. This paper introduces Sparse Expert Activation
Pruning (SEAP), a training-free pruning method that selectively retains
task-relevant parameters to reduce inference overhead. Inspired by the
clustering patterns of hidden states and activations in LLMs, SEAP identifies
task-specific expert activation patterns and prunes the model while preserving
task performance and enhancing computational efficiency. Experimental results
demonstrate that SEAP significantly reduces computational overhead while
maintaining competitive accuracy. Notably, at 50% pruning, SEAP surpasses both
WandA and FLAP by over 20%, and at 20% pruning, it incurs only a 2.2%
performance drop compared to the dense model. These findings highlight SEAP's
scalability and effectiveness, making it a promising approach for optimizing
large-scale LLMs.