WINA: Weight Informed Neuron Activation for Accelerating Large Language
  Model Inference
Authors: name: Sihan Chen; name: Dan Zhao; name: Jongwoo Ko; name: Colby Banbury; name: Huiping Zhuang; name: Luming Liang; name: Tianyi Chen
Published: 2025-05-26 02:37:32

Abstract: The growing computational demands of large language models (LLMs) make
efficient inference and activation strategies increasingly critical. While
recent approaches, such as Mixture-of-Experts (MoE), leverage selective
activation but require specialized training, training-free sparse activation
methods offer broader applicability and superior resource efficiency through
their plug-and-play design. However, many existing methods rely solely on
hidden state magnitudes to determine activation, resulting in high
approximation errors and suboptimal inference accuracy. To address these
limitations, we propose WINA (Weight Informed Neuron Activation), a novel,
simple, and training-free sparse activation framework that jointly considers
hidden state magnitudes and the column-wise ell_2-norms of weight matrices.
We show that this leads to a sparsification strategy that obtains optimal
approximation error bounds with theoretical guarantees tighter than existing
techniques. Empirically, WINA also outperforms state-of-the-art methods (e.g.,
TEAL) by up to 2.94% in average performance at the same sparsity levels,
across a diverse set of LLM architectures and datasets. These results position
WINA as a new performance frontier for training-free sparse activation in LLM
inference, advancing training-free sparse activation methods and setting a
robust baseline for efficient inference. The source code is available at
https://github.com/microsoft/wina.