Top-nσ: Not All Logits Are You Need
Authors: name: Chenxia Tang; name: Jianchun Liu; name: Hongli Xu; name: Liusheng Huang
Published: 2024-11-12 08:46:43

Abstract: Large language models (LLMs) typically employ greedy decoding or
low-temperature sampling for reasoning tasks, reflecting a perceived trade-off
between diversity and accuracy. We challenge this convention by introducing
top-nsigma, a novel sampling method that operates directly on pre-softmax
logits by leveraging a statistical threshold. Our key insight is that logits
naturally separate into a Gaussian-distributed noisy region and a distinct
informative region, enabling efficient token filtering without complex
probability manipulations. Unlike existing methods (e.g., top-p, min-p)
that inadvertently include more noise tokens at higher temperatures,
top-nsigma maintains a stable sampling space regardless of temperature
scaling. We also provide a theoretical analysis of top-nsigma to better
understand its behavior. The extensive experimental results across four
reasoning-focused datasets demonstrate that our method not only outperforms
existing sampling approaches but also surpasses greedy decoding, while
maintaining consistent performance even at high temperatures.