Title: Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning
Primary area: foundation or frontier models, including LLMs
Keywords: LLM Reasoning, RL, Diversity

Abstract: It is widely recognized that reinforcement learning (RL) fine-tuning of large language models often leads to diversity collapse, where outputs lack variety. This phenomenon has been linked to a sharpening effect induced by RL. Prior work has proposed a range of heuristics to counteract this effect, but these methods are ad hoc: they frequently trade off correctness for diversity, their effectiveness varies across tasks, and in some cases they even contradict one another. In this work, we place these observations on a rigorous foundation. We first provide a formal proof of why RL fine-tuning exhibits sharpening. Building directly on this analysis, we introduce a principled method—differential smoothing—that provably improves both correctness and diversity, outperforming vanilla RL as well as widely used entropy-based heuristics. Our theory precisely characterizes when existing heuristics help and why they fail, while showing that differential smoothing is universally superior. Extensive experiments with models from 1B to 7B parameters, across domains including CountDown and real-world mathematical reasoning, demonstrate consistent gains. Differential smoothing improves both Pass@1 (correctness) and Pass@k (diversity), with up to 6.7\% improvements on AIME24 dataset.

TL;DR: We propose a novel method to improve RL fine-tuning of LLM