Title: Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, LLMs, Post-Training, Reasoning, theorem proving, Lean, f-divergences, Amari $\alpha$-divergences, Distributional Matching, diversity

Abstract: Reinforcement Learning (RL) has become the _de facto_ standard for tuning LLMs to solve tasks involving reasoning.
However, growing evidence shows that such models often suffer from a significant loss in diversity.
We argue that this arises because RL implicitly optimizes the Reverse KL to a target distribution, which concentrates on certain high-probability regions of the target while neglecting others. 
In this work, we instead begin from an explicit target distribution, obtained by filtering out incorrect answers while preserving the relative probabilities of correct ones.
Starting from a pre-trained LLM, we approximate this target distribution using Amari's $\alpha$-divergence family, which unifies prior approaches and enables direct control of the precision–diversity trade-off by interpolating between mode-seeking and mass-covering divergences.
On a Lean theorem-proving benchmark, our method achieves state-of-the-art performance along the coverage–precision Pareto frontier.

TL;DR: We propose using a family of divergences that span mode seeking to mode covering to balance between precision and diversity in training LLMs for reasoning tasks