Jointly Reinforcing Diversity and Quality in Language Model Generations
Authors: name: Tianjian Li, name: Yiming Zhang, name: Ping Yu, name: Swarnadeep Saha, name: Daniel Khashabi, name: Jason Weston, name: Jack Lanchantin, name: Tianlu Wang
Organization: AI at Meta

Post-training of Large Language Models (LMs) often prioritizes accuracy and
helpfulness at the expense of diversity. This creates a tension: while
post-training improves response quality, it also sharpens output distributions
and reduces the range of ideas, limiting the usefulness of LMs in creative and
exploratory tasks such as brainstorming, storytelling, or problem solving. We
address this challenge with Diversity-Aware Reinforcement Learning (DARLING), a
framework that jointly optimizes for response quality and semantic diversity.
At its core, DARLING introduces a learned partition function to measure
diversity beyond surface-level lexical variations. This diversity signal is
then combined with a quality reward during online reinforcement learning,
encouraging models to generate outputs that are both high-quality and distinct.
Experiments across multiple model families and sizes show that DARLING
generalizes to two regimes: non-verifiable tasks (instruction following and
creative writing) and verifiable tasks (competition math). On five benchmarks
in the first setting, DARLING consistently outperforms quality-only RL
baselines, producing outputs that are simultaneously of higher quality and
novelty. In the second setting, DARLING achieves higher pass@1 (solution
quality) and pass@k (solution variety). Most strikingly, explicitly optimizing
for diversity catalyzes exploration in online RL, which manifests itself as
higher-quality responses.