NeurIPS 2025

Diversity-Aware Policy Optimization for Large
Language Model Reasoning
Jian Yao1, Ran Cheng1,2,3 ∗, Xingyu Wu1, Jibin Wu1,2, Kay Chen Tan1
1 Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University
2 Department of Computing, The Hong Kong Polytechnic University
3 The Hong Kong Polytechnic University Shenzhen Research Institute, Shenzhen, China
nigel97.yao@connect.polyu.hk, {ran-peter.cheng, xingy.wu, jibin.wu, kctan}@polyu.edu.hk
Abstract
The reasoning capabilities of large language models (LLMs) have advanced rapidly,
particularly following the release of DeepSeek-R1, which has inspired a surge of
research into data quality and reinforcement learning (RL) algorithms. Despite the
pivotal role diversity plays in RL, its influence on LLM reasoning remains largely
underexplored. To bridge this gap, this work presents a systematic investigation
into the impact of diversity in RL-based training for LLM reasoning, and proposes
a novel diversity-aware policy optimization method. Across evaluations on 12
LLMs, we observe a strong positive correlation between the solution diversity
and Potential@k (a novel metric quantifying an LLM’s reasoning potential) in
high-performing models. This finding motivates our method to explicitly promote
diversity during RL training. Specifically, we design a token-level diversity and
reformulate it into a practical objective, then we selectively apply it to positive
samples. Integrated into the R1-zero training framework, our method achieves
a 3.5% average improvement across four mathematical reasoning benchmarks,
while generating more diverse and robust solutions. The code is available at
https://github.com/nigelyaoj/R1_zero_Div.
1
Introduction
Recently, the reasoning capabilities of large language models (LLMs) have made remarkable progress,
with significant improvements showcased by OpenAI-o1 [35], DeepSeek-R1 [13], and Kimi-k1.5
[43]. Among these advancements, two key innovations have contributed signif...