Rethinking the Sampling Criteria in Reinforcement Learning for LLM
  Reasoning: A Competence-Difficulty Alignment Perspective
Authors: name: Deyang Kong, name: Qi Guo, name: Xiangyu Xi, name: Wei Wang, name: Jingang Wang, name: Xunliang Cai, name: Shikun Zhang, name: Wei Ye

Reinforcement learning exhibits potential in enhancing the reasoning
abilities of large language models, yet it is hard to scale for the low sample
efficiency during the rollout phase. Existing methods attempt to improve
efficiency by scheduling problems based on problem difficulties. However, these
approaches suffer from unstable and biased estimations of problem difficulty
and fail to capture the alignment between model competence and problem
difficulty in RL training, leading to suboptimal results. To tackle these
limitations, this paper introduces Competence-Difficulty
Alignment Sampling (CDAS), which enables accurate
and stable estimation of problem difficulties by aggregating historical
performance discrepancies of problems. Then the model competence is quantified
to adaptively select problems whose difficulty is in alignment with the model's
current competence using a fixed-point system. Experimental results across a
range of challenging mathematical benchmarks show that CDAS achieves great
improvements in both accuracy and efficiency. CDAS attains the highest average
accuracy against baselines and exhibits significant speed advantages compared
to Dynamic Sampling, a competitive strategy in DAPO, which is 2.33
times slower than CDAS.