CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning
  in Large Language Models
Authors: name: Runpeng Dai; name: Linfeng Song; name: Haolin Liu; name: Zhenwen Liang; name: Dian Yu; name: Haitao Mi; name: Zhaopeng Tu; name: Rui Liu; name: Tong Zheng; name: Hongtu Zhu; name: Dong Yu
Published: 2025-09-11 17:59:17

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm
for enhancing the reasoning ability of Large Language Models (LLMs). Yet
current RLVR methods often explore poorly, leading to premature convergence and
entropy collapse. To address this challenge, we introduce Curiosity-Driven
Exploration (CDE), a framework that leverages the model's own intrinsic sense
of curiosity to guide exploration. We formalize curiosity with signals from
both the actor and the critic: for the actor, we use perplexity over its
generated response, and for the critic, we use the variance of value estimates
from a multi-head architecture. Both signals serve as an exploration bonus
within the RLVR framework to guide the model. Our theoretical analysis shows
that the actor-wise bonus inherently penalizes overconfident errors and
promotes diversity among correct responses; moreover, we connect the
critic-wise bonus to the well-established count-based exploration bonus in RL.
Empirically, our method achieves an approximate +3 point improvement over
standard RLVR using GRPO/PPO on AIME benchmarks. Further analysis identifies a
calibration collapse mechanism within RLVR, shedding light on common LLM
failure modes.