Random Policy Valuation is Enough for LLM Reasoning with Verifiable
  Rewards
Authors: name: Haoran He, name: Yuxiao Ye, name: Qingpeng Cai, name: Chen Hu, name: Binxing Jiao, name: Daxin Jiang, name: Ling Pan

RL with Verifiable Rewards (RLVR) has emerged as a promising paradigm for
improving the reasoning abilities of large language models (LLMs). Current
methods rely primarily on policy optimization frameworks like PPO and GRPO,
which follow generalized policy iteration that alternates between evaluating
the current policy's value and improving the policy based on evaluation. While
effective, they often suffer from training instability and diversity collapse,
requiring complex heuristic tricks and careful tuning. We observe that standard
RLVR in math reasoning can be formalized as a specialized finite-horizon Markov
Decision Process with deterministic state transitions, tree-structured
dynamics, and binary terminal rewards. Though large in scale, the underlying
structure is simpler than general-purpose control settings for which popular RL
algorithms (e.g., PPO) were developed, suggesting that several sophisticated
techniques in existing methods may be reduced or even omitted. Based on this
insight, we prove a surprising result: the optimal action can be recovered from
the Q-function of a fixed uniformly random policy, thereby bypassing the
generalized policy iteration loop and its associated heuristics. We introduce
Random Policy Valuation for Diverse Reasoning (ROVER) to translate this
principle into a practical and scalable algorithm for LLM math reasoning, a
minimalist yet highly effective RL method that samples actions from a softmax
over these uniform-policy Q-values. ROVER preserves diversity throughout
training, allowing sustained exploration of multiple valid pathways. Across
multiple base models and standard math reasoning benchmarks, ROVER demonstrates
superior performance in both quality (+8.2 on pass@1,
+16.8 on pass@256) and diversity (+17.6\%), despite
its radical simplification compared to strong, complicated existing methods.