FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable
  Reasoning
Authors: name: Yuyang Ding; name: Chi Zhang; name: Juntao Li; name: Haibin Lin; name: Xin Liu; name: Min Zhang
Published: 2025-10-26 05:49:38

Abstract: Reinforcement learning with verifiable rewards (RLVR) has emerged as a
promising paradigm for enhancing the reasoning capabilities of large language
models (LLMs). In this context, models explore reasoning trajectories and
exploit rollouts with correct answers as positive signals for policy
optimization. However, these rollouts might involve flawed patterns such as
answer-guessing and jump-in-reasoning. Such flawed-positive rollouts are
rewarded identically to fully correct ones, causing policy models to
internalize these unreliable reasoning patterns. In this work, we first conduct
a systematic study of flawed-positive rollouts in RL and find that they enable
rapid capability gains during the early optimization stage, while constraining
reasoning capability later by reinforcing unreliable patterns. Building on
these insights, we propose Flawed-Aware Policy Optimization (FAPO), which
presents a parameter-free reward penalty for flawed-positive rollouts, enabling
the policy to leverage them as useful shortcuts in the warm-up stage, securing
stable early gains, while gradually shifting optimization toward reliable
reasoning in the later refinement stage. To accurately and comprehensively
detect flawed-positive rollouts, we introduce a generative reward model (GenRM)
with a process-level reward that precisely localizes reasoning errors.
Experiments show that FAPO is effective in broad domains, improving outcome
correctness, process reliability, and training stability without increasing the
token budget.